feat: scheduler + CLI + docker-compose + README
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.
Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
(departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
all_wechat/all).
- Conversations for the message run are discovered from the local customer
mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).
Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
— long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.
Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.
Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).
Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,310 @@
|
||||
"""Command-line entry point for running syncs + the scheduler (M6, Part B).
|
||||
|
||||
Three invocations::
|
||||
|
||||
# One-shot full sync (every entity) into the configured DB:
|
||||
python -m app.sync.cli sync
|
||||
|
||||
# One-shot single-entity sync (departments / friends / messages / ...):
|
||||
python -m app.sync.cli sync --entity friends
|
||||
|
||||
# Long-running scheduler (APScheduler; full sync daily + messages every 4h):
|
||||
python -m app.sync.cli scheduler
|
||||
|
||||
Auth + DB
|
||||
---------
|
||||
Auth comes from env / ``.env`` via the same helpers the smoke script uses
|
||||
(:func:`scripts.smoke_wechat._push_settings_into_environ` +
|
||||
:func:`scripts.smoke_wechat.build_client_from_env`) — session-cookie mode
|
||||
(``YK_USERID`` + ``YK_COMPANY``) is preferred; pcLogin mode is wired but
|
||||
deferred.
|
||||
|
||||
The DB is taken from :attr:`Settings.db_url` (MySQL in prod, SQLite in dev /
|
||||
tests). Tables are created if missing (``Base.metadata.create_all``) on every
|
||||
command so a fresh checkout works without a separate migration step — the
|
||||
CLI is intentionally idempotent / bootstrappy that way. Real schema evolution
|
||||
goes through Alembic; this is a safety net for the operator.
|
||||
|
||||
Output
|
||||
------
|
||||
Progress is printed as JSON lines (one per entity) so the output is both
|
||||
human-skimmable and machine-parseable. ``--quiet`` suppresses the per-run
|
||||
JSON lines; ``--help`` lists the entities.
|
||||
|
||||
Exit codes: ``0`` success, ``1`` exception during run, ``2`` bad CLI usage /
|
||||
missing auth.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import signal
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import models # noqa: F401 (register tables on Base.metadata)
|
||||
from app.db.base import Base
|
||||
from app.sync.scheduler import ENTITY_FUNCS, SyncScheduler
|
||||
|
||||
logger = logging.getLogger("app.sync.cli")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# argparse setup
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="python -m app.sync.cli",
|
||||
description=(
|
||||
"Run a one-shot yunke sync or start the periodic scheduler. "
|
||||
"Auth + DB come from .env (session-cookie mode preferred)."
|
||||
),
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True, metavar="<command>")
|
||||
|
||||
# ``sync`` ----------------------------------------------------------------
|
||||
p_sync = sub.add_parser(
|
||||
"sync",
|
||||
help="Run a one-shot sync (all entities by default).",
|
||||
description=(
|
||||
"Run a one-shot sync into the configured DB. Without --entity, "
|
||||
"runs the full set (org tree + messages + phone calls + SMS). "
|
||||
"With --entity <name>, runs just that one."
|
||||
),
|
||||
)
|
||||
p_sync.add_argument(
|
||||
"--entity",
|
||||
choices=sorted(ENTITY_FUNCS),
|
||||
default="all",
|
||||
help="Entity to sync (default: all).",
|
||||
)
|
||||
p_sync.add_argument(
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="Suppress per-run JSON progress lines (errors still logged).",
|
||||
)
|
||||
|
||||
# ``scheduler`` -----------------------------------------------------------
|
||||
p_sched = sub.add_parser(
|
||||
"scheduler",
|
||||
help="Start the APScheduler (long-running).",
|
||||
description=(
|
||||
"Start the APScheduler loop: full sync daily at FULL_SYNC_HOUR "
|
||||
"(default 02:00 local), incremental message sync every "
|
||||
"MESSAGE_INTERVAL_HOURS (default 4). Runs forever until "
|
||||
"interrupted (Ctrl-C / SIGTERM)."
|
||||
),
|
||||
)
|
||||
p_sched.add_argument(
|
||||
"--full-sync-hour",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Local-time hour (0-23) for the daily full sync. "
|
||||
"Defaults to FULL_SYNC_HOUR env var or 2."
|
||||
),
|
||||
)
|
||||
p_sched.add_argument(
|
||||
"--message-interval-hours",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Hours between incremental message syncs. "
|
||||
"Defaults to MESSAGE_INTERVAL_HOURS env var or 4."
|
||||
),
|
||||
)
|
||||
p_sched.add_argument(
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="Suppress per-run JSON progress lines (errors still logged).",
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Command implementations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _cmd_sync(args: argparse.Namespace) -> int:
|
||||
"""Run a one-shot sync and print a per-entity JSON-line summary."""
|
||||
settings = get_settings()
|
||||
|
||||
# Create tables if missing (idempotent). Lets a fresh container boot
|
||||
# without a separate migration step.
|
||||
engine, session_maker = await _init_db(settings.db_url)
|
||||
try:
|
||||
scheduler = SyncScheduler(
|
||||
engine=engine, session_maker=session_maker
|
||||
)
|
||||
if args.quiet:
|
||||
# The --quiet flag only governs *our* explicit log echo; the
|
||||
# orchestrator's own per-entity logger.exception stays on so
|
||||
# failures are never silent.
|
||||
logging.getLogger("app.sync").setLevel(logging.WARNING)
|
||||
results = await scheduler.run_once(args.entity)
|
||||
_print_sync_summary(args.entity, results)
|
||||
# Exit non-zero only if every result failed; a partial failure is
|
||||
# still useful work delivered and should be exit 0 (the JSON line
|
||||
# surfaces which entities failed).
|
||||
any_success = any(r.status == "success" for r in results)
|
||||
return 0 if any_success or not results else 1
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def _cmd_scheduler(args: argparse.Namespace) -> int:
|
||||
"""Start the long-running scheduler and block until interrupted."""
|
||||
# Env-var fallbacks so docker-compose / .env can drive the cadence
|
||||
# without re-editing the compose command.
|
||||
full_sync_hour = args.full_sync_hour
|
||||
if full_sync_hour is None:
|
||||
full_sync_hour = _env_int("FULL_SYNC_HOUR", default=2)
|
||||
message_interval_hours = args.message_interval_hours
|
||||
if message_interval_hours is None:
|
||||
message_interval_hours = _env_int("MESSAGE_INTERVAL_HOURS", default=4)
|
||||
|
||||
if not (0 <= full_sync_hour <= 23):
|
||||
print(
|
||||
f"full-sync-hour must be 0..23, got {full_sync_hour}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
scheduler = SyncScheduler(
|
||||
full_sync_hour=full_sync_hour,
|
||||
message_interval_hours=message_interval_hours,
|
||||
)
|
||||
await scheduler.start()
|
||||
print(
|
||||
f"[scheduler] running: full sync daily at hour {full_sync_hour}, "
|
||||
f"messages every {message_interval_hours}h. Ctrl-C to stop.",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
# Block forever until SIGINT / SIGTERM. AsyncIOScheduler runs jobs on
|
||||
# its own event loop thread; we just need to keep the process alive.
|
||||
stop_event = asyncio.Event()
|
||||
|
||||
def _signal_handler(*_: Any) -> None:
|
||||
print("\n[scheduler] stop signal received, shutting down...", flush=True)
|
||||
stop_event.set()
|
||||
|
||||
# KeyboardInterrupt is delivered as SIGINT on POSIX, but on Windows
|
||||
# the convention is a Ctrl-C event; loop.add_signal_handler isn't
|
||||
# implemented there. Install both: signal.signal for Windows + the
|
||||
# asyncio version where available.
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
try:
|
||||
asyncio.get_running_loop().add_signal_handler(sig, _signal_handler)
|
||||
except (NotImplementedError, RuntimeError):
|
||||
# Windows / non-main thread — fall back to the stdlib handler.
|
||||
signal.signal(sig, _signal_handler)
|
||||
|
||||
try:
|
||||
await stop_event.wait()
|
||||
finally:
|
||||
await scheduler.stop()
|
||||
return 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _init_db(db_url: str) -> tuple[Any, Any]:
|
||||
"""Create the async engine + session maker and ensure tables exist.
|
||||
|
||||
Returns ``(engine, session_maker)``. The caller owns disposal. Tables
|
||||
are created with ``Base.metadata.create_all`` — idempotent, so this is
|
||||
safe to call on a warm DB.
|
||||
"""
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
engine = create_async_engine(db_url, pool_pre_ping=True, future=True)
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
session_maker = async_sessionmaker(engine, expire_on_commit=False)
|
||||
return engine, session_maker
|
||||
|
||||
|
||||
def _env_int(name: str, *, default: int) -> int:
|
||||
"""Read an int env var; fall back to ``default`` on missing / unparseable."""
|
||||
import os
|
||||
|
||||
raw = os.environ.get(name, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
print(
|
||||
f"[cli] {name}={raw!r} is not an int; using default {default}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return default
|
||||
|
||||
|
||||
def _print_sync_summary(entity: str, results: list[Any]) -> None:
|
||||
"""Print a compact human summary of a finished run (after the JSON lines)."""
|
||||
if not results:
|
||||
print(f"[sync] {entity}: no results (run failed before any entity completed)")
|
||||
return
|
||||
fetched = sum(r.fetched for r in results)
|
||||
upserted = sum(r.upserted for r in results)
|
||||
failed = [r.entity for r in results if r.status != "success"]
|
||||
line = (
|
||||
f"[sync] {entity}: {len(results)} sub-run(s), "
|
||||
f"fetched={fetched}, upserted={upserted}"
|
||||
)
|
||||
if failed:
|
||||
line += f", FAILED={failed}"
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""CLI entry point. Returns a process exit code."""
|
||||
# Configure logging so the scheduler's JSON line + the orchestrator's
|
||||
# exception tracebacks are visible. Quiet flag is applied per-command
|
||||
# after parsing.
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
|
||||
parser = _build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
if args.command == "sync":
|
||||
return asyncio.run(_cmd_sync(args))
|
||||
if args.command == "scheduler":
|
||||
return asyncio.run(_cmd_scheduler(args))
|
||||
except KeyboardInterrupt:
|
||||
# So `python -m app.sync.cli scheduler` exits cleanly on Ctrl-C.
|
||||
return 0
|
||||
except ValueError as exc:
|
||||
# Bad entity name or auth misconfig surfaced as a ValueError —
|
||||
# print a clear message instead of a traceback.
|
||||
print(f"[cli] {exc}", file=sys.stderr)
|
||||
return 2
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.exception("cli command failed")
|
||||
print(f"[cli] {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# argparse with required=True subparsers won't get here, but keep the
|
||||
# lint happy.
|
||||
parser.print_help(sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,501 @@
|
||||
"""APScheduler-based periodic sync for the yunke data mirror (M6, Part A).
|
||||
|
||||
Wraps the orchestrator's ``sync_*`` functions behind a long-running scheduler
|
||||
so the data mirror stays fresh without manual kicks. Two paces:
|
||||
|
||||
* **Daily full sync** at a configurable hour (default ``02:00`` local time,
|
||||
off-hours) — re-scrapes the org tree, accounts, friends, groups, voice
|
||||
calls, plus phone calls and SMS (all entities that have no upstream
|
||||
``since`` cursor and therefore rely on idempotent re-upsert).
|
||||
* **Incremental message sync** more frequently (default every **4 hours**).
|
||||
Messages DO have a real ``[last_timestamp, now]`` window (see
|
||||
:func:`app.sync.orchestrator.sync_wechat_messages`), so a fast cadence
|
||||
catches new chats promptly without re-pulling history. The conversation
|
||||
list is discovered from the already-mirrored ``customer`` table.
|
||||
|
||||
Auth: the scheduler builds a :class:`YunkeClient` per run from env (honoring
|
||||
``.env`` via :func:`scripts.smoke_wechat._push_settings_into_environ`).
|
||||
Session-cookie mode (``YK_USERID`` + ``YK_COMPANY`` [+ ``YK_COOKIE``]) is
|
||||
preferred and is the only mode exercised in production today; pcLogin mode
|
||||
(``YUNKE_USERNAME`` + ``YUNKE_PASSWORD``) is wired but deferred (see README
|
||||
"Current limitations").
|
||||
|
||||
Failure policy: every run is wrapped in a ``try/except`` that logs the error
|
||||
and lets the scheduler keep its next appointment. A single broken entity
|
||||
(scraper 500, network blip) does NOT bring down the loop. Per-entity
|
||||
failures are also captured by the orchestrator as ``status='failed'``
|
||||
``SyncRun`` rows.
|
||||
|
||||
Threading: :class:`SyncScheduler` is built on :class:`AsyncIOScheduler` and
|
||||
is intended to be ``await``-ed inside a long-running asyncio process (the
|
||||
CLI's ``scheduler`` command — see :mod:`app.sync.cli`). ``run_once`` is
|
||||
also awaitable for manual / API-triggered syncs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.cron import CronTrigger
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import models # noqa: F401 (register tables on Base.metadata)
|
||||
from app.db.base import Base
|
||||
from app.session.yunke_client import YunkeClient
|
||||
from app.storage.object_store import ObjectStore
|
||||
from app.sync.orchestrator import (
|
||||
SyncResult,
|
||||
sync_all_wechat,
|
||||
sync_departments,
|
||||
sync_phone_calls,
|
||||
sync_sms,
|
||||
sync_wechat_accounts,
|
||||
sync_wechat_friends,
|
||||
sync_wechat_groups,
|
||||
sync_wechat_messages,
|
||||
sync_wechat_voice_calls,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
__all__ = ["SyncScheduler", "ENTITY_FUNCS"]
|
||||
|
||||
# Cap on the number of conversations one message-sync run will pull. Bounds
|
||||
# runtime + upstream request volume per run (the orchestrator paces each
|
||||
# request, so 200 conversations × N pages each can still take a while).
|
||||
DEFAULT_MAX_CONVERSATIONS_PER_RUN = 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Conversation discovery (drives the incremental message sync)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _discover_conversations(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
limit: int = DEFAULT_MAX_CONVERSATIONS_PER_RUN,
|
||||
) -> list[tuple[str, str]]:
|
||||
"""Return distinct ``(owner_account_id, friend_id)`` WeChat pairs to sync.
|
||||
|
||||
Source: the local ``customer`` mirror (where
|
||||
``channel='wechat'`` and ``owner_account_id`` is non-empty). Reading the
|
||||
already-mirrored table — instead of the upstream ``talkerList`` endpoint
|
||||
— keeps the discovery call off the risk-control budget and makes the
|
||||
incremental run purely a function of the customers we already know
|
||||
about. Order is unspecified; ``limit`` caps work per run.
|
||||
"""
|
||||
from app.db.models import Customer
|
||||
|
||||
stmt = (
|
||||
sa.select(Customer.owner_account_id, Customer.friend_id)
|
||||
.where(Customer.channel == "wechat")
|
||||
.where(Customer.owner_account_id.is_not(None))
|
||||
.where(Customer.owner_account_id != "")
|
||||
.distinct()
|
||||
.limit(limit)
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
return [
|
||||
(owner, friend_id)
|
||||
for owner, friend_id in rows
|
||||
if owner and friend_id
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entity registry — name -> async callable(session, client, store) -> list[SyncResult]
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# One entry per logical "thing the scheduler can run". Each callable returns a
|
||||
# list of SyncResult (one for the entity, or one per conversation for the
|
||||
# message entry). Keeping this as a registry means run_once("friends") and the
|
||||
# scheduler's daily job can share the same dispatch.
|
||||
|
||||
|
||||
async def _run_departments(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_departments(session, client)]
|
||||
|
||||
|
||||
async def _run_accounts(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_accounts(session, client)]
|
||||
|
||||
|
||||
async def _run_friends(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_friends(session, client)]
|
||||
|
||||
|
||||
async def _run_groups(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_groups(session, client)]
|
||||
|
||||
|
||||
async def _run_voice_calls(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_voice_calls(session, client)]
|
||||
|
||||
|
||||
async def _run_phone_calls(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_phone_calls(session, client)]
|
||||
|
||||
|
||||
async def _run_sms(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_sms(session, client)]
|
||||
|
||||
|
||||
async def _run_messages(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
"""Incremental message sync across all locally-known WeChat conversations."""
|
||||
conversations = await _discover_conversations(session)
|
||||
results: list[SyncResult] = []
|
||||
for wechat_id, friend_id in conversations:
|
||||
results.append(
|
||||
await sync_wechat_messages(
|
||||
session, client, store, wechat_id, friend_id
|
||||
)
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
async def _run_all_wechat(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
"""Full WeChat sequence: org/entities first, then per-conversation messages.
|
||||
|
||||
Conversations are discovered from the local ``customer`` mirror so the
|
||||
daily job is self-contained (no separate message kick needed).
|
||||
"""
|
||||
conversations = await _discover_conversations(session)
|
||||
return await sync_all_wechat(
|
||||
session, client, store, conversations=conversations
|
||||
)
|
||||
|
||||
|
||||
async def _run_all(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
"""Everything: full WeChat sequence + phone calls + SMS.
|
||||
|
||||
This is the entity the scheduler's daily job runs.
|
||||
"""
|
||||
results = await _run_all_wechat(session, client, store)
|
||||
results.append(await sync_phone_calls(session, client))
|
||||
results.append(await sync_sms(session, client))
|
||||
return results
|
||||
|
||||
|
||||
# Logical name -> runner. ``messages`` is the fast-cadence entry; ``all`` is
|
||||
# the daily full-sync entry. The single-entity names are exposed for manual
|
||||
# ``run_once("<entity>")`` triggers from the CLI / API.
|
||||
ENTITY_FUNCS: dict[
|
||||
str, Callable[[AsyncSession, YunkeClient, ObjectStore], Awaitable[list[SyncResult]]]
|
||||
] = {
|
||||
"departments": _run_departments,
|
||||
"accounts": _run_accounts,
|
||||
"friends": _run_friends,
|
||||
"groups": _run_groups,
|
||||
"voice_calls": _run_voice_calls,
|
||||
"phone_calls": _run_phone_calls,
|
||||
"sms": _run_sms,
|
||||
"messages": _run_messages,
|
||||
"all_wechat": _run_all_wechat,
|
||||
"all": _run_all,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON-line logger — one line per run, easy to ship to any aggregator
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _log_run(
|
||||
entity: str,
|
||||
results: list[SyncResult],
|
||||
*,
|
||||
started_at: dt.datetime,
|
||||
error: str | None = None,
|
||||
) -> None:
|
||||
"""Emit one JSON-line log entry summarising a finished run.
|
||||
|
||||
Always emits exactly one line, even on failure, so a log shipper can
|
||||
parse the stream uniformly. ``level`` is ``info`` on full success,
|
||||
``warning`` if any sub-result failed, ``error`` if the run itself blew
|
||||
up before producing results.
|
||||
"""
|
||||
finished_at = dt.datetime.now(dt.UTC)
|
||||
fetched = sum(r.fetched for r in results)
|
||||
upserted = sum(r.upserted for r in results)
|
||||
failed = [r.entity for r in results if r.status != "success"]
|
||||
|
||||
if error is not None:
|
||||
level = "error"
|
||||
elif failed:
|
||||
level = "warning"
|
||||
else:
|
||||
level = "info"
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"ts": finished_at.isoformat(),
|
||||
"event": "sync_run",
|
||||
"entity": entity,
|
||||
"started_at": started_at.isoformat(),
|
||||
"duration_sec": round((finished_at - started_at).total_seconds(), 3),
|
||||
"results": [
|
||||
{
|
||||
"entity": r.entity,
|
||||
"status": r.status,
|
||||
"fetched": r.fetched,
|
||||
"upserted": r.upserted,
|
||||
"channel": r.channel,
|
||||
"scope_id": r.scope_id,
|
||||
"error": r.error,
|
||||
}
|
||||
for r in results
|
||||
],
|
||||
"totals": {"fetched": fetched, "upserted": upserted, "count": len(results)},
|
||||
"failed_entities": failed,
|
||||
"level": level,
|
||||
}
|
||||
if error is not None:
|
||||
payload["error"] = error
|
||||
|
||||
line = json.dumps(payload, ensure_ascii=False, default=str)
|
||||
# Logger so operators can route with structured-log filters; the JSON body
|
||||
# is the whole message (no %-format) so a downstream parser sees one
|
||||
# object per line.
|
||||
logger.info(line)
|
||||
# Also echo to stdout for human/`docker logs` consumers; flush so a
|
||||
# piped reader sees it immediately.
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SyncScheduler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SyncScheduler:
|
||||
"""Periodic-sync driver wrapping an :class:`AsyncIOScheduler`.
|
||||
|
||||
Construction is cheap (no I/O); :meth:`start` opens the engines and
|
||||
registers jobs, :meth:`stop` tears them down. :meth:`run_once` is a
|
||||
standalone manual trigger that does NOT require :meth:`start` — useful
|
||||
from the CLI / API.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
full_sync_hour:
|
||||
Local-time hour at which the daily full sync fires (0..23). Default 2.
|
||||
message_interval_hours:
|
||||
Hours between incremental message syncs. Default 4.
|
||||
session_maker / engine:
|
||||
Optional injected async engine + session maker (tests inject an
|
||||
in-memory SQLite). When omitted, a production engine + maker are
|
||||
built from :attr:`Settings.db_url`.
|
||||
timezone:
|
||||
APScheduler timezone string for the daily cron. Defaults to the
|
||||
local machine timezone (matches operator expectation).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
full_sync_hour: int = 2,
|
||||
message_interval_hours: int = 4,
|
||||
engine: Any | None = None,
|
||||
session_maker: async_sessionmaker[AsyncSession] | None = None,
|
||||
timezone: str | None = None,
|
||||
max_conversations_per_run: int = DEFAULT_MAX_CONVERSATIONS_PER_RUN,
|
||||
) -> None:
|
||||
self.full_sync_hour = full_sync_hour
|
||||
self.message_interval_hours = message_interval_hours
|
||||
self.timezone = timezone
|
||||
self.max_conversations_per_run = max_conversations_per_run
|
||||
|
||||
# Engine ownership flag — if the caller injected an engine, they own
|
||||
# its lifecycle (a test will dispose it). Otherwise we own + dispose
|
||||
# on :meth:`stop`.
|
||||
self._owns_engine = engine is None
|
||||
if engine is not None:
|
||||
self._engine = engine
|
||||
self._session_maker = session_maker or async_sessionmaker(
|
||||
engine, expire_on_commit=False
|
||||
)
|
||||
else:
|
||||
settings = get_settings()
|
||||
self._engine = create_async_engine(
|
||||
settings.db_url, pool_pre_ping=True, future=True
|
||||
)
|
||||
self._session_maker = async_sessionmaker(
|
||||
self._engine, expire_on_commit=False
|
||||
)
|
||||
|
||||
self._scheduler: AsyncIOScheduler | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Open the engine, create tables if missing, register jobs, start.
|
||||
|
||||
Idempotent: calling :meth:`start` twice without an intervening
|
||||
:meth:`stop` is a no-op (the second call logs and returns).
|
||||
"""
|
||||
if self._scheduler is not None:
|
||||
logger.warning("SyncScheduler.start() called twice; ignoring.")
|
||||
return
|
||||
|
||||
# Create tables if missing (mirrors scripts.smoke_wechat). Cheap on a
|
||||
# warm DB (a no-op CREATE that already-exists). Lets the scheduler
|
||||
# boot on a fresh container without a separate migration step.
|
||||
async with self._engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
sched = AsyncIOScheduler(timezone=self.timezone)
|
||||
sched.add_job(
|
||||
self._safe_run,
|
||||
CronTrigger(hour=self.full_sync_hour, timezone=self.timezone),
|
||||
args=["all"],
|
||||
id="yunke_full_sync",
|
||||
replace_existing=True,
|
||||
coalesce=True,
|
||||
max_instances=1,
|
||||
)
|
||||
sched.add_job(
|
||||
self._safe_run,
|
||||
IntervalTrigger(hours=self.message_interval_hours),
|
||||
args=["messages"],
|
||||
id="yunke_message_sync",
|
||||
replace_existing=True,
|
||||
coalesce=True,
|
||||
max_instances=1,
|
||||
)
|
||||
sched.start()
|
||||
self._scheduler = sched
|
||||
logger.info(
|
||||
"SyncScheduler started: full sync daily at hour=%s, messages every %sh",
|
||||
self.full_sync_hour,
|
||||
self.message_interval_hours,
|
||||
)
|
||||
|
||||
async def stop(self, *, wait: bool = True) -> None:
|
||||
"""Stop the scheduler and (if owned) dispose the engine."""
|
||||
if self._scheduler is not None:
|
||||
self._scheduler.shutdown(wait=wait)
|
||||
self._scheduler = None
|
||||
if self._owns_engine:
|
||||
await self._engine.dispose()
|
||||
|
||||
async def run_once(self, entity: str = "all") -> list[SyncResult]:
|
||||
"""Run one sync for ``entity`` (e.g. ``"friends"``, ``"all"``).
|
||||
|
||||
Does NOT require :meth:`start` — opens its own client + session for
|
||||
the run and tears them down before returning. Returns the
|
||||
orchestrator's :class:`SyncResult` list (one per entity / per
|
||||
conversation). Raises ``ValueError`` on an unknown entity name so
|
||||
the CLI can surface usage cleanly.
|
||||
"""
|
||||
if entity not in ENTITY_FUNCS:
|
||||
raise ValueError(
|
||||
f"unknown entity: {entity!r}. "
|
||||
f"Known: {sorted(ENTITY_FUNCS)}"
|
||||
)
|
||||
# _safe_run handles the client build + try/except + logging; surface
|
||||
# the results it produced (possibly an empty list on hard failure).
|
||||
return await self._safe_run(entity, log=True)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internals
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _safe_run(
|
||||
self, entity: str, *, log: bool = True
|
||||
) -> list[SyncResult]:
|
||||
"""Run one entity, swallowing + logging any unexpected exception.
|
||||
|
||||
This is the failure-policy wrapper that protects the scheduler loop:
|
||||
a raised exception becomes a logged error and an empty result list,
|
||||
and the scheduler keeps its next appointment.
|
||||
"""
|
||||
started_at = dt.datetime.now(dt.UTC)
|
||||
client: YunkeClient | None = None
|
||||
results: list[SyncResult] = []
|
||||
try:
|
||||
client = await _build_client_from_env()
|
||||
async with self._session_maker() as session:
|
||||
runner = ENTITY_FUNCS[entity]
|
||||
results = await runner(session, client, ObjectStore())
|
||||
await session.commit()
|
||||
except Exception as exc: # noqa: BLE001 — scheduler MUST NOT crash
|
||||
logger.exception("sync %s failed", entity)
|
||||
if log:
|
||||
_log_run(entity, results, started_at=started_at, error=repr(exc))
|
||||
return results
|
||||
|
||||
if log:
|
||||
_log_run(entity, results, started_at=started_at)
|
||||
return results
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Client factory (env-driven; session-cookie mode preferred)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _build_client_from_env() -> YunkeClient:
|
||||
"""Construct + authenticate a :class:`YunkeClient` from env / ``.env``.
|
||||
|
||||
Lifts non-empty auth values from :func:`app.config.get_settings` (which
|
||||
loads ``.env``) into ``os.environ`` first, then delegates to
|
||||
:func:`scripts.smoke_wechat.build_client_from_env` so the precedence
|
||||
(session-cookie mode wins; pcLogin fallback) stays in one place.
|
||||
|
||||
In login mode, performs the actual ``await client.login()`` (real RSA +
|
||||
``pcLogin``); in session-cookie mode the client is already authenticated
|
||||
and no network call is made here.
|
||||
|
||||
Raises ``RuntimeError`` if no auth env is configured at all.
|
||||
"""
|
||||
# Imported lazily so a misconfigured ``scripts`` module never breaks
|
||||
# import of :mod:`app.sync.scheduler` (e.g. in unit tests that inject
|
||||
# their own engine / client).
|
||||
from scripts.smoke_wechat import (
|
||||
_push_settings_into_environ,
|
||||
build_client_from_env,
|
||||
)
|
||||
|
||||
_push_settings_into_environ()
|
||||
client = build_client_from_env()
|
||||
if client is None:
|
||||
raise RuntimeError(
|
||||
"no yunke auth configured: set YK_USERID+YK_COMPANY (preferred) "
|
||||
"or YUNKE_USERNAME+YUNKE_PASSWORD in .env / environment."
|
||||
)
|
||||
if not client.logged_in:
|
||||
# pcLogin mode — actually log in now. May raise; the caller's
|
||||
# _safe_run wrapper will catch + log.
|
||||
await client.login()
|
||||
return client
|
||||
Reference in New Issue
Block a user