feat: scheduler + CLI + docker-compose + README
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.
Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
(departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
all_wechat/all).
- Conversations for the message run are discovered from the local customer
mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).
Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
— long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.
Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.
Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).
Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
@@ -0,0 +1,25 @@
|
||||
# Build artifacts we never want inside the image.
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.egg-info/
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
|
||||
# Local DBs / scratch files (the container should use the mysql service).
|
||||
*.db
|
||||
*.sqlite
|
||||
*.sqlite3
|
||||
smoke.db
|
||||
|
||||
# Local env — the docker-compose ``environment:`` block injects runtime env.
|
||||
.env
|
||||
|
||||
# Local media storage (mounted as a volume in docker-compose).
|
||||
media/
|
||||
|
||||
# Editor / OS noise.
|
||||
.vscode/
|
||||
.idea/
|
||||
.DS_Store
|
||||
@@ -0,0 +1,35 @@
|
||||
# Backend image: FastAPI + APScheduler + Alembic + scrapers.
|
||||
#
|
||||
# Keeps the image small (python:3.12-slim) and the layer cache hot by copying
|
||||
# only pyproject.toml first and installing deps before the source lands. The
|
||||
# same image serves both the API (uvicorn) and the scheduler
|
||||
# (``python -m app.sync.cli scheduler``) — the docker-compose ``command:`` key
|
||||
# selects which entrypoint runs.
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Build-time / runtime deps for asyncmy (compiled C extension) + httpx +
|
||||
# cryptography.``libmariadb`` is what asyncmy links against on Debian.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
libmariadb-dev \
|
||||
libffi-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies first (cache-friendly: only re-installs when
|
||||
# pyproject.toml changes). The project itself is installed in editable-like
|
||||
# mode by copying the package in afterwards.
|
||||
COPY pyproject.toml ./
|
||||
COPY app/__init__.py app/__init__.py
|
||||
COPY app/ ./app/
|
||||
|
||||
# Install the package + dev dependencies (the dev extras include aiosqlite +
|
||||
# pytest, which are tiny and convenient for in-container smoke checks). Pin
|
||||
# the install to the local source so the package import path resolves.
|
||||
RUN pip install --no-cache-dir -e ".[dev]"
|
||||
|
||||
# Default command — overridden per-service in docker-compose. The API service
|
||||
# runs uvicorn; the scheduler service runs the CLI.
|
||||
CMD ["uvicorn", "app.api.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
@@ -0,0 +1,310 @@
|
||||
"""Command-line entry point for running syncs + the scheduler (M6, Part B).
|
||||
|
||||
Three invocations::
|
||||
|
||||
# One-shot full sync (every entity) into the configured DB:
|
||||
python -m app.sync.cli sync
|
||||
|
||||
# One-shot single-entity sync (departments / friends / messages / ...):
|
||||
python -m app.sync.cli sync --entity friends
|
||||
|
||||
# Long-running scheduler (APScheduler; full sync daily + messages every 4h):
|
||||
python -m app.sync.cli scheduler
|
||||
|
||||
Auth + DB
|
||||
---------
|
||||
Auth comes from env / ``.env`` via the same helpers the smoke script uses
|
||||
(:func:`scripts.smoke_wechat._push_settings_into_environ` +
|
||||
:func:`scripts.smoke_wechat.build_client_from_env`) — session-cookie mode
|
||||
(``YK_USERID`` + ``YK_COMPANY``) is preferred; pcLogin mode is wired but
|
||||
deferred.
|
||||
|
||||
The DB is taken from :attr:`Settings.db_url` (MySQL in prod, SQLite in dev /
|
||||
tests). Tables are created if missing (``Base.metadata.create_all``) on every
|
||||
command so a fresh checkout works without a separate migration step — the
|
||||
CLI is intentionally idempotent / bootstrappy that way. Real schema evolution
|
||||
goes through Alembic; this is a safety net for the operator.
|
||||
|
||||
Output
|
||||
------
|
||||
Progress is printed as JSON lines (one per entity) so the output is both
|
||||
human-skimmable and machine-parseable. ``--quiet`` suppresses the per-run
|
||||
JSON lines; ``--help`` lists the entities.
|
||||
|
||||
Exit codes: ``0`` success, ``1`` exception during run, ``2`` bad CLI usage /
|
||||
missing auth.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import signal
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import models # noqa: F401 (register tables on Base.metadata)
|
||||
from app.db.base import Base
|
||||
from app.sync.scheduler import ENTITY_FUNCS, SyncScheduler
|
||||
|
||||
logger = logging.getLogger("app.sync.cli")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# argparse setup
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="python -m app.sync.cli",
|
||||
description=(
|
||||
"Run a one-shot yunke sync or start the periodic scheduler. "
|
||||
"Auth + DB come from .env (session-cookie mode preferred)."
|
||||
),
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True, metavar="<command>")
|
||||
|
||||
# ``sync`` ----------------------------------------------------------------
|
||||
p_sync = sub.add_parser(
|
||||
"sync",
|
||||
help="Run a one-shot sync (all entities by default).",
|
||||
description=(
|
||||
"Run a one-shot sync into the configured DB. Without --entity, "
|
||||
"runs the full set (org tree + messages + phone calls + SMS). "
|
||||
"With --entity <name>, runs just that one."
|
||||
),
|
||||
)
|
||||
p_sync.add_argument(
|
||||
"--entity",
|
||||
choices=sorted(ENTITY_FUNCS),
|
||||
default="all",
|
||||
help="Entity to sync (default: all).",
|
||||
)
|
||||
p_sync.add_argument(
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="Suppress per-run JSON progress lines (errors still logged).",
|
||||
)
|
||||
|
||||
# ``scheduler`` -----------------------------------------------------------
|
||||
p_sched = sub.add_parser(
|
||||
"scheduler",
|
||||
help="Start the APScheduler (long-running).",
|
||||
description=(
|
||||
"Start the APScheduler loop: full sync daily at FULL_SYNC_HOUR "
|
||||
"(default 02:00 local), incremental message sync every "
|
||||
"MESSAGE_INTERVAL_HOURS (default 4). Runs forever until "
|
||||
"interrupted (Ctrl-C / SIGTERM)."
|
||||
),
|
||||
)
|
||||
p_sched.add_argument(
|
||||
"--full-sync-hour",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Local-time hour (0-23) for the daily full sync. "
|
||||
"Defaults to FULL_SYNC_HOUR env var or 2."
|
||||
),
|
||||
)
|
||||
p_sched.add_argument(
|
||||
"--message-interval-hours",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Hours between incremental message syncs. "
|
||||
"Defaults to MESSAGE_INTERVAL_HOURS env var or 4."
|
||||
),
|
||||
)
|
||||
p_sched.add_argument(
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="Suppress per-run JSON progress lines (errors still logged).",
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Command implementations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _cmd_sync(args: argparse.Namespace) -> int:
|
||||
"""Run a one-shot sync and print a per-entity JSON-line summary."""
|
||||
settings = get_settings()
|
||||
|
||||
# Create tables if missing (idempotent). Lets a fresh container boot
|
||||
# without a separate migration step.
|
||||
engine, session_maker = await _init_db(settings.db_url)
|
||||
try:
|
||||
scheduler = SyncScheduler(
|
||||
engine=engine, session_maker=session_maker
|
||||
)
|
||||
if args.quiet:
|
||||
# The --quiet flag only governs *our* explicit log echo; the
|
||||
# orchestrator's own per-entity logger.exception stays on so
|
||||
# failures are never silent.
|
||||
logging.getLogger("app.sync").setLevel(logging.WARNING)
|
||||
results = await scheduler.run_once(args.entity)
|
||||
_print_sync_summary(args.entity, results)
|
||||
# Exit non-zero only if every result failed; a partial failure is
|
||||
# still useful work delivered and should be exit 0 (the JSON line
|
||||
# surfaces which entities failed).
|
||||
any_success = any(r.status == "success" for r in results)
|
||||
return 0 if any_success or not results else 1
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def _cmd_scheduler(args: argparse.Namespace) -> int:
|
||||
"""Start the long-running scheduler and block until interrupted."""
|
||||
# Env-var fallbacks so docker-compose / .env can drive the cadence
|
||||
# without re-editing the compose command.
|
||||
full_sync_hour = args.full_sync_hour
|
||||
if full_sync_hour is None:
|
||||
full_sync_hour = _env_int("FULL_SYNC_HOUR", default=2)
|
||||
message_interval_hours = args.message_interval_hours
|
||||
if message_interval_hours is None:
|
||||
message_interval_hours = _env_int("MESSAGE_INTERVAL_HOURS", default=4)
|
||||
|
||||
if not (0 <= full_sync_hour <= 23):
|
||||
print(
|
||||
f"full-sync-hour must be 0..23, got {full_sync_hour}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
scheduler = SyncScheduler(
|
||||
full_sync_hour=full_sync_hour,
|
||||
message_interval_hours=message_interval_hours,
|
||||
)
|
||||
await scheduler.start()
|
||||
print(
|
||||
f"[scheduler] running: full sync daily at hour {full_sync_hour}, "
|
||||
f"messages every {message_interval_hours}h. Ctrl-C to stop.",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
# Block forever until SIGINT / SIGTERM. AsyncIOScheduler runs jobs on
|
||||
# its own event loop thread; we just need to keep the process alive.
|
||||
stop_event = asyncio.Event()
|
||||
|
||||
def _signal_handler(*_: Any) -> None:
|
||||
print("\n[scheduler] stop signal received, shutting down...", flush=True)
|
||||
stop_event.set()
|
||||
|
||||
# KeyboardInterrupt is delivered as SIGINT on POSIX, but on Windows
|
||||
# the convention is a Ctrl-C event; loop.add_signal_handler isn't
|
||||
# implemented there. Install both: signal.signal for Windows + the
|
||||
# asyncio version where available.
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
try:
|
||||
asyncio.get_running_loop().add_signal_handler(sig, _signal_handler)
|
||||
except (NotImplementedError, RuntimeError):
|
||||
# Windows / non-main thread — fall back to the stdlib handler.
|
||||
signal.signal(sig, _signal_handler)
|
||||
|
||||
try:
|
||||
await stop_event.wait()
|
||||
finally:
|
||||
await scheduler.stop()
|
||||
return 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _init_db(db_url: str) -> tuple[Any, Any]:
|
||||
"""Create the async engine + session maker and ensure tables exist.
|
||||
|
||||
Returns ``(engine, session_maker)``. The caller owns disposal. Tables
|
||||
are created with ``Base.metadata.create_all`` — idempotent, so this is
|
||||
safe to call on a warm DB.
|
||||
"""
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
engine = create_async_engine(db_url, pool_pre_ping=True, future=True)
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
session_maker = async_sessionmaker(engine, expire_on_commit=False)
|
||||
return engine, session_maker
|
||||
|
||||
|
||||
def _env_int(name: str, *, default: int) -> int:
|
||||
"""Read an int env var; fall back to ``default`` on missing / unparseable."""
|
||||
import os
|
||||
|
||||
raw = os.environ.get(name, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
print(
|
||||
f"[cli] {name}={raw!r} is not an int; using default {default}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return default
|
||||
|
||||
|
||||
def _print_sync_summary(entity: str, results: list[Any]) -> None:
|
||||
"""Print a compact human summary of a finished run (after the JSON lines)."""
|
||||
if not results:
|
||||
print(f"[sync] {entity}: no results (run failed before any entity completed)")
|
||||
return
|
||||
fetched = sum(r.fetched for r in results)
|
||||
upserted = sum(r.upserted for r in results)
|
||||
failed = [r.entity for r in results if r.status != "success"]
|
||||
line = (
|
||||
f"[sync] {entity}: {len(results)} sub-run(s), "
|
||||
f"fetched={fetched}, upserted={upserted}"
|
||||
)
|
||||
if failed:
|
||||
line += f", FAILED={failed}"
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""CLI entry point. Returns a process exit code."""
|
||||
# Configure logging so the scheduler's JSON line + the orchestrator's
|
||||
# exception tracebacks are visible. Quiet flag is applied per-command
|
||||
# after parsing.
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
|
||||
parser = _build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
if args.command == "sync":
|
||||
return asyncio.run(_cmd_sync(args))
|
||||
if args.command == "scheduler":
|
||||
return asyncio.run(_cmd_scheduler(args))
|
||||
except KeyboardInterrupt:
|
||||
# So `python -m app.sync.cli scheduler` exits cleanly on Ctrl-C.
|
||||
return 0
|
||||
except ValueError as exc:
|
||||
# Bad entity name or auth misconfig surfaced as a ValueError —
|
||||
# print a clear message instead of a traceback.
|
||||
print(f"[cli] {exc}", file=sys.stderr)
|
||||
return 2
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.exception("cli command failed")
|
||||
print(f"[cli] {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# argparse with required=True subparsers won't get here, but keep the
|
||||
# lint happy.
|
||||
parser.print_help(sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,501 @@
|
||||
"""APScheduler-based periodic sync for the yunke data mirror (M6, Part A).
|
||||
|
||||
Wraps the orchestrator's ``sync_*`` functions behind a long-running scheduler
|
||||
so the data mirror stays fresh without manual kicks. Two paces:
|
||||
|
||||
* **Daily full sync** at a configurable hour (default ``02:00`` local time,
|
||||
off-hours) — re-scrapes the org tree, accounts, friends, groups, voice
|
||||
calls, plus phone calls and SMS (all entities that have no upstream
|
||||
``since`` cursor and therefore rely on idempotent re-upsert).
|
||||
* **Incremental message sync** more frequently (default every **4 hours**).
|
||||
Messages DO have a real ``[last_timestamp, now]`` window (see
|
||||
:func:`app.sync.orchestrator.sync_wechat_messages`), so a fast cadence
|
||||
catches new chats promptly without re-pulling history. The conversation
|
||||
list is discovered from the already-mirrored ``customer`` table.
|
||||
|
||||
Auth: the scheduler builds a :class:`YunkeClient` per run from env (honoring
|
||||
``.env`` via :func:`scripts.smoke_wechat._push_settings_into_environ`).
|
||||
Session-cookie mode (``YK_USERID`` + ``YK_COMPANY`` [+ ``YK_COOKIE``]) is
|
||||
preferred and is the only mode exercised in production today; pcLogin mode
|
||||
(``YUNKE_USERNAME`` + ``YUNKE_PASSWORD``) is wired but deferred (see README
|
||||
"Current limitations").
|
||||
|
||||
Failure policy: every run is wrapped in a ``try/except`` that logs the error
|
||||
and lets the scheduler keep its next appointment. A single broken entity
|
||||
(scraper 500, network blip) does NOT bring down the loop. Per-entity
|
||||
failures are also captured by the orchestrator as ``status='failed'``
|
||||
``SyncRun`` rows.
|
||||
|
||||
Threading: :class:`SyncScheduler` is built on :class:`AsyncIOScheduler` and
|
||||
is intended to be ``await``-ed inside a long-running asyncio process (the
|
||||
CLI's ``scheduler`` command — see :mod:`app.sync.cli`). ``run_once`` is
|
||||
also awaitable for manual / API-triggered syncs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.cron import CronTrigger
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import models # noqa: F401 (register tables on Base.metadata)
|
||||
from app.db.base import Base
|
||||
from app.session.yunke_client import YunkeClient
|
||||
from app.storage.object_store import ObjectStore
|
||||
from app.sync.orchestrator import (
|
||||
SyncResult,
|
||||
sync_all_wechat,
|
||||
sync_departments,
|
||||
sync_phone_calls,
|
||||
sync_sms,
|
||||
sync_wechat_accounts,
|
||||
sync_wechat_friends,
|
||||
sync_wechat_groups,
|
||||
sync_wechat_messages,
|
||||
sync_wechat_voice_calls,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
__all__ = ["SyncScheduler", "ENTITY_FUNCS"]
|
||||
|
||||
# Cap on the number of conversations one message-sync run will pull. Bounds
|
||||
# runtime + upstream request volume per run (the orchestrator paces each
|
||||
# request, so 200 conversations × N pages each can still take a while).
|
||||
DEFAULT_MAX_CONVERSATIONS_PER_RUN = 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Conversation discovery (drives the incremental message sync)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _discover_conversations(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
limit: int = DEFAULT_MAX_CONVERSATIONS_PER_RUN,
|
||||
) -> list[tuple[str, str]]:
|
||||
"""Return distinct ``(owner_account_id, friend_id)`` WeChat pairs to sync.
|
||||
|
||||
Source: the local ``customer`` mirror (where
|
||||
``channel='wechat'`` and ``owner_account_id`` is non-empty). Reading the
|
||||
already-mirrored table — instead of the upstream ``talkerList`` endpoint
|
||||
— keeps the discovery call off the risk-control budget and makes the
|
||||
incremental run purely a function of the customers we already know
|
||||
about. Order is unspecified; ``limit`` caps work per run.
|
||||
"""
|
||||
from app.db.models import Customer
|
||||
|
||||
stmt = (
|
||||
sa.select(Customer.owner_account_id, Customer.friend_id)
|
||||
.where(Customer.channel == "wechat")
|
||||
.where(Customer.owner_account_id.is_not(None))
|
||||
.where(Customer.owner_account_id != "")
|
||||
.distinct()
|
||||
.limit(limit)
|
||||
)
|
||||
rows = (await session.execute(stmt)).all()
|
||||
return [
|
||||
(owner, friend_id)
|
||||
for owner, friend_id in rows
|
||||
if owner and friend_id
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entity registry — name -> async callable(session, client, store) -> list[SyncResult]
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# One entry per logical "thing the scheduler can run". Each callable returns a
|
||||
# list of SyncResult (one for the entity, or one per conversation for the
|
||||
# message entry). Keeping this as a registry means run_once("friends") and the
|
||||
# scheduler's daily job can share the same dispatch.
|
||||
|
||||
|
||||
async def _run_departments(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_departments(session, client)]
|
||||
|
||||
|
||||
async def _run_accounts(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_accounts(session, client)]
|
||||
|
||||
|
||||
async def _run_friends(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_friends(session, client)]
|
||||
|
||||
|
||||
async def _run_groups(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_groups(session, client)]
|
||||
|
||||
|
||||
async def _run_voice_calls(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_wechat_voice_calls(session, client)]
|
||||
|
||||
|
||||
async def _run_phone_calls(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_phone_calls(session, client)]
|
||||
|
||||
|
||||
async def _run_sms(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
return [await sync_sms(session, client)]
|
||||
|
||||
|
||||
async def _run_messages(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
"""Incremental message sync across all locally-known WeChat conversations."""
|
||||
conversations = await _discover_conversations(session)
|
||||
results: list[SyncResult] = []
|
||||
for wechat_id, friend_id in conversations:
|
||||
results.append(
|
||||
await sync_wechat_messages(
|
||||
session, client, store, wechat_id, friend_id
|
||||
)
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
async def _run_all_wechat(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
"""Full WeChat sequence: org/entities first, then per-conversation messages.
|
||||
|
||||
Conversations are discovered from the local ``customer`` mirror so the
|
||||
daily job is self-contained (no separate message kick needed).
|
||||
"""
|
||||
conversations = await _discover_conversations(session)
|
||||
return await sync_all_wechat(
|
||||
session, client, store, conversations=conversations
|
||||
)
|
||||
|
||||
|
||||
async def _run_all(
|
||||
session: AsyncSession, client: YunkeClient, store: ObjectStore
|
||||
) -> list[SyncResult]:
|
||||
"""Everything: full WeChat sequence + phone calls + SMS.
|
||||
|
||||
This is the entity the scheduler's daily job runs.
|
||||
"""
|
||||
results = await _run_all_wechat(session, client, store)
|
||||
results.append(await sync_phone_calls(session, client))
|
||||
results.append(await sync_sms(session, client))
|
||||
return results
|
||||
|
||||
|
||||
# Logical name -> runner. ``messages`` is the fast-cadence entry; ``all`` is
|
||||
# the daily full-sync entry. The single-entity names are exposed for manual
|
||||
# ``run_once("<entity>")`` triggers from the CLI / API.
|
||||
ENTITY_FUNCS: dict[
|
||||
str, Callable[[AsyncSession, YunkeClient, ObjectStore], Awaitable[list[SyncResult]]]
|
||||
] = {
|
||||
"departments": _run_departments,
|
||||
"accounts": _run_accounts,
|
||||
"friends": _run_friends,
|
||||
"groups": _run_groups,
|
||||
"voice_calls": _run_voice_calls,
|
||||
"phone_calls": _run_phone_calls,
|
||||
"sms": _run_sms,
|
||||
"messages": _run_messages,
|
||||
"all_wechat": _run_all_wechat,
|
||||
"all": _run_all,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON-line logger — one line per run, easy to ship to any aggregator
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _log_run(
|
||||
entity: str,
|
||||
results: list[SyncResult],
|
||||
*,
|
||||
started_at: dt.datetime,
|
||||
error: str | None = None,
|
||||
) -> None:
|
||||
"""Emit one JSON-line log entry summarising a finished run.
|
||||
|
||||
Always emits exactly one line, even on failure, so a log shipper can
|
||||
parse the stream uniformly. ``level`` is ``info`` on full success,
|
||||
``warning`` if any sub-result failed, ``error`` if the run itself blew
|
||||
up before producing results.
|
||||
"""
|
||||
finished_at = dt.datetime.now(dt.UTC)
|
||||
fetched = sum(r.fetched for r in results)
|
||||
upserted = sum(r.upserted for r in results)
|
||||
failed = [r.entity for r in results if r.status != "success"]
|
||||
|
||||
if error is not None:
|
||||
level = "error"
|
||||
elif failed:
|
||||
level = "warning"
|
||||
else:
|
||||
level = "info"
|
||||
|
||||
payload: dict[str, Any] = {
|
||||
"ts": finished_at.isoformat(),
|
||||
"event": "sync_run",
|
||||
"entity": entity,
|
||||
"started_at": started_at.isoformat(),
|
||||
"duration_sec": round((finished_at - started_at).total_seconds(), 3),
|
||||
"results": [
|
||||
{
|
||||
"entity": r.entity,
|
||||
"status": r.status,
|
||||
"fetched": r.fetched,
|
||||
"upserted": r.upserted,
|
||||
"channel": r.channel,
|
||||
"scope_id": r.scope_id,
|
||||
"error": r.error,
|
||||
}
|
||||
for r in results
|
||||
],
|
||||
"totals": {"fetched": fetched, "upserted": upserted, "count": len(results)},
|
||||
"failed_entities": failed,
|
||||
"level": level,
|
||||
}
|
||||
if error is not None:
|
||||
payload["error"] = error
|
||||
|
||||
line = json.dumps(payload, ensure_ascii=False, default=str)
|
||||
# Logger so operators can route with structured-log filters; the JSON body
|
||||
# is the whole message (no %-format) so a downstream parser sees one
|
||||
# object per line.
|
||||
logger.info(line)
|
||||
# Also echo to stdout for human/`docker logs` consumers; flush so a
|
||||
# piped reader sees it immediately.
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SyncScheduler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SyncScheduler:
|
||||
"""Periodic-sync driver wrapping an :class:`AsyncIOScheduler`.
|
||||
|
||||
Construction is cheap (no I/O); :meth:`start` opens the engines and
|
||||
registers jobs, :meth:`stop` tears them down. :meth:`run_once` is a
|
||||
standalone manual trigger that does NOT require :meth:`start` — useful
|
||||
from the CLI / API.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
full_sync_hour:
|
||||
Local-time hour at which the daily full sync fires (0..23). Default 2.
|
||||
message_interval_hours:
|
||||
Hours between incremental message syncs. Default 4.
|
||||
session_maker / engine:
|
||||
Optional injected async engine + session maker (tests inject an
|
||||
in-memory SQLite). When omitted, a production engine + maker are
|
||||
built from :attr:`Settings.db_url`.
|
||||
timezone:
|
||||
APScheduler timezone string for the daily cron. Defaults to the
|
||||
local machine timezone (matches operator expectation).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
full_sync_hour: int = 2,
|
||||
message_interval_hours: int = 4,
|
||||
engine: Any | None = None,
|
||||
session_maker: async_sessionmaker[AsyncSession] | None = None,
|
||||
timezone: str | None = None,
|
||||
max_conversations_per_run: int = DEFAULT_MAX_CONVERSATIONS_PER_RUN,
|
||||
) -> None:
|
||||
self.full_sync_hour = full_sync_hour
|
||||
self.message_interval_hours = message_interval_hours
|
||||
self.timezone = timezone
|
||||
self.max_conversations_per_run = max_conversations_per_run
|
||||
|
||||
# Engine ownership flag — if the caller injected an engine, they own
|
||||
# its lifecycle (a test will dispose it). Otherwise we own + dispose
|
||||
# on :meth:`stop`.
|
||||
self._owns_engine = engine is None
|
||||
if engine is not None:
|
||||
self._engine = engine
|
||||
self._session_maker = session_maker or async_sessionmaker(
|
||||
engine, expire_on_commit=False
|
||||
)
|
||||
else:
|
||||
settings = get_settings()
|
||||
self._engine = create_async_engine(
|
||||
settings.db_url, pool_pre_ping=True, future=True
|
||||
)
|
||||
self._session_maker = async_sessionmaker(
|
||||
self._engine, expire_on_commit=False
|
||||
)
|
||||
|
||||
self._scheduler: AsyncIOScheduler | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Open the engine, create tables if missing, register jobs, start.
|
||||
|
||||
Idempotent: calling :meth:`start` twice without an intervening
|
||||
:meth:`stop` is a no-op (the second call logs and returns).
|
||||
"""
|
||||
if self._scheduler is not None:
|
||||
logger.warning("SyncScheduler.start() called twice; ignoring.")
|
||||
return
|
||||
|
||||
# Create tables if missing (mirrors scripts.smoke_wechat). Cheap on a
|
||||
# warm DB (a no-op CREATE that already-exists). Lets the scheduler
|
||||
# boot on a fresh container without a separate migration step.
|
||||
async with self._engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
sched = AsyncIOScheduler(timezone=self.timezone)
|
||||
sched.add_job(
|
||||
self._safe_run,
|
||||
CronTrigger(hour=self.full_sync_hour, timezone=self.timezone),
|
||||
args=["all"],
|
||||
id="yunke_full_sync",
|
||||
replace_existing=True,
|
||||
coalesce=True,
|
||||
max_instances=1,
|
||||
)
|
||||
sched.add_job(
|
||||
self._safe_run,
|
||||
IntervalTrigger(hours=self.message_interval_hours),
|
||||
args=["messages"],
|
||||
id="yunke_message_sync",
|
||||
replace_existing=True,
|
||||
coalesce=True,
|
||||
max_instances=1,
|
||||
)
|
||||
sched.start()
|
||||
self._scheduler = sched
|
||||
logger.info(
|
||||
"SyncScheduler started: full sync daily at hour=%s, messages every %sh",
|
||||
self.full_sync_hour,
|
||||
self.message_interval_hours,
|
||||
)
|
||||
|
||||
async def stop(self, *, wait: bool = True) -> None:
|
||||
"""Stop the scheduler and (if owned) dispose the engine."""
|
||||
if self._scheduler is not None:
|
||||
self._scheduler.shutdown(wait=wait)
|
||||
self._scheduler = None
|
||||
if self._owns_engine:
|
||||
await self._engine.dispose()
|
||||
|
||||
async def run_once(self, entity: str = "all") -> list[SyncResult]:
|
||||
"""Run one sync for ``entity`` (e.g. ``"friends"``, ``"all"``).
|
||||
|
||||
Does NOT require :meth:`start` — opens its own client + session for
|
||||
the run and tears them down before returning. Returns the
|
||||
orchestrator's :class:`SyncResult` list (one per entity / per
|
||||
conversation). Raises ``ValueError`` on an unknown entity name so
|
||||
the CLI can surface usage cleanly.
|
||||
"""
|
||||
if entity not in ENTITY_FUNCS:
|
||||
raise ValueError(
|
||||
f"unknown entity: {entity!r}. "
|
||||
f"Known: {sorted(ENTITY_FUNCS)}"
|
||||
)
|
||||
# _safe_run handles the client build + try/except + logging; surface
|
||||
# the results it produced (possibly an empty list on hard failure).
|
||||
return await self._safe_run(entity, log=True)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internals
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def _safe_run(
|
||||
self, entity: str, *, log: bool = True
|
||||
) -> list[SyncResult]:
|
||||
"""Run one entity, swallowing + logging any unexpected exception.
|
||||
|
||||
This is the failure-policy wrapper that protects the scheduler loop:
|
||||
a raised exception becomes a logged error and an empty result list,
|
||||
and the scheduler keeps its next appointment.
|
||||
"""
|
||||
started_at = dt.datetime.now(dt.UTC)
|
||||
client: YunkeClient | None = None
|
||||
results: list[SyncResult] = []
|
||||
try:
|
||||
client = await _build_client_from_env()
|
||||
async with self._session_maker() as session:
|
||||
runner = ENTITY_FUNCS[entity]
|
||||
results = await runner(session, client, ObjectStore())
|
||||
await session.commit()
|
||||
except Exception as exc: # noqa: BLE001 — scheduler MUST NOT crash
|
||||
logger.exception("sync %s failed", entity)
|
||||
if log:
|
||||
_log_run(entity, results, started_at=started_at, error=repr(exc))
|
||||
return results
|
||||
|
||||
if log:
|
||||
_log_run(entity, results, started_at=started_at)
|
||||
return results
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Client factory (env-driven; session-cookie mode preferred)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _build_client_from_env() -> YunkeClient:
|
||||
"""Construct + authenticate a :class:`YunkeClient` from env / ``.env``.
|
||||
|
||||
Lifts non-empty auth values from :func:`app.config.get_settings` (which
|
||||
loads ``.env``) into ``os.environ`` first, then delegates to
|
||||
:func:`scripts.smoke_wechat.build_client_from_env` so the precedence
|
||||
(session-cookie mode wins; pcLogin fallback) stays in one place.
|
||||
|
||||
In login mode, performs the actual ``await client.login()`` (real RSA +
|
||||
``pcLogin``); in session-cookie mode the client is already authenticated
|
||||
and no network call is made here.
|
||||
|
||||
Raises ``RuntimeError`` if no auth env is configured at all.
|
||||
"""
|
||||
# Imported lazily so a misconfigured ``scripts`` module never breaks
|
||||
# import of :mod:`app.sync.scheduler` (e.g. in unit tests that inject
|
||||
# their own engine / client).
|
||||
from scripts.smoke_wechat import (
|
||||
_push_settings_into_environ,
|
||||
build_client_from_env,
|
||||
)
|
||||
|
||||
_push_settings_into_environ()
|
||||
client = build_client_from_env()
|
||||
if client is None:
|
||||
raise RuntimeError(
|
||||
"no yunke auth configured: set YK_USERID+YK_COMPANY (preferred) "
|
||||
"or YUNKE_USERNAME+YUNKE_PASSWORD in .env / environment."
|
||||
)
|
||||
if not client.logged_in:
|
||||
# pcLogin mode — actually log in now. May raise; the caller's
|
||||
# _safe_run wrapper will catch + log.
|
||||
await client.login()
|
||||
return client
|
||||
@@ -0,0 +1,229 @@
|
||||
"""Tests for ``python -m app.sync.cli`` (M6 Part B).
|
||||
|
||||
The CLI's network / DB / scheduler-loop paths are exercised manually (they
|
||||
need real env + a running DB). These tests verify the testable surface:
|
||||
|
||||
* ``--help`` and the subcommand parsers build without error.
|
||||
* The ``sync`` command dispatches into :class:`SyncScheduler.run_once` with
|
||||
the right entity and prints a summary line.
|
||||
* An unknown subcommand / bad ``--entity`` exits non-zero.
|
||||
* Env-var fallbacks (``FULL_SYNC_HOUR`` / ``MESSAGE_INTERVAL_HOURS``) work.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from app.sync import cli as cli_module
|
||||
from app.sync.orchestrator import SyncResult
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Argument parser
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parser_help_lists_both_subcommands(capsys):
|
||||
"""``--help`` mentions both ``sync`` and ``scheduler``."""
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
cli_module._build_parser().parse_args(["--help"])
|
||||
assert exc.value.code == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "sync" in out
|
||||
assert "scheduler" in out
|
||||
|
||||
|
||||
def test_parser_requires_subcommand():
|
||||
"""No subcommand -> argparse errors (exit code 2)."""
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
cli_module._build_parser().parse_args([])
|
||||
assert exc.value.code == 2
|
||||
|
||||
|
||||
def test_parser_sync_entity_choices_match_registry():
|
||||
"""``--entity`` choices mirror the scheduler registry."""
|
||||
parser = cli_module._build_parser()
|
||||
args = parser.parse_args(["sync", "--entity", "friends"])
|
||||
assert args.command == "sync"
|
||||
assert args.entity == "friends"
|
||||
|
||||
|
||||
def test_parser_rejects_unknown_entity():
|
||||
"""An unknown entity name is rejected by argparse (exit code 2)."""
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
cli_module._build_parser().parse_args(["sync", "--entity", "nope"])
|
||||
assert exc.value.code == 2
|
||||
|
||||
|
||||
def test_parser_scheduler_flags_default_none():
|
||||
"""``--full-sync-hour`` and ``--message-interval-hours`` default to None (env-fallback path)."""
|
||||
args = cli_module._build_parser().parse_args(["scheduler"])
|
||||
assert args.full_sync_hour is None
|
||||
assert args.message_interval_hours is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# main() — exit codes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_main_returns_2_on_unknown_subcommand(capsys):
|
||||
"""An unknown subcommand surfaces as exit code 2 (argparse)."""
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
cli_module.main(["bogus"])
|
||||
assert exc.value.code == 2
|
||||
|
||||
|
||||
def test_main_value_error_becomes_exit_2(monkeypatch, capsys):
|
||||
"""A ValueError surfaced from a command (e.g. unknown entity) is exit 2."""
|
||||
async def _boom(_args):
|
||||
raise ValueError("unknown entity: does_not_exist")
|
||||
monkeypatch.setattr(cli_module, "_cmd_sync", _boom)
|
||||
rc = cli_module.main(["sync", "--entity", "friends"])
|
||||
assert rc == 2
|
||||
err = capsys.readouterr().err
|
||||
assert "unknown entity" in err
|
||||
|
||||
|
||||
def test_main_generic_exception_becomes_exit_1(monkeypatch):
|
||||
"""Any non-ValueError exception is exit 1 (don't print misleading usage)."""
|
||||
async def _boom(_args):
|
||||
raise RuntimeError("kaboom")
|
||||
monkeypatch.setattr(cli_module, "_cmd_sync", _boom)
|
||||
rc = cli_module.main(["sync"])
|
||||
assert rc == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _cmd_sync — dispatch + summary printing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cmd_sync_dispatches_run_once_and_prints_summary(monkeypatch, capsys):
|
||||
"""``sync`` builds a SyncScheduler, calls run_once(<entity>), prints a summary."""
|
||||
# ``_cmd_sync`` is async — wrap a coroutine-returning stub via patch.
|
||||
fake_results = [
|
||||
SyncResult(entity="wechat_friend", status="success", fetched=3, upserted=3),
|
||||
SyncResult(entity="message", status="success", fetched=10, upserted=10, channel="wechat"),
|
||||
]
|
||||
|
||||
async def _fake_init_db(_db_url):
|
||||
engine = MagicMock()
|
||||
engine.dispose = AsyncMock()
|
||||
sm = MagicMock()
|
||||
return engine, sm
|
||||
|
||||
scheduler_mock = MagicMock()
|
||||
scheduler_mock.run_once = AsyncMock(return_value=list(fake_results))
|
||||
|
||||
def _ctor(*, engine, session_maker):
|
||||
# Returned to _cmd_sync which then calls .run_once() on it.
|
||||
return scheduler_mock
|
||||
|
||||
monkeypatch.setattr(cli_module, "_init_db", _fake_init_db)
|
||||
monkeypatch.setattr(cli_module, "SyncScheduler", _ctor)
|
||||
|
||||
args = cli_module._build_parser().parse_args(["sync", "--entity", "friends"])
|
||||
rc = asyncio.run(cli_module._cmd_sync(args))
|
||||
|
||||
assert rc == 0
|
||||
scheduler_mock.run_once.assert_awaited_once_with("friends")
|
||||
out = capsys.readouterr().out
|
||||
assert "[sync] friends" in out
|
||||
assert "fetched=13" in out
|
||||
assert "upserted=13" in out
|
||||
|
||||
|
||||
def test_cmd_sync_partial_failure_is_exit_0(monkeypatch):
|
||||
"""If at least one sub-result succeeded, exit 0 (partial delivery)."""
|
||||
fake_results = [
|
||||
SyncResult(entity="department", status="success", fetched=2, upserted=2),
|
||||
SyncResult(entity="wechat_friend", status="failed", fetched=0, upserted=0, error="x"),
|
||||
]
|
||||
async def _fake_init_db(_db_url):
|
||||
engine = MagicMock()
|
||||
engine.dispose = AsyncMock()
|
||||
return engine, MagicMock()
|
||||
scheduler_mock = MagicMock()
|
||||
scheduler_mock.run_once = AsyncMock(return_value=list(fake_results))
|
||||
monkeypatch.setattr(cli_module, "_init_db", _fake_init_db)
|
||||
monkeypatch.setattr(cli_module, "SyncScheduler", lambda **kw: scheduler_mock)
|
||||
|
||||
args = cli_module._build_parser().parse_args(["sync"])
|
||||
rc = asyncio.run(cli_module._cmd_sync(args))
|
||||
assert rc == 0
|
||||
|
||||
|
||||
def test_cmd_sync_all_failed_is_exit_1(monkeypatch):
|
||||
"""Every sub-run failed -> exit 1 (nothing useful delivered)."""
|
||||
fake_results = [
|
||||
SyncResult(entity="department", status="failed", fetched=0, upserted=0, error="x"),
|
||||
SyncResult(entity="wechat_friend", status="failed", fetched=0, upserted=0, error="y"),
|
||||
]
|
||||
async def _fake_init_db(_db_url):
|
||||
engine = MagicMock()
|
||||
engine.dispose = AsyncMock()
|
||||
return engine, MagicMock()
|
||||
scheduler_mock = MagicMock()
|
||||
scheduler_mock.run_once = AsyncMock(return_value=list(fake_results))
|
||||
monkeypatch.setattr(cli_module, "_init_db", _fake_init_db)
|
||||
monkeypatch.setattr(cli_module, "SyncScheduler", lambda **kw: scheduler_mock)
|
||||
|
||||
args = cli_module._build_parser().parse_args(["sync"])
|
||||
rc = asyncio.run(cli_module._cmd_sync(args))
|
||||
assert rc == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Env-var fallbacks for the scheduler command
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_env_int_returns_default_when_missing(monkeypatch):
|
||||
"""Missing env var -> default."""
|
||||
monkeypatch.delenv("FULL_SYNC_HOUR", raising=False)
|
||||
assert cli_module._env_int("FULL_SYNC_HOUR", default=42) == 42
|
||||
|
||||
|
||||
def test_env_int_parses_value(monkeypatch):
|
||||
"""A clean int env var is parsed."""
|
||||
monkeypatch.setenv("FULL_SYNC_HOUR", "13")
|
||||
assert cli_module._env_int("FULL_SYNC_HOUR", default=2) == 13
|
||||
|
||||
|
||||
def test_env_int_falls_back_on_garbage(monkeypatch, capsys):
|
||||
"""Non-int env var falls back to default + prints a warning."""
|
||||
monkeypatch.setenv("FULL_SYNC_HOUR", "two")
|
||||
assert cli_module._env_int("FULL_SYNC_HOUR", default=2) == 2
|
||||
err = capsys.readouterr().err
|
||||
assert "FULL_SYNC_HOUR" in err
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _init_db — create_all is invoked
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_init_db_creates_tables():
|
||||
"""_init_db returns an engine + session maker and runs create_all.
|
||||
|
||||
Uses an in-memory SQLite engine via the real SQLAlchemy path so the
|
||||
create_all call is exercised end-to-end (not mocked).
|
||||
"""
|
||||
from sqlalchemy import text
|
||||
|
||||
engine, sm = await cli_module._init_db("sqlite+aiosqlite:///:memory:")
|
||||
try:
|
||||
# Tables created: customer is one of the registered tables.
|
||||
from app.db.base import Base
|
||||
|
||||
assert "customer" in Base.metadata.tables
|
||||
# engine + session_maker both usable (no exception = pass).
|
||||
async with sm() as s:
|
||||
await s.execute(text("SELECT 1"))
|
||||
finally:
|
||||
await engine.dispose()
|
||||
@@ -0,0 +1,389 @@
|
||||
"""Tests for app.sync.scheduler — periodic-sync wrapper (M6 Part A).
|
||||
|
||||
These tests do NOT hit the network and do NOT start a real APScheduler loop
|
||||
(that path is exercised manually via ``python -m app.sync.cli scheduler``).
|
||||
They verify the testable surface of :class:`SyncScheduler`:
|
||||
|
||||
* ``run_once`` dispatches by entity name to the right orchestrator function.
|
||||
* ``run_once("<unknown>")`` raises ``ValueError`` with the known-entity list.
|
||||
* ``_discover_conversations`` reads distinct WeChat pairs from the DB.
|
||||
* ``_log_run`` emits one JSON line per run with sensible totals + level.
|
||||
* The daily vs. message-every-N-hours schedule is registered with the right
|
||||
triggers when :meth:`start` is called (using a fake scheduler fixture).
|
||||
* Failures inside a runner are swallowed and logged, not re-raised.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from app.db.base import Base
|
||||
from app.db.models import Customer
|
||||
from app.sync import scheduler as scheduler_module
|
||||
from app.sync.scheduler import (
|
||||
ENTITY_FUNCS,
|
||||
SyncScheduler,
|
||||
_discover_conversations,
|
||||
_log_run,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _make_engine():
|
||||
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
return engine
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def engine():
|
||||
eng = await _make_engine()
|
||||
try:
|
||||
yield eng
|
||||
finally:
|
||||
await eng.dispose()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def session_maker(engine):
|
||||
return async_sessionmaker(engine, expire_on_commit=False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ENTITY_FUNCS registry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_entity_funcs_has_expected_names():
|
||||
"""The registry exposes the documented manual-trigger entity names."""
|
||||
expected = {
|
||||
"departments",
|
||||
"accounts",
|
||||
"friends",
|
||||
"groups",
|
||||
"voice_calls",
|
||||
"phone_calls",
|
||||
"sms",
|
||||
"messages",
|
||||
"all_wechat",
|
||||
"all",
|
||||
}
|
||||
assert expected <= set(ENTITY_FUNCS)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _discover_conversations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_discover_conversations_returns_distinct_wechat_pairs(session_maker):
|
||||
"""Distinct (owner, friend_id) for channel='wechat' only; non-wechat + nulls dropped."""
|
||||
async with session_maker() as s:
|
||||
s.add_all(
|
||||
[
|
||||
Customer(
|
||||
friend_id="f1",
|
||||
channel="wechat",
|
||||
owner_account_id="wx_a",
|
||||
),
|
||||
Customer(
|
||||
friend_id="f2",
|
||||
channel="wechat",
|
||||
owner_account_id="wx_a",
|
||||
),
|
||||
Customer(
|
||||
friend_id="f3",
|
||||
channel="wechat",
|
||||
owner_account_id="wx_b",
|
||||
),
|
||||
# Different channel — should be excluded.
|
||||
Customer(
|
||||
friend_id="p1@s.whatsapp.net",
|
||||
channel="whatsapp",
|
||||
owner_account_id="wx_a",
|
||||
),
|
||||
# Null owner — should be excluded.
|
||||
Customer(
|
||||
friend_id="orphan",
|
||||
channel="wechat",
|
||||
owner_account_id=None,
|
||||
),
|
||||
]
|
||||
)
|
||||
await s.commit()
|
||||
|
||||
pairs = await _discover_conversations(s)
|
||||
assert sorted(pairs) == sorted([("wx_a", "f1"), ("wx_a", "f2"), ("wx_b", "f3")])
|
||||
|
||||
|
||||
async def test_discover_conversations_respects_limit(session_maker):
|
||||
"""``limit`` caps the work per message-sync run."""
|
||||
async with session_maker() as s:
|
||||
for i in range(10):
|
||||
s.add(
|
||||
Customer(
|
||||
friend_id=f"f{i}",
|
||||
channel="wechat",
|
||||
owner_account_id="wx_a",
|
||||
)
|
||||
)
|
||||
await s.commit()
|
||||
|
||||
pairs = await _discover_conversations(s, limit=3)
|
||||
assert len(pairs) == 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _log_run
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_log_run_emits_one_json_line(capsys, caplog):
|
||||
"""One JSON line, level=info when all sub-runs succeed."""
|
||||
results = [
|
||||
_mk_result("department", "success", 5, 5),
|
||||
_mk_result("wechat_friend", "success", 3, 3),
|
||||
]
|
||||
with caplog.at_level(logging.INFO, logger="app.sync.scheduler"):
|
||||
_log_run("all", results, started_at=dt.datetime.now(dt.UTC))
|
||||
out = capsys.readouterr().out.strip().splitlines()
|
||||
assert len(out) == 1
|
||||
payload = json.loads(out[0])
|
||||
assert payload["event"] == "sync_run"
|
||||
assert payload["entity"] == "all"
|
||||
assert payload["level"] == "info"
|
||||
assert payload["totals"] == {"fetched": 8, "upserted": 8, "count": 2}
|
||||
assert payload["failed_entities"] == []
|
||||
assert len(payload["results"]) == 2
|
||||
|
||||
|
||||
def test_log_run_level_warning_when_any_failed(capsys):
|
||||
"""A failed sub-result bumps the level to warning + lists the failed entity."""
|
||||
results = [
|
||||
_mk_result("department", "success", 5, 5),
|
||||
_mk_result("wechat_friend", "failed", 0, 0, error="boom"),
|
||||
]
|
||||
_log_run("all", results, started_at=dt.datetime.now(dt.UTC))
|
||||
payload = json.loads(capsys.readouterr().out.strip())
|
||||
assert payload["level"] == "warning"
|
||||
assert payload["failed_entities"] == ["wechat_friend"]
|
||||
assert payload["totals"]["fetched"] == 5
|
||||
|
||||
|
||||
def test_log_run_level_error_on_top_level_failure(capsys):
|
||||
"""A top-level error (no results at all) -> level=error + error field."""
|
||||
_log_run("all", [], started_at=dt.datetime.now(dt.UTC), error="AuthFailed")
|
||||
payload = json.loads(capsys.readouterr().out.strip())
|
||||
assert payload["level"] == "error"
|
||||
assert payload["error"] == "AuthFailed"
|
||||
assert payload["totals"]["count"] == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SyncScheduler.run_once — dispatch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_run_once_unknown_entity_raises(engine, session_maker):
|
||||
"""An unknown entity name raises ValueError listing the known entities."""
|
||||
sched = SyncScheduler(engine=engine, session_maker=session_maker)
|
||||
with pytest.raises(ValueError, match="unknown entity"):
|
||||
await sched.run_once("does_not_exist")
|
||||
|
||||
|
||||
async def test_run_once_dispatches_to_entity_runner(engine, session_maker, monkeypatch):
|
||||
"""run_once('friends') calls _run_friends (which calls sync_wechat_friends)."""
|
||||
# Stub _build_client_from_env so no network / login happens.
|
||||
fake_client = MagicMock()
|
||||
monkeypatch.setattr(
|
||||
scheduler_module,
|
||||
"_build_client_from_env",
|
||||
AsyncMock(return_value=fake_client),
|
||||
)
|
||||
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
async def fake_runner(session, client, store):
|
||||
captured["session"] = session
|
||||
captured["client"] = client
|
||||
captured["store"] = store
|
||||
return [_mk_result("wechat_friend", "success", 7, 7)]
|
||||
|
||||
# Patch the registry entry, not the orchestrator function.
|
||||
monkeypatch.setitem(ENTITY_FUNCS, "friends", fake_runner)
|
||||
|
||||
sched = SyncScheduler(engine=engine, session_maker=session_maker)
|
||||
results = await sched.run_once("friends")
|
||||
|
||||
assert len(results) == 1
|
||||
assert results[0].entity == "wechat_friend"
|
||||
assert results[0].fetched == 7
|
||||
assert captured["client"] is fake_client
|
||||
assert captured["store"] is not None
|
||||
|
||||
|
||||
async def test_run_once_swallows_runner_exception(engine, session_maker, monkeypatch, capsys):
|
||||
"""A blown-up runner is logged + swallowed (returns []; scheduler keeps going)."""
|
||||
fake_client = MagicMock()
|
||||
monkeypatch.setattr(
|
||||
scheduler_module,
|
||||
"_build_client_from_env",
|
||||
AsyncMock(return_value=fake_client),
|
||||
)
|
||||
|
||||
async def boom(session, client, store):
|
||||
raise RuntimeError("upstream 500")
|
||||
|
||||
monkeypatch.setitem(ENTITY_FUNCS, "groups", boom)
|
||||
|
||||
sched = SyncScheduler(engine=engine, session_maker=session_maker)
|
||||
# Must NOT raise.
|
||||
results = await sched.run_once("groups")
|
||||
assert results == []
|
||||
out = capsys.readouterr().out
|
||||
assert "sync_run" in out
|
||||
payload = json.loads(out.strip().splitlines()[-1])
|
||||
assert payload["level"] == "error"
|
||||
assert "upstream 500" in payload["error"]
|
||||
|
||||
|
||||
async def test_run_once_swallows_client_build_failure(engine, session_maker, monkeypatch):
|
||||
"""If auth env is missing -> _build_client_from_env raises -> logged + empty list."""
|
||||
monkeypatch.setattr(
|
||||
scheduler_module,
|
||||
"_build_client_from_env",
|
||||
AsyncMock(side_effect=RuntimeError("no auth configured")),
|
||||
)
|
||||
sched = SyncScheduler(engine=engine, session_maker=session_maker)
|
||||
results = await sched.run_once("all")
|
||||
assert results == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SyncScheduler.start — job registration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_start_registers_two_jobs_with_expected_triggers(engine, session_maker):
|
||||
"""start() wires a daily CronTrigger (full) + an IntervalTrigger (messages)."""
|
||||
sched = SyncScheduler(
|
||||
full_sync_hour=3,
|
||||
message_interval_hours=6,
|
||||
engine=engine,
|
||||
session_maker=session_maker,
|
||||
)
|
||||
# Replace the real AsyncIOScheduler with a spy before start() creates it,
|
||||
# so we can assert on add_job without running the loop.
|
||||
spy = MagicMock()
|
||||
spy.add_job = MagicMock()
|
||||
with patch.object(scheduler_module, "AsyncIOScheduler", return_value=spy):
|
||||
await sched.start()
|
||||
|
||||
# Two jobs registered.
|
||||
assert spy.add_job.call_count == 2
|
||||
job_ids = {c.kwargs["id"] for c in spy.add_job.call_args_list}
|
||||
assert job_ids == {"yunke_full_sync", "yunke_message_sync"}
|
||||
|
||||
# Full sync -> CronTrigger(hour=3); messages -> IntervalTrigger(hours=6).
|
||||
# add_job was called as (func, trigger, kwargs...) so trigger is args[1].
|
||||
full_call = next(
|
||||
c for c in spy.add_job.call_args_list if c.kwargs["id"] == "yunke_full_sync"
|
||||
)
|
||||
full_trigger = full_call.args[1]
|
||||
# CronTrigger exposes the parsed schedule via str().
|
||||
assert str(full_trigger) == "cron[hour='3']"
|
||||
|
||||
msg_call = next(
|
||||
c for c in spy.add_job.call_args_list if c.kwargs["id"] == "yunke_message_sync"
|
||||
)
|
||||
interval = msg_call.args[1]
|
||||
# IntervalTrigger exposes its interval as a timedelta on .interval`.
|
||||
assert interval.interval.total_seconds() == 6 * 3600
|
||||
|
||||
# The full-sync job passes args=["all"] (so it hits _run_all).
|
||||
assert full_call.kwargs["args"] == ["all"]
|
||||
assert msg_call.kwargs["args"] == ["messages"]
|
||||
|
||||
# coalesce=True + max_instances=1 protects against overlap.
|
||||
for c in spy.add_job.call_args_list:
|
||||
assert c.kwargs["coalesce"] is True
|
||||
assert c.kwargs["max_instances"] == 1
|
||||
|
||||
|
||||
async def test_start_is_idempotent(engine, session_maker):
|
||||
"""A second start() is a no-op (does not register a second scheduler)."""
|
||||
sched = SyncScheduler(engine=engine, session_maker=session_maker)
|
||||
with patch.object(scheduler_module, "AsyncIOScheduler", return_value=MagicMock()):
|
||||
await sched.start()
|
||||
first = sched._scheduler
|
||||
await sched.start() # second call — ignored.
|
||||
assert sched._scheduler is first
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stop / teardown
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_stop_disposes_owned_engine():
|
||||
"""When the scheduler owns its engine, stop() disposes it."""
|
||||
# Patch create_async_engine so we don't try to connect to a real DB.
|
||||
fake_engine = MagicMock()
|
||||
fake_engine.dispose = AsyncMock()
|
||||
fake_engine.begin = MagicMock()
|
||||
# start() will call engine.begin() to create tables; fake it.
|
||||
class _Ctx:
|
||||
async def __aenter__(self):
|
||||
conn = MagicMock()
|
||||
conn.run_sync = AsyncMock()
|
||||
return conn
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
return None
|
||||
|
||||
fake_engine.begin.return_value = _Ctx()
|
||||
|
||||
with patch.object(scheduler_module, "create_async_engine", return_value=fake_engine):
|
||||
sched = SyncScheduler()
|
||||
assert sched._owns_engine is True
|
||||
await sched.stop()
|
||||
fake_engine.dispose.assert_awaited_once()
|
||||
|
||||
|
||||
async def test_stop_does_not_dispose_injected_engine(engine, session_maker):
|
||||
"""An engine injected by the caller is NOT disposed on stop()."""
|
||||
sched = SyncScheduler(engine=engine, session_maker=session_maker)
|
||||
assert sched._owns_engine is False
|
||||
await sched.stop() # must not dispose — caller (test fixture) owns it.
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _mk_result(
|
||||
entity: str, status: str, fetched: int, upserted: int, *, error: str | None = None):
|
||||
from app.sync.orchestrator import SyncResult
|
||||
|
||||
return SyncResult(
|
||||
entity=entity,
|
||||
status=status,
|
||||
fetched=fetched,
|
||||
upserted=upserted,
|
||||
error=error,
|
||||
)
|
||||
Reference in New Issue
Block a user