xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: scheduler + CLI + docker-compose + README

Browse Source 
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.

Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
  message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
  (departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
  all_wechat/all).
- Conversations for the message run are discovered from the local customer
  mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
  so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).

Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
  — long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
  mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.

Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
  image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
  reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
  SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
  backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.

Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).

Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-05 12:11:29 +08:00
parent c8311bb496
commit df29dd027a
11 changed files with 1933 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
yunke-server/tests/test_sync_cli.py
+229
View File
@@ -0,0 +1,229 @@
"""Tests for ``python -m app.sync.cli`` (M6 Part B).
The CLI's network / DB / scheduler-loop paths are exercised manually (they
need real env + a running DB). These tests verify the testable surface:
* ``--help`` and the subcommand parsers build without error.
* The ``sync`` command dispatches into :class:`SyncScheduler.run_once` with
the right entity and prints a summary line.
* An unknown subcommand / bad ``--entity`` exits non-zero.
* Env-var fallbacks (``FULL_SYNC_HOUR`` / ``MESSAGE_INTERVAL_HOURS``) work.
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from app.sync import cli as cli_module
from app.sync.orchestrator import SyncResult
# ---------------------------------------------------------------------------
# Argument parser
# ---------------------------------------------------------------------------
def test_parser_help_lists_both_subcommands(capsys):
"""``--help`` mentions both ``sync`` and ``scheduler``."""
with pytest.raises(SystemExit) as exc:
cli_module._build_parser().parse_args(["--help"])
assert exc.value.code == 0
out = capsys.readouterr().out
assert "sync" in out
assert "scheduler" in out
def test_parser_requires_subcommand():
"""No subcommand -> argparse errors (exit code 2)."""
with pytest.raises(SystemExit) as exc:
cli_module._build_parser().parse_args([])
assert exc.value.code == 2
def test_parser_sync_entity_choices_match_registry():
"""``--entity`` choices mirror the scheduler registry."""
parser = cli_module._build_parser()
args = parser.parse_args(["sync", "--entity", "friends"])
assert args.command == "sync"
assert args.entity == "friends"
def test_parser_rejects_unknown_entity():
"""An unknown entity name is rejected by argparse (exit code 2)."""
with pytest.raises(SystemExit) as exc:
cli_module._build_parser().parse_args(["sync", "--entity", "nope"])
assert exc.value.code == 2
def test_parser_scheduler_flags_default_none():
"""``--full-sync-hour`` and ``--message-interval-hours`` default to None (env-fallback path)."""
args = cli_module._build_parser().parse_args(["scheduler"])
assert args.full_sync_hour is None
assert args.message_interval_hours is None
# ---------------------------------------------------------------------------
# main() — exit codes
# ---------------------------------------------------------------------------
def test_main_returns_2_on_unknown_subcommand(capsys):
"""An unknown subcommand surfaces as exit code 2 (argparse)."""
with pytest.raises(SystemExit) as exc:
cli_module.main(["bogus"])
assert exc.value.code == 2
def test_main_value_error_becomes_exit_2(monkeypatch, capsys):
"""A ValueError surfaced from a command (e.g. unknown entity) is exit 2."""
async def _boom(_args):
raise ValueError("unknown entity: does_not_exist")
monkeypatch.setattr(cli_module, "_cmd_sync", _boom)
rc = cli_module.main(["sync", "--entity", "friends"])
assert rc == 2
err = capsys.readouterr().err
assert "unknown entity" in err
def test_main_generic_exception_becomes_exit_1(monkeypatch):
"""Any non-ValueError exception is exit 1 (don't print misleading usage)."""
async def _boom(_args):
raise RuntimeError("kaboom")
monkeypatch.setattr(cli_module, "_cmd_sync", _boom)
rc = cli_module.main(["sync"])
assert rc == 1
# ---------------------------------------------------------------------------
# _cmd_sync — dispatch + summary printing
# ---------------------------------------------------------------------------
def test_cmd_sync_dispatches_run_once_and_prints_summary(monkeypatch, capsys):
"""``sync`` builds a SyncScheduler, calls run_once(<entity>), prints a summary."""
# ``_cmd_sync`` is async — wrap a coroutine-returning stub via patch.
fake_results = [
SyncResult(entity="wechat_friend", status="success", fetched=3, upserted=3),
SyncResult(entity="message", status="success", fetched=10, upserted=10, channel="wechat"),
]
async def _fake_init_db(_db_url):
engine = MagicMock()
engine.dispose = AsyncMock()
sm = MagicMock()
return engine, sm
scheduler_mock = MagicMock()
scheduler_mock.run_once = AsyncMock(return_value=list(fake_results))
def _ctor(*, engine, session_maker):
# Returned to _cmd_sync which then calls .run_once() on it.
return scheduler_mock
monkeypatch.setattr(cli_module, "_init_db", _fake_init_db)
monkeypatch.setattr(cli_module, "SyncScheduler", _ctor)
args = cli_module._build_parser().parse_args(["sync", "--entity", "friends"])
rc = asyncio.run(cli_module._cmd_sync(args))
assert rc == 0
scheduler_mock.run_once.assert_awaited_once_with("friends")
out = capsys.readouterr().out
assert "[sync] friends" in out
assert "fetched=13" in out
assert "upserted=13" in out
def test_cmd_sync_partial_failure_is_exit_0(monkeypatch):
"""If at least one sub-result succeeded, exit 0 (partial delivery)."""
fake_results = [
SyncResult(entity="department", status="success", fetched=2, upserted=2),
SyncResult(entity="wechat_friend", status="failed", fetched=0, upserted=0, error="x"),
]
async def _fake_init_db(_db_url):
engine = MagicMock()
engine.dispose = AsyncMock()
return engine, MagicMock()
scheduler_mock = MagicMock()
scheduler_mock.run_once = AsyncMock(return_value=list(fake_results))
monkeypatch.setattr(cli_module, "_init_db", _fake_init_db)
monkeypatch.setattr(cli_module, "SyncScheduler", lambda **kw: scheduler_mock)
args = cli_module._build_parser().parse_args(["sync"])
rc = asyncio.run(cli_module._cmd_sync(args))
assert rc == 0
def test_cmd_sync_all_failed_is_exit_1(monkeypatch):
"""Every sub-run failed -> exit 1 (nothing useful delivered)."""
fake_results = [
SyncResult(entity="department", status="failed", fetched=0, upserted=0, error="x"),
SyncResult(entity="wechat_friend", status="failed", fetched=0, upserted=0, error="y"),
]
async def _fake_init_db(_db_url):
engine = MagicMock()
engine.dispose = AsyncMock()
return engine, MagicMock()
scheduler_mock = MagicMock()
scheduler_mock.run_once = AsyncMock(return_value=list(fake_results))
monkeypatch.setattr(cli_module, "_init_db", _fake_init_db)
monkeypatch.setattr(cli_module, "SyncScheduler", lambda **kw: scheduler_mock)
args = cli_module._build_parser().parse_args(["sync"])
rc = asyncio.run(cli_module._cmd_sync(args))
assert rc == 1
# ---------------------------------------------------------------------------
# Env-var fallbacks for the scheduler command
# ---------------------------------------------------------------------------
def test_env_int_returns_default_when_missing(monkeypatch):
"""Missing env var -> default."""
monkeypatch.delenv("FULL_SYNC_HOUR", raising=False)
assert cli_module._env_int("FULL_SYNC_HOUR", default=42) == 42
def test_env_int_parses_value(monkeypatch):
"""A clean int env var is parsed."""
monkeypatch.setenv("FULL_SYNC_HOUR", "13")
assert cli_module._env_int("FULL_SYNC_HOUR", default=2) == 13
def test_env_int_falls_back_on_garbage(monkeypatch, capsys):
"""Non-int env var falls back to default + prints a warning."""
monkeypatch.setenv("FULL_SYNC_HOUR", "two")
assert cli_module._env_int("FULL_SYNC_HOUR", default=2) == 2
err = capsys.readouterr().err
assert "FULL_SYNC_HOUR" in err
# ---------------------------------------------------------------------------
# _init_db — create_all is invoked
# ---------------------------------------------------------------------------
async def test_init_db_creates_tables():
"""_init_db returns an engine + session maker and runs create_all.
Uses an in-memory SQLite engine via the real SQLAlchemy path so the
create_all call is exercised end-to-end (not mocked).
"""
from sqlalchemy import text
engine, sm = await cli_module._init_db("sqlite+aiosqlite:///:memory:")
try:
# Tables created: customer is one of the registered tables.
from app.db.base import Base
assert "customer" in Base.metadata.tables
# engine + session_maker both usable (no exception = pass).
async with sm() as s:
await s.execute(text("SELECT 1"))
finally:
await engine.dispose()
yunke-server/tests/test_sync_scheduler.py
+389
View File
@@ -0,0 +1,389 @@
"""Tests for app.sync.scheduler — periodic-sync wrapper (M6 Part A).
These tests do NOT hit the network and do NOT start a real APScheduler loop
(that path is exercised manually via ``python -m app.sync.cli scheduler``).
They verify the testable surface of :class:`SyncScheduler`:
* ``run_once`` dispatches by entity name to the right orchestrator function.
* ``run_once("<unknown>")`` raises ``ValueError`` with the known-entity list.
* ``_discover_conversations`` reads distinct WeChat pairs from the DB.
* ``_log_run`` emits one JSON line per run with sensible totals + level.
* The daily vs. message-every-N-hours schedule is registered with the right
triggers when :meth:`start` is called (using a fake scheduler fixture).
* Failures inside a runner are swallowed and logged, not re-raised.
"""
from __future__ import annotations
import datetime as dt
import json
import logging
from collections.abc import AsyncIterator
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
import sqlalchemy as sa
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.db.base import Base
from app.db.models import Customer
from app.sync import scheduler as scheduler_module
from app.sync.scheduler import (
ENTITY_FUNCS,
SyncScheduler,
_discover_conversations,
_log_run,
)
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
async def _make_engine():
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
return engine
@pytest.fixture
async def engine():
eng = await _make_engine()
try:
yield eng
finally:
await eng.dispose()
@pytest.fixture
async def session_maker(engine):
return async_sessionmaker(engine, expire_on_commit=False)
# ---------------------------------------------------------------------------
# ENTITY_FUNCS registry
# ---------------------------------------------------------------------------
def test_entity_funcs_has_expected_names():
"""The registry exposes the documented manual-trigger entity names."""
expected = {
"departments",
"accounts",
"friends",
"groups",
"voice_calls",
"phone_calls",
"sms",
"messages",
"all_wechat",
"all",
}
assert expected <= set(ENTITY_FUNCS)
# ---------------------------------------------------------------------------
# _discover_conversations
# ---------------------------------------------------------------------------
async def test_discover_conversations_returns_distinct_wechat_pairs(session_maker):
"""Distinct (owner, friend_id) for channel='wechat' only; non-wechat + nulls dropped."""
async with session_maker() as s:
s.add_all(
[
Customer(
friend_id="f1",
channel="wechat",
owner_account_id="wx_a",
),
Customer(
friend_id="f2",
channel="wechat",
owner_account_id="wx_a",
),
Customer(
friend_id="f3",
channel="wechat",
owner_account_id="wx_b",
),
# Different channel — should be excluded.
Customer(
friend_id="p1@s.whatsapp.net",
channel="whatsapp",
owner_account_id="wx_a",
),
# Null owner — should be excluded.
Customer(
friend_id="orphan",
channel="wechat",
owner_account_id=None,
),
]
)
await s.commit()
pairs = await _discover_conversations(s)
assert sorted(pairs) == sorted([("wx_a", "f1"), ("wx_a", "f2"), ("wx_b", "f3")])
async def test_discover_conversations_respects_limit(session_maker):
"""``limit`` caps the work per message-sync run."""
async with session_maker() as s:
for i in range(10):
s.add(
Customer(
friend_id=f"f{i}",
channel="wechat",
owner_account_id="wx_a",
)
)
await s.commit()
pairs = await _discover_conversations(s, limit=3)
assert len(pairs) == 3
# ---------------------------------------------------------------------------
# _log_run
# ---------------------------------------------------------------------------
def test_log_run_emits_one_json_line(capsys, caplog):
"""One JSON line, level=info when all sub-runs succeed."""
results = [
_mk_result("department", "success", 5, 5),
_mk_result("wechat_friend", "success", 3, 3),
]
with caplog.at_level(logging.INFO, logger="app.sync.scheduler"):
_log_run("all", results, started_at=dt.datetime.now(dt.UTC))
out = capsys.readouterr().out.strip().splitlines()
assert len(out) == 1
payload = json.loads(out[0])
assert payload["event"] == "sync_run"
assert payload["entity"] == "all"
assert payload["level"] == "info"
assert payload["totals"] == {"fetched": 8, "upserted": 8, "count": 2}
assert payload["failed_entities"] == []
assert len(payload["results"]) == 2
def test_log_run_level_warning_when_any_failed(capsys):
"""A failed sub-result bumps the level to warning + lists the failed entity."""
results = [
_mk_result("department", "success", 5, 5),
_mk_result("wechat_friend", "failed", 0, 0, error="boom"),
]
_log_run("all", results, started_at=dt.datetime.now(dt.UTC))
payload = json.loads(capsys.readouterr().out.strip())
assert payload["level"] == "warning"
assert payload["failed_entities"] == ["wechat_friend"]
assert payload["totals"]["fetched"] == 5
def test_log_run_level_error_on_top_level_failure(capsys):
"""A top-level error (no results at all) -> level=error + error field."""
_log_run("all", [], started_at=dt.datetime.now(dt.UTC), error="AuthFailed")
payload = json.loads(capsys.readouterr().out.strip())
assert payload["level"] == "error"
assert payload["error"] == "AuthFailed"
assert payload["totals"]["count"] == 0
# ---------------------------------------------------------------------------
# SyncScheduler.run_once — dispatch
# ---------------------------------------------------------------------------
async def test_run_once_unknown_entity_raises(engine, session_maker):
"""An unknown entity name raises ValueError listing the known entities."""
sched = SyncScheduler(engine=engine, session_maker=session_maker)
with pytest.raises(ValueError, match="unknown entity"):
await sched.run_once("does_not_exist")
async def test_run_once_dispatches_to_entity_runner(engine, session_maker, monkeypatch):
"""run_once('friends') calls _run_friends (which calls sync_wechat_friends)."""
# Stub _build_client_from_env so no network / login happens.
fake_client = MagicMock()
monkeypatch.setattr(
scheduler_module,
"_build_client_from_env",
AsyncMock(return_value=fake_client),
)
captured: dict[str, Any] = {}
async def fake_runner(session, client, store):
captured["session"] = session
captured["client"] = client
captured["store"] = store
return [_mk_result("wechat_friend", "success", 7, 7)]
# Patch the registry entry, not the orchestrator function.
monkeypatch.setitem(ENTITY_FUNCS, "friends", fake_runner)
sched = SyncScheduler(engine=engine, session_maker=session_maker)
results = await sched.run_once("friends")
assert len(results) == 1
assert results[0].entity == "wechat_friend"
assert results[0].fetched == 7
assert captured["client"] is fake_client
assert captured["store"] is not None
async def test_run_once_swallows_runner_exception(engine, session_maker, monkeypatch, capsys):
"""A blown-up runner is logged + swallowed (returns []; scheduler keeps going)."""
fake_client = MagicMock()
monkeypatch.setattr(
scheduler_module,
"_build_client_from_env",
AsyncMock(return_value=fake_client),
)
async def boom(session, client, store):
raise RuntimeError("upstream 500")
monkeypatch.setitem(ENTITY_FUNCS, "groups", boom)
sched = SyncScheduler(engine=engine, session_maker=session_maker)
# Must NOT raise.
results = await sched.run_once("groups")
assert results == []
out = capsys.readouterr().out
assert "sync_run" in out
payload = json.loads(out.strip().splitlines()[-1])
assert payload["level"] == "error"
assert "upstream 500" in payload["error"]
async def test_run_once_swallows_client_build_failure(engine, session_maker, monkeypatch):
"""If auth env is missing -> _build_client_from_env raises -> logged + empty list."""
monkeypatch.setattr(
scheduler_module,
"_build_client_from_env",
AsyncMock(side_effect=RuntimeError("no auth configured")),
)
sched = SyncScheduler(engine=engine, session_maker=session_maker)
results = await sched.run_once("all")
assert results == []
# ---------------------------------------------------------------------------
# SyncScheduler.start — job registration
# ---------------------------------------------------------------------------
async def test_start_registers_two_jobs_with_expected_triggers(engine, session_maker):
"""start() wires a daily CronTrigger (full) + an IntervalTrigger (messages)."""
sched = SyncScheduler(
full_sync_hour=3,
message_interval_hours=6,
engine=engine,
session_maker=session_maker,
)
# Replace the real AsyncIOScheduler with a spy before start() creates it,
# so we can assert on add_job without running the loop.
spy = MagicMock()
spy.add_job = MagicMock()
with patch.object(scheduler_module, "AsyncIOScheduler", return_value=spy):
await sched.start()
# Two jobs registered.
assert spy.add_job.call_count == 2
job_ids = {c.kwargs["id"] for c in spy.add_job.call_args_list}
assert job_ids == {"yunke_full_sync", "yunke_message_sync"}
# Full sync -> CronTrigger(hour=3); messages -> IntervalTrigger(hours=6).
# add_job was called as (func, trigger, kwargs...) so trigger is args[1].
full_call = next(
c for c in spy.add_job.call_args_list if c.kwargs["id"] == "yunke_full_sync"
)
full_trigger = full_call.args[1]
# CronTrigger exposes the parsed schedule via str().
assert str(full_trigger) == "cron[hour='3']"
msg_call = next(
c for c in spy.add_job.call_args_list if c.kwargs["id"] == "yunke_message_sync"
)
interval = msg_call.args[1]
# IntervalTrigger exposes its interval as a timedelta on .interval`.
assert interval.interval.total_seconds() == 6 * 3600
# The full-sync job passes args=["all"] (so it hits _run_all).
assert full_call.kwargs["args"] == ["all"]
assert msg_call.kwargs["args"] == ["messages"]
# coalesce=True + max_instances=1 protects against overlap.
for c in spy.add_job.call_args_list:
assert c.kwargs["coalesce"] is True
assert c.kwargs["max_instances"] == 1
async def test_start_is_idempotent(engine, session_maker):
"""A second start() is a no-op (does not register a second scheduler)."""
sched = SyncScheduler(engine=engine, session_maker=session_maker)
with patch.object(scheduler_module, "AsyncIOScheduler", return_value=MagicMock()):
await sched.start()
first = sched._scheduler
await sched.start() # second call — ignored.
assert sched._scheduler is first
# ---------------------------------------------------------------------------
# Stop / teardown
# ---------------------------------------------------------------------------
async def test_stop_disposes_owned_engine():
"""When the scheduler owns its engine, stop() disposes it."""
# Patch create_async_engine so we don't try to connect to a real DB.
fake_engine = MagicMock()
fake_engine.dispose = AsyncMock()
fake_engine.begin = MagicMock()
# start() will call engine.begin() to create tables; fake it.
class _Ctx:
async def __aenter__(self):
conn = MagicMock()
conn.run_sync = AsyncMock()
return conn
async def __aexit__(self, *a):
return None
fake_engine.begin.return_value = _Ctx()
with patch.object(scheduler_module, "create_async_engine", return_value=fake_engine):
sched = SyncScheduler()
assert sched._owns_engine is True
await sched.stop()
fake_engine.dispose.assert_awaited_once()
async def test_stop_does_not_dispose_injected_engine(engine, session_maker):
"""An engine injected by the caller is NOT disposed on stop()."""
sched = SyncScheduler(engine=engine, session_maker=session_maker)
assert sched._owns_engine is False
await sched.stop() # must not dispose — caller (test fixture) owns it.
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _mk_result(
entity: str, status: str, fetched: int, upserted: int, *, error: str | None = None):
from app.sync.orchestrator import SyncResult
return SyncResult(
entity=entity,
status=status,
fetched=fetched,
upserted=upserted,
error=error,
)