M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.
Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
(departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
all_wechat/all).
- Conversations for the message run are discovered from the local customer
mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).
Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
— long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.
Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.
Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).
Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.
Co-Authored-By: Claude <noreply@anthropic.com>
Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
isOnJob / departmentId / startdate / enddate + empty userId/salesname/
weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.
Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 2.7 — the coordination layer tying scrapers + repos together with
resumable, incremental sync.
app/sync/watermarks.py — SyncState CRUD keyed on (entity_type, channel,
scope_id): get_watermark / get_last_timestamp / get_last_version /
set_watermark (upsert). A partial advance (timestamp-only) does not clobber
the other cursor with NULL.
app/sync/orchestrator.py — the sync brain:
* sync_wechat_messages: time-window incremental. Reads last_timestamp for
(message, wechat, "<wechat_id>:<friend_id>"), queries [last_ts, now] via
scrape_wechat_messages, upserts each, best-effort downloads media, advances
the watermark to max(timestamp). A re-run with no new messages queries an
empty window and fetches nothing — the watermark advanced past.
* sync_{departments,wechat_accounts,wechat_friends,wechat_groups,
wechat_voice_calls}: full re-scrape + idempotent upsert (these list APIs
have no since-param; 'incremental' = re-scrape all, rely on PK idempotency).
Watermark records last_synced_at only.
* sync_all_wechat: coordinator running departments -> accounts -> friends ->
groups -> voice_calls, then per-conversation messages for an explicit
(wechat_id, friend_id) list. Auto-discovery deferred.
Each run logs a SyncRun row (started -> success/failed, counts, error).
Failure policy: a scraper that raises is caught, recorded as failed, and a
failed SyncResult is returned — the orchestrator does not re-raise, so one
broken entity does not abort the batch. Media-download failures are
best-effort. The orchestrator never commits; callers own the transaction.
Tests (TDD red -> green; in-memory SQLite; scrapers/repos monkeypatched):
- watermarks: set/get, upsert (same key twice -> 1 row), partial-advance
preserves the other cursor, defaults when absent.
- orchestrator messages: first run upserts 2 + advances watermark to T2;
second run scrape called with start==T2 and upserts 0 (incremental proof);
media failure does not abort; SyncRun success recorded.
- orchestrator friends: 3 records upserted, SyncRun success, watermark
last_synced_at set.
- failure policy: scraper raises -> SyncRun failed + failed result, no raise
(both friends and message paths).
- sync_all_wechat: 5 entities + 2 message runs in order, all success.
Suite: 149 -> 163 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念