feat: scheduler + CLI + docker-compose + README
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.
Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
(departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
all_wechat/all).
- Conversations for the message run are discovered from the local customer
mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).
Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
— long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.
Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.
Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).
Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
@@ -0,0 +1,177 @@
|
||||
# yunke — 云客 CRM 数据本地化系统
|
||||
|
||||
A self-hosted mirror of the yunke CRM (`https://crm2.yunkecn.com`): a Python
|
||||
backend that periodically scrapes WeChat / WhatsApp / phone-call / SMS data
|
||||
into MySQL + a local object store, plus a Vue SPA for browsing it. The
|
||||
system is read-only against the upstream CRM — every upstream call is a
|
||||
list / search endpoint; nothing writes back to yunke.
|
||||
|
||||
## What's here
|
||||
|
||||
```
|
||||
.
|
||||
├── yunke-server/ Python backend (FastAPI + APScheduler + scrapers + repos)
|
||||
├── yunkes/ Vue 3 + Element Plus frontend (Vite build → nginx)
|
||||
├── docker-compose.yml Five-service stack (mysql / redis / backend / scheduler / frontend)
|
||||
└── docs/ Design + plan documents
|
||||
```
|
||||
|
||||
## Architecture (1 minute version)
|
||||
|
||||
The backend has three concentric layers:
|
||||
|
||||
1. **Session layer** (`app/session/yunke_client.py`) — a single
|
||||
`YunkeClient` chokepoint that injects the auth headers, paces requests
|
||||
with a jittered rate limiter, backs off on risk-control signals (HTTP
|
||||
429 or 风控 body markers), and re-logs-in once on 401.
|
||||
2. **Scraper layer** (`app/scraper/*`) — one paginator + parser per
|
||||
upstream endpoint, yielding snake_case dicts. Pure-ish: takes a
|
||||
`YunkeClient`, yields records, never touches the DB.
|
||||
3. **Sync / persistence layer** (`app/sync/orchestrator.py` +
|
||||
`app/storage/repos.py`) — the orchestrator runs each entity's
|
||||
`scrape → upsert → advance watermark` loop, logs every run as a
|
||||
`SyncRun` row, and never re-raises a per-entity failure (one broken
|
||||
entity doesn't abort a batch). The repos do dialect-aware idempotent
|
||||
upserts (SQLite for tests, MySQL for prod) so a re-scrape of the same
|
||||
row is one refreshed row, not a duplicate.
|
||||
|
||||
Messages have a real incremental window (`[last_timestamp, now]`); every
|
||||
other entity has no upstream `since` param and relies on idempotent
|
||||
re-scrape. See the [design doc](docs/superpowers/specs/2026-08-01-yunke-data-mirror-design.md)
|
||||
for the full data dictionary + risk-control rationale.
|
||||
|
||||
## Quick start
|
||||
|
||||
### Dev mode (no Docker)
|
||||
|
||||
You need: Python 3.11+, Node 20+, and (optionally) a local MySQL. SQLite
|
||||
works for a quick smoke.
|
||||
|
||||
Backend:
|
||||
|
||||
```bash
|
||||
cd yunke-server
|
||||
python -m venv .venv
|
||||
.venv/Scripts/activate # Windows
|
||||
# source .venv/bin/activate # *nix
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Configure auth + DB (see .env.example). Session-cookie mode preferred:
|
||||
# YK_USERID=...
|
||||
# YK_COMPANY=...
|
||||
# YK_COOKIE="user=...; userToken=..."
|
||||
cp .env.example .env
|
||||
# edit .env
|
||||
|
||||
# Run the API:
|
||||
uvicorn app.api.main:app --reload
|
||||
```
|
||||
|
||||
Frontend (separate shell):
|
||||
|
||||
```bash
|
||||
cd yunkes
|
||||
npm install
|
||||
npm run dev # serves on :5173, proxies /api → :8000
|
||||
```
|
||||
|
||||
Open <http://localhost:5173>.
|
||||
|
||||
One-shot sync into the configured DB:
|
||||
|
||||
```bash
|
||||
cd yunke-server
|
||||
python -m app.sync.cli sync # everything
|
||||
python -m app.sync.cli sync --entity friends # one entity
|
||||
python -m app.sync.cli --help # full usage
|
||||
```
|
||||
|
||||
### Docker mode (full stack)
|
||||
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
That brings up:
|
||||
|
||||
| Service | Image / build | Role | Port |
|
||||
|-----------|---------------------|------------------------------------------------------------|------------|
|
||||
| `mysql` | `mysql:8.0` | Persistent data store | 3306 |
|
||||
| `redis` | `redis:7-alpine` | Cache / queue (future use; not yet wired into the app) | — |
|
||||
| `backend` | `yunke-server/` | FastAPI viewing API (uvicorn) | 8000 |
|
||||
| `scheduler` | `yunke-server/` | APScheduler loop — daily full sync + every-4h messages | — |
|
||||
| `frontend`| `yunkes/` | SPA served by nginx, reverse-proxies `/api` to backend:8000 | 5173 → 80 |
|
||||
|
||||
Open <http://localhost:5173>. The scheduler emits one JSON-line log per
|
||||
run; tail with `docker compose logs -f scheduler`.
|
||||
|
||||
## Configuration
|
||||
|
||||
All configuration is via environment / `.env` (the docker-compose stack
|
||||
passes the host `.env` through to the containers). See
|
||||
`yunke-server/.env.example` for the full list; the load-bearing ones:
|
||||
|
||||
| Variable | Purpose |
|
||||
|-----------------------|---------------------------------------------------------------------------|
|
||||
| `YK_USERID` | Existing-session mode (preferred). Pairs with `YK_COMPANY` (+ `YK_COOKIE`). |
|
||||
| `YK_COMPANY` | Existing-session mode. |
|
||||
| `YK_COOKIE` | Optional raw `Cookie:` header for session mode. |
|
||||
| `YUNKE_USERNAME` | pcLogin mode (deferred — see "Limitations"). Pairs with `YUNKE_PASSWORD`. |
|
||||
| `YUNKE_PASSWORD` | pcLogin mode. |
|
||||
| `DB_URL` | SQLAlchemy async URL. Prod: `mysql+asyncmy://...`. Dev: `sqlite+aiosqlite:///./smoke.db`. |
|
||||
| `REDIS_URL` | Redis URL (currently unused; reserved for future cache / queue). |
|
||||
| `MEDIA_DIR` | Where downloaded media blobs land (sha256-deduped). |
|
||||
| `ENCRYPTION_KEY` | Fernet key for encrypting stored credentials. Blank for tests only. |
|
||||
| `API_TOKEN` | Shared bearer token for the viewing API. Blank = auth disabled (local/intranet trust). |
|
||||
| `FULL_SYNC_HOUR` | Scheduler: local-time hour for the daily full sync (default `2`). |
|
||||
| `MESSAGE_INTERVAL_HOURS` | Scheduler: hours between incremental message syncs (default `4`). |
|
||||
|
||||
Auth precedence: session-cookie mode wins when both are set (the existing
|
||||
live session is always preferred over a fresh pcLogin).
|
||||
|
||||
## Running the scheduler manually
|
||||
|
||||
```bash
|
||||
cd yunke-server
|
||||
python -m app.sync.cli scheduler # uses FULL_SYNC_HOUR + MESSAGE_INTERVAL_HOURS
|
||||
python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6
|
||||
```
|
||||
|
||||
Entity names accepted by `sync --entity` and `run_once`:
|
||||
`departments`, `accounts`, `friends`, `groups`, `voice_calls`,
|
||||
`phone_calls`, `sms`, `messages`, `all_wechat`, `all` (default).
|
||||
|
||||
## Current limitations
|
||||
|
||||
- **pcLogin deferred.** Programmatic login (`getSecureKey` → `pcLogin`) is
|
||||
wired and exercises real RSA encryption, but the final `pcLogin` body
|
||||
field names are inferred and the live CRM has been observed rejecting
|
||||
them. **Use session-cookie mode** (`YK_USERID` + `YK_COMPANY` + the
|
||||
captured `YK_COOKIE`) until that's resolved. Capture the cookie/userid
|
||||
from a logged-in browser tab on `crm2.yunkecn.com`.
|
||||
- **WhatsApp voice / media deferred.** The WhatsApp scraper covers the
|
||||
friend list + chat records; voice-call and media-download paths for
|
||||
WhatsApp are not yet wired.
|
||||
- **Rate-limited pacing.** Request cadence is deliberately conservative
|
||||
(default ≈ one request every 1.0–1.8 s with jitter; exponential backoff
|
||||
on 风控 signals). A full first-time sync is therefore slow (hours for
|
||||
thousands of conversations). Tunable via `REQUEST_MIN_INTERVAL` /
|
||||
`REQUEST_JITTER` / `MAX_REQUESTS_PER_RUN` in `app/config.py` — loosen
|
||||
only after evidence that the CRM tolerates a faster pace.
|
||||
- **Conversation discovery is local-only.** The incremental message sync
|
||||
reads the conversation list from the already-mirrored `customer` table,
|
||||
not the upstream `talkerList` endpoint. A first-time-empty mirror will
|
||||
therefore not pull any messages until at least one full friend sync has
|
||||
populated customers.
|
||||
- **Redis is provisioned but unused.** The compose stack includes a Redis
|
||||
service for future cache / queue use; nothing in the app reads or writes
|
||||
it yet.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd yunke-server && .venv/Scripts/python.exe -m pytest -q # 323 tests
|
||||
cd yunkes && npm run build # clean production build
|
||||
cd yunke-server && python -m app.sync.cli --help # CLI usage
|
||||
docker compose config # validates the stack
|
||||
```
|
||||
Reference in New Issue
Block a user