# yunke — 云客 CRM 数据本地化系统 A self-hosted mirror of the yunke CRM (`https://crm2.yunkecn.com`): a Python backend that periodically scrapes WeChat / WhatsApp / phone-call / SMS data into MySQL + a local object store, plus a Vue SPA for browsing it. The system is read-only against the upstream CRM — every upstream call is a list / search endpoint; nothing writes back to yunke. ## What's here ``` . ├── yunke-server/ Python backend (FastAPI + APScheduler + scrapers + repos) ├── yunkes/ Vue 3 + Element Plus frontend (Vite build → nginx) ├── docker-compose.yml Five-service stack (mysql / redis / backend / scheduler / frontend) └── docs/ Design + plan documents ``` ## Architecture (1 minute version) The backend has three concentric layers: 1. **Session layer** (`app/session/yunke_client.py`) — a single `YunkeClient` chokepoint that injects the auth headers, paces requests with a jittered rate limiter, backs off on risk-control signals (HTTP 429 or 风控 body markers), and re-logs-in once on 401. 2. **Scraper layer** (`app/scraper/*`) — one paginator + parser per upstream endpoint, yielding snake_case dicts. Pure-ish: takes a `YunkeClient`, yields records, never touches the DB. 3. **Sync / persistence layer** (`app/sync/orchestrator.py` + `app/storage/repos.py`) — the orchestrator runs each entity's `scrape → upsert → advance watermark` loop, logs every run as a `SyncRun` row, and never re-raises a per-entity failure (one broken entity doesn't abort a batch). The repos do dialect-aware idempotent upserts (SQLite for tests, MySQL for prod) so a re-scrape of the same row is one refreshed row, not a duplicate. Messages have a real incremental window (`[last_timestamp, now]`); every other entity has no upstream `since` param and relies on idempotent re-scrape. See the [design doc](docs/superpowers/specs/2026-08-01-yunke-data-mirror-design.md) for the full data dictionary + risk-control rationale. ## Quick start ### Dev mode (no Docker) You need: Python 3.11+, Node 20+, and (optionally) a local MySQL. SQLite works for a quick smoke. Backend: ```bash cd yunke-server python -m venv .venv .venv/Scripts/activate # Windows # source .venv/bin/activate # *nix pip install -e ".[dev]" # Configure auth + DB (see .env.example). Session-cookie mode preferred: # YK_USERID=... # YK_COMPANY=... # YK_COOKIE="user=...; userToken=..." cp .env.example .env # edit .env # Run the API: uvicorn app.api.main:app --reload ``` Frontend (separate shell): ```bash cd yunkes npm install npm run dev # serves on :5173, proxies /api → :8000 ``` Open . One-shot sync into the configured DB: ```bash cd yunke-server python -m app.sync.cli sync # everything python -m app.sync.cli sync --entity friends # one entity python -m app.sync.cli --help # full usage ``` ### Docker mode (full stack) ```bash docker compose up --build ``` That brings up: | Service | Image / build | Role | Port | |-----------|---------------------|------------------------------------------------------------|------------| | `mysql` | `mysql:8.0` | Persistent data store | 3306 | | `redis` | `redis:7-alpine` | Cache / queue (future use; not yet wired into the app) | — | | `backend` | `yunke-server/` | FastAPI viewing API (uvicorn) | 8000 | | `scheduler` | `yunke-server/` | APScheduler loop — page-based sync (daily full + every-4h messages; needs Playwright/Chromium in the image) | — | | `frontend`| `yunkes/` | SPA served by nginx, reverse-proxies `/api` to backend:8000 | 5173 → 80 | Open . The scheduler emits one JSON-line log per run; tail with `docker compose logs -f scheduler`. ## Configuration All configuration is via environment / `.env` (the docker-compose stack passes the host `.env` through to the containers). See `yunke-server/.env.example` for the full list; the load-bearing ones: | Variable | Purpose | |-----------------------|---------------------------------------------------------------------------| | `YK_USERID` | Existing-session mode (preferred). Pairs with `YK_COMPANY` (+ `YK_COOKIE`). | | `YK_COMPANY` | Existing-session mode. | | `YK_COOKIE` | Optional raw `Cookie:` header for session mode. | | `YUNKE_USERNAME` | pcLogin mode (deferred — see "Limitations"). Pairs with `YUNKE_PASSWORD`. | | `YUNKE_PASSWORD` | pcLogin mode. | | `DB_URL` | SQLAlchemy async URL. Prod: `mysql+asyncmy://...`. Dev: `sqlite+aiosqlite:///./smoke.db`. | | `REDIS_URL` | Redis URL (currently unused; reserved for future cache / queue). | | `MEDIA_DIR` | Where downloaded media blobs land (sha256-deduped). | | `ENCRYPTION_KEY` | Fernet key for encrypting stored credentials. Blank for tests only. | | `API_TOKEN` | Shared bearer token for the viewing API. Blank = auth disabled (local/intranet trust). | | `FULL_SYNC_HOUR` | Scheduler: local-time hour for the daily full sync (default `2`). | | `MESSAGE_INTERVAL_HOURS` | Scheduler: hours between incremental message syncs (default `4`). | Auth precedence: session-cookie mode wins when both are set (the existing live session is always preferred over a fresh pcLogin). ## Running the scheduler manually ```bash cd yunke-server python -m app.sync.cli scheduler # uses FULL_SYNC_HOUR + MESSAGE_INTERVAL_HOURS python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6 ``` > **2026-08-20 起:定时同步全部页面方式(Playwright 浏览器抓取),不走 API 直调** > —— 每日全量跑 `scripts/full_sync_all.sh`,消息增量跑 > `scripts/scrape_via_page.py --entity messages`。API 直调 > (`sync --entity ...` / `run_once`) 保留为手动调试入口,不再挂任何定时触发器。 > 容器部署需保证镜像内有 Playwright + Chromium。 Entity names accepted by `sync --entity` and `run_once`: `departments`, `accounts`, `friends`, `groups`, `voice_calls`, `phone_calls`, `sms`, `messages`, `all_wechat`, `all` (default). ## Current limitations - **pcLogin deferred.** Programmatic login (`getSecureKey` → `pcLogin`) is wired and exercises real RSA encryption, but the final `pcLogin` body field names are inferred and the live CRM has been observed rejecting them. **Use session-cookie mode** (`YK_USERID` + `YK_COMPANY` + the captured `YK_COOKIE`) until that's resolved. Capture the cookie/userid from a logged-in browser tab on `crm2.yunkecn.com`. - **WhatsApp voice / media deferred.** The WhatsApp scraper covers the friend list + chat records; voice-call and media-download paths for WhatsApp are not yet wired. - **Rate-limited pacing.** Request cadence is deliberately conservative (default ≈ one request every 1.0–1.8 s with jitter; exponential backoff on 风控 signals). A full first-time sync is therefore slow (hours for thousands of conversations). Tunable via `REQUEST_MIN_INTERVAL` / `REQUEST_JITTER` / `MAX_REQUESTS_PER_RUN` in `app/config.py` — loosen only after evidence that the CRM tolerates a faster pace. - **Conversation discovery is local-only.** The incremental message sync reads the conversation list from the already-mirrored `customer` table, not the upstream `talkerList` endpoint. A first-time-empty mirror will therefore not pull any messages until at least one full friend sync has populated customers. - **Redis is provisioned but unused.** The compose stack includes a Redis service for future cache / queue use; nothing in the app reads or writes it yet. ## Verification ```bash cd yunke-server && .venv/Scripts/python.exe -m pytest -q # 323 tests cd yunkes && npm run build # clean production build cd yunke-server && python -m app.sync.cli --help # CLI usage docker compose config # validates the stack ```