yunke/README.md

yunke — 云客 CRM 数据本地化系统

A self-hosted mirror of the yunke CRM (https://crm2.yunkecn.com): a Python backend that periodically scrapes WeChat / WhatsApp / phone-call / SMS data into MySQL + a local object store, plus a Vue SPA for browsing it. The system is read-only against the upstream CRM — every upstream call is a list / search endpoint; nothing writes back to yunke.

What's here

.
├── yunke-server/     Python backend (FastAPI + APScheduler + scrapers + repos)
├── yunkes/           Vue 3 + Element Plus frontend (Vite build → nginx)
├── docker-compose.yml  Five-service stack (mysql / redis / backend / scheduler / frontend)
└── docs/             Design + plan documents

Architecture (1 minute version)

The backend has three concentric layers:

1. Session layer (app/session/yunke_client.py) — a single YunkeClient chokepoint that injects the auth headers, paces requests with a jittered rate limiter, backs off on risk-control signals (HTTP 429 or 风控 body markers), and re-logs-in once on 401.
2. Scraper layer (app/scraper/*) — one paginator + parser per upstream endpoint, yielding snake_case dicts. Pure-ish: takes a YunkeClient, yields records, never touches the DB.
3. Sync / persistence layer (app/sync/orchestrator.py + app/storage/repos.py) — the orchestrator runs each entity's scrape → upsert → advance watermark loop, logs every run as a SyncRun row, and never re-raises a per-entity failure (one broken entity doesn't abort a batch). The repos do dialect-aware idempotent upserts (SQLite for tests, MySQL for prod) so a re-scrape of the same row is one refreshed row, not a duplicate.

Messages have a real incremental window ([last_timestamp, now]); every other entity has no upstream since param and relies on idempotent re-scrape. See the design doc for the full data dictionary + risk-control rationale.

Quick start

Dev mode (no Docker)

You need: Python 3.11+, Node 20+, and (optionally) a local MySQL. SQLite works for a quick smoke.

Backend:

cd yunke-server
python -m venv .venv
.venv/Scripts/activate         # Windows
# source .venv/bin/activate    # *nix
pip install -e ".[dev]"

# Configure auth + DB (see .env.example). Session-cookie mode preferred:
#   YK_USERID=...
#   YK_COMPANY=...
#   YK_COOKIE="user=...; userToken=..."
cp .env.example .env
# edit .env

# Run the API:
uvicorn app.api.main:app --reload

Frontend (separate shell):

cd yunkes
npm install
npm run dev        # serves on :5173, proxies /api → :8000

Open http://localhost:5173.

One-shot sync into the configured DB:

cd yunke-server
python -m app.sync.cli sync                       # everything
python -m app.sync.cli sync --entity friends      # one entity
python -m app.sync.cli --help                     # full usage

Docker mode (full stack)

docker compose up --build

That brings up:

Service	Image / build	Role	Port
mysql	mysql:8.0	Persistent data store	3306
redis	redis:7-alpine	Cache / queue (future use; not yet wired into the app)	—
backend	yunke-server/	FastAPI viewing API (uvicorn)	8000
scheduler	yunke-server/	APScheduler loop — daily full sync + every-4h messages	—
frontend	yunkes/	SPA served by nginx, reverse-proxies /api to backend:8000	5173 → 80

Open http://localhost:5173. The scheduler emits one JSON-line log per run; tail with docker compose logs -f scheduler.

Configuration

All configuration is via environment / .env (the docker-compose stack passes the host .env through to the containers). See yunke-server/.env.example for the full list; the load-bearing ones:

Variable	Purpose
YK_USERID	Existing-session mode (preferred). Pairs with YK_COMPANY (+ YK_COOKIE).
YK_COMPANY	Existing-session mode.
YK_COOKIE	Optional raw Cookie: header for session mode.
YUNKE_USERNAME	pcLogin mode (deferred — see "Limitations"). Pairs with YUNKE_PASSWORD.
YUNKE_PASSWORD	pcLogin mode.
DB_URL	SQLAlchemy async URL. Prod: mysql+asyncmy://.... Dev: sqlite+aiosqlite:///./smoke.db.
REDIS_URL	Redis URL (currently unused; reserved for future cache / queue).
MEDIA_DIR	Where downloaded media blobs land (sha256-deduped).
ENCRYPTION_KEY	Fernet key for encrypting stored credentials. Blank for tests only.
API_TOKEN	Shared bearer token for the viewing API. Blank = auth disabled (local/intranet trust).
FULL_SYNC_HOUR	Scheduler: local-time hour for the daily full sync (default 2).
MESSAGE_INTERVAL_HOURS	Scheduler: hours between incremental message syncs (default 4).

Auth precedence: session-cookie mode wins when both are set (the existing live session is always preferred over a fresh pcLogin).

Running the scheduler manually

cd yunke-server
python -m app.sync.cli scheduler                  # uses FULL_SYNC_HOUR + MESSAGE_INTERVAL_HOURS
python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6

Entity names accepted by sync --entity and run_once: departments, accounts, friends, groups, voice_calls, phone_calls, sms, messages, all_wechat, all (default).

Current limitations

• pcLogin deferred. Programmatic login (getSecureKey → pcLogin) is wired and exercises real RSA encryption, but the final pcLogin body field names are inferred and the live CRM has been observed rejecting them. Use session-cookie mode (YK_USERID + YK_COMPANY + the captured YK_COOKIE) until that's resolved. Capture the cookie/userid from a logged-in browser tab on crm2.yunkecn.com.
• WhatsApp voice / media deferred. The WhatsApp scraper covers the friend list + chat records; voice-call and media-download paths for WhatsApp are not yet wired.
• Rate-limited pacing. Request cadence is deliberately conservative (default ≈ one request every 1.0–1.8 s with jitter; exponential backoff on 风控 signals). A full first-time sync is therefore slow (hours for thousands of conversations). Tunable via REQUEST_MIN_INTERVAL / REQUEST_JITTER / MAX_REQUESTS_PER_RUN in app/config.py — loosen only after evidence that the CRM tolerates a faster pace.
• Conversation discovery is local-only. The incremental message sync reads the conversation list from the already-mirrored customer table, not the upstream talkerList endpoint. A first-time-empty mirror will therefore not pull any messages until at least one full friend sync has populated customers.
• Redis is provisioned but unused. The compose stack includes a Redis service for future cache / queue use; nothing in the app reads or writes it yet.

Verification

cd yunke-server && .venv/Scripts/python.exe -m pytest -q   # 323 tests
cd yunkes && npm run build                                  # clean production build
cd yunke-server && python -m app.sync.cli --help            # CLI usage
docker compose config                                       # validates the stack