浅念
192 lines
8.7 KiB
Markdown
192 lines
8.7 KiB
Markdown
# yunke — 云客 CRM 数据本地化系统
|
||
|
||
A self-hosted mirror of the yunke CRM (`https://crm2.yunkecn.com`): a Python
|
||
backend that periodically scrapes WeChat / WhatsApp / phone-call / SMS data
|
||
into MySQL + a local object store, plus a Vue SPA for browsing it. The
|
||
system is read-only against the upstream CRM — every upstream call is a
|
||
list / search endpoint; nothing writes back to yunke.
|
||
|
||
## What's here
|
||
|
||
```
|
||
.
|
||
├── yunke-server/ Python backend (FastAPI + APScheduler + scrapers + repos)
|
||
├── yunkes/ Vue 3 + Element Plus frontend (Vite build → nginx)
|
||
├── docker-compose.yml Five-service stack (mysql / redis / backend / scheduler / frontend)
|
||
└── docs/ Design + plan documents
|
||
```
|
||
|
||
## Architecture (1 minute version)
|
||
|
||
The backend has three concentric layers:
|
||
|
||
1. **Session layer** (`app/session/yunke_client.py`) — a single
|
||
`YunkeClient` chokepoint that injects the auth headers, paces requests
|
||
with a jittered rate limiter, backs off on risk-control signals (HTTP
|
||
429 or 风控 body markers), and re-logs-in once on 401.
|
||
2. **Scraper layer** (`app/scraper/*`) — one paginator + parser per
|
||
upstream endpoint, yielding snake_case dicts. Pure-ish: takes a
|
||
`YunkeClient`, yields records, never touches the DB.
|
||
3. **Sync / persistence layer** (`app/sync/orchestrator.py` +
|
||
`app/storage/repos.py`) — the orchestrator runs each entity's
|
||
`scrape → upsert → advance watermark` loop, logs every run as a
|
||
`SyncRun` row, and never re-raises a per-entity failure (one broken
|
||
entity doesn't abort a batch). The repos do dialect-aware idempotent
|
||
upserts (SQLite for tests, MySQL for prod) so a re-scrape of the same
|
||
row is one refreshed row, not a duplicate.
|
||
|
||
Messages have a real incremental window (`[last_timestamp, now]`); every
|
||
other entity has no upstream `since` param and relies on idempotent
|
||
re-scrape. See the [design doc](docs/superpowers/specs/2026-08-01-yunke-data-mirror-design.md)
|
||
for the full data dictionary + risk-control rationale.
|
||
|
||
## Quick start
|
||
|
||
### Dev mode (no Docker)
|
||
|
||
You need: Python 3.11+, Node 20+, and (optionally) a local MySQL. SQLite
|
||
works for a quick smoke.
|
||
|
||
Backend:
|
||
|
||
```bash
|
||
cd yunke-server
|
||
python -m venv .venv
|
||
.venv/Scripts/activate # Windows
|
||
# source .venv/bin/activate # *nix
|
||
pip install -e ".[dev]"
|
||
|
||
# Configure auth + DB (see .env.example). Session-cookie mode preferred:
|
||
# YK_USERID=...
|
||
# YK_COMPANY=...
|
||
# YK_COOKIE="user=...; userToken=..."
|
||
cp .env.example .env
|
||
# edit .env
|
||
|
||
# Run the API:
|
||
uvicorn app.api.main:app --reload
|
||
```
|
||
|
||
Frontend (separate shell):
|
||
|
||
```bash
|
||
cd yunkes
|
||
npm install
|
||
npm run dev # serves on :5173, proxies /api → :8000
|
||
```
|
||
|
||
Open <http://localhost:5173>.
|
||
|
||
One-shot sync into the configured DB:
|
||
|
||
```bash
|
||
cd yunke-server
|
||
python -m app.sync.cli sync # everything
|
||
python -m app.sync.cli sync --entity friends # one entity
|
||
python -m app.sync.cli --help # full usage
|
||
```
|
||
|
||
One-shot analytics backfills / self-heal (all idempotent):
|
||
|
||
```bash
|
||
python -m app.sync.cli daily-stats # msg_daily_stats 日汇总全量重建
|
||
python -m app.sync.cli staff-daily-stats # 员工日汇总(stat_daily_staff)全量重建
|
||
python -m app.sync.cli sensitive-scan --full # 敏感词全量回填+建扫描水位(首跑必须 --full)
|
||
```
|
||
|
||
### Docker mode (full stack)
|
||
|
||
```bash
|
||
docker compose up --build
|
||
```
|
||
|
||
That brings up:
|
||
|
||
| Service | Image / build | Role | Port |
|
||
|-----------|---------------------|------------------------------------------------------------|------------|
|
||
| `mysql` | `mysql:8.0` | Persistent data store | 3306 |
|
||
| `redis` | `redis:7-alpine` | Cache / queue (future use; not yet wired into the app) | — |
|
||
| `backend` | `yunke-server/` | FastAPI viewing API (uvicorn) | 8000 |
|
||
| `scheduler` | `yunke-server/` | APScheduler loop — page-based sync (daily full + every-4h messages; needs Playwright/Chromium in the image) | — |
|
||
| `frontend`| `yunkes/` | SPA served by nginx, reverse-proxies `/api` to backend:8000 | 5173 → 80 |
|
||
|
||
Open <http://localhost:5173>. The scheduler emits one JSON-line log per
|
||
run; tail with `docker compose logs -f scheduler`.
|
||
|
||
## Configuration
|
||
|
||
All configuration is via environment / `.env` (the docker-compose stack
|
||
passes the host `.env` through to the containers). See
|
||
`yunke-server/.env.example` for the full list; the load-bearing ones:
|
||
|
||
| Variable | Purpose |
|
||
|-----------------------|---------------------------------------------------------------------------|
|
||
| `YK_USERID` | Existing-session mode (preferred). Pairs with `YK_COMPANY` (+ `YK_COOKIE`). |
|
||
| `YK_COMPANY` | Existing-session mode. |
|
||
| `YK_COOKIE` | Optional raw `Cookie:` header for session mode. |
|
||
| `YUNKE_USERNAME` | pcLogin mode (deferred — see "Limitations"). Pairs with `YUNKE_PASSWORD`. |
|
||
| `YUNKE_PASSWORD` | pcLogin mode. |
|
||
| `DB_URL` | SQLAlchemy async URL. Prod: `mysql+asyncmy://...`. Dev: `sqlite+aiosqlite:///./smoke.db`. |
|
||
| `REDIS_URL` | Redis URL (currently unused; reserved for future cache / queue). |
|
||
| `MEDIA_DIR` | Where downloaded media blobs land (sha256-deduped). |
|
||
| `ENCRYPTION_KEY` | Fernet key for encrypting stored credentials. Blank for tests only. |
|
||
| `API_TOKEN` | Shared bearer token for the viewing API. Blank = auth disabled (local/intranet trust). |
|
||
| `FULL_SYNC_HOUR` | Scheduler: local-time hour for the daily full sync (default `2`). |
|
||
| `MESSAGE_INTERVAL_HOURS` | Scheduler: hours between incremental message syncs (default `4`). |
|
||
|
||
Auth precedence: session-cookie mode wins when both are set (the existing
|
||
live session is always preferred over a fresh pcLogin).
|
||
|
||
## Running the scheduler manually
|
||
|
||
```bash
|
||
cd yunke-server
|
||
python -m app.sync.cli scheduler # uses FULL_SYNC_HOUR + MESSAGE_INTERVAL_HOURS
|
||
python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6
|
||
```
|
||
|
||
> **2026-08-20 起:定时同步全部页面方式(Playwright 浏览器抓取),不走 API 直调**
|
||
> —— 每日全量跑 `scripts/full_sync_all.sh`,消息增量跑
|
||
> `scripts/scrape_via_page.py --entity messages`。API 直调
|
||
> (`sync --entity ...` / `run_once`) 保留为手动调试入口,不再挂任何定时触发器。
|
||
> 容器部署需保证镜像内有 Playwright + Chromium。
|
||
|
||
Entity names accepted by `sync --entity` and `run_once`:
|
||
`departments`, `accounts`, `friends`, `groups`, `voice_calls`,
|
||
`phone_calls`, `sms`, `messages`, `all_wechat`, `all` (default).
|
||
|
||
## Current limitations
|
||
|
||
- **pcLogin deferred.** Programmatic login (`getSecureKey` → `pcLogin`) is
|
||
wired and exercises real RSA encryption, but the final `pcLogin` body
|
||
field names are inferred and the live CRM has been observed rejecting
|
||
them. **Use session-cookie mode** (`YK_USERID` + `YK_COMPANY` + the
|
||
captured `YK_COOKIE`) until that's resolved. Capture the cookie/userid
|
||
from a logged-in browser tab on `crm2.yunkecn.com`.
|
||
- **WhatsApp voice / media deferred.** The WhatsApp scraper covers the
|
||
friend list + chat records; voice-call and media-download paths for
|
||
WhatsApp are not yet wired.
|
||
- **Rate-limited pacing.** Request cadence is deliberately conservative
|
||
(default ≈ one request every 1.0–1.8 s with jitter; exponential backoff
|
||
on 风控 signals). A full first-time sync is therefore slow (hours for
|
||
thousands of conversations). Tunable via `REQUEST_MIN_INTERVAL` /
|
||
`REQUEST_JITTER` / `MAX_REQUESTS_PER_RUN` in `app/config.py` — loosen
|
||
only after evidence that the CRM tolerates a faster pace.
|
||
- **Conversation discovery is local-only.** The incremental message sync
|
||
reads the conversation list from the already-mirrored `customer` table,
|
||
not the upstream `talkerList` endpoint. A first-time-empty mirror will
|
||
therefore not pull any messages until at least one full friend sync has
|
||
populated customers.
|
||
- **Redis is provisioned but unused.** The compose stack includes a Redis
|
||
service for future cache / queue use; nothing in the app reads or writes
|
||
it yet.
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
cd yunke-server && .venv/Scripts/python.exe -m pytest -q # 574 tests
|
||
cd yunkes && npm run build # clean production build
|
||
cd yunke-server && python -m app.sync.cli --help # CLI usage
|
||
docker compose config # validates the stack
|
||
```
|