yunke/README.md

# yunke — 云客 CRM 数据本地化系统

A self-hosted mirror of the yunke CRM (`https://crm2.yunkecn.com`): a Python
backend that periodically scrapes WeChat / WhatsApp / phone-call / SMS data
into MySQL + a local object store, plus a Vue SPA for browsing it. The
system is read-only against the upstream CRM — every upstream call is a
list / search endpoint; nothing writes back to yunke.

## What's here

```
.
├── yunke-server/     Python backend (FastAPI + APScheduler + scrapers + repos)
├── yunkes/           Vue 3 + Element Plus frontend (Vite build → nginx)
├── docker-compose.yml  Five-service stack (mysql / redis / backend / scheduler / frontend)
└── docs/             Design + plan documents
```

## Architecture (1 minute version)

The backend has three concentric layers:

1. **Session layer** (`app/session/yunke_client.py`) — a single
   `YunkeClient` chokepoint that injects the auth headers, paces requests
   with a jittered rate limiter, backs off on risk-control signals (HTTP
   429 or 风控 body markers), and re-logs-in once on 401.
2. **Scraper layer** (`app/scraper/*`) — one paginator + parser per
   upstream endpoint, yielding snake_case dicts. Pure-ish: takes a
   `YunkeClient`, yields records, never touches the DB.
3. **Sync / persistence layer** (`app/sync/orchestrator.py` +
   `app/storage/repos.py`) — the orchestrator runs each entity's
   `scrape → upsert → advance watermark` loop, logs every run as a
   `SyncRun` row, and never re-raises a per-entity failure (one broken
   entity doesn't abort a batch). The repos do dialect-aware idempotent
   upserts (SQLite for tests, MySQL for prod) so a re-scrape of the same
   row is one refreshed row, not a duplicate.

Messages have a real incremental window (`[last_timestamp, now]`); every
other entity has no upstream `since` param and relies on idempotent
re-scrape. See the [design doc](docs/superpowers/specs/2026-08-01-yunke-data-mirror-design.md)
for the full data dictionary + risk-control rationale.

## Quick start

### Dev mode (no Docker)

You need: Python 3.11+, Node 20+, and (optionally) a local MySQL. SQLite
works for a quick smoke.

Backend:

```bash
cd yunke-server
python -m venv .venv
.venv/Scripts/activate         # Windows
# source .venv/bin/activate    # *nix
pip install -e ".[dev]"

# Configure auth + DB (see .env.example). Session-cookie mode preferred:
#   YK_USERID=...
#   YK_COMPANY=...
#   YK_COOKIE="user=...; userToken=..."
cp .env.example .env
# edit .env

# Run the API:
uvicorn app.api.main:app --reload
```

Frontend (separate shell):

```bash
cd yunkes
npm install
npm run dev        # serves on :5173, proxies /api → :8000
```

Open <http://localhost:5173>.

One-shot sync into the configured DB:

```bash
cd yunke-server
python -m app.sync.cli sync                       # everything
python -m app.sync.cli sync --entity friends      # one entity
python -m app.sync.cli --help                     # full usage
```

One-shot analytics backfills / self-heal (all idempotent):

```bash
python -m app.sync.cli daily-stats                # msg_daily_stats 日汇总全量重建
python -m app.sync.cli staff-daily-stats          # 员工日汇总(stat_daily_staff)全量重建
python -m app.sync.cli sensitive-scan --full      # 敏感词全量回填+建扫描水位(首跑必须 --full)
```

### Docker mode (full stack)

```bash
docker compose up --build
```

That brings up:

| Service   | Image / build       | Role                                                       | Port       |
|-----------|---------------------|------------------------------------------------------------|------------|
| `mysql`   | `mysql:8.0`         | Persistent data store                                      | 3306       |
| `redis`   | `redis:7-alpine`    | Cache / queue (future use; not yet wired into the app)     | —          |
| `backend` | `yunke-server/`     | FastAPI viewing API (uvicorn)                              | 8000       |
| `scheduler` | `yunke-server/`   | APScheduler loop — page-based sync (daily full + every-4h messages; needs Playwright/Chromium in the image) | — |
| `frontend`| `yunkes/`           | SPA served by nginx, reverse-proxies `/api` to backend:8000 | 5173 → 80 |

Open <http://localhost:5173>. The scheduler emits one JSON-line log per
run; tail with `docker compose logs -f scheduler`.

## Configuration

All configuration is via environment / `.env` (the docker-compose stack
passes the host `.env` through to the containers). See
`yunke-server/.env.example` for the full list; the load-bearing ones:

| Variable              | Purpose                                                                   |
|-----------------------|---------------------------------------------------------------------------|
| `YK_USERID`           | Existing-session mode (preferred). Pairs with `YK_COMPANY` (+ `YK_COOKIE`). |
| `YK_COMPANY`          | Existing-session mode.                                                    |
| `YK_COOKIE`           | Optional raw `Cookie:` header for session mode.                           |
| `YUNKE_USERNAME`      | pcLogin mode (deferred — see "Limitations"). Pairs with `YUNKE_PASSWORD`. |
| `YUNKE_PASSWORD`      | pcLogin mode.                                                             |
| `DB_URL`              | SQLAlchemy async URL. Prod: `mysql+asyncmy://...`. Dev: `sqlite+aiosqlite:///./smoke.db`. |
| `REDIS_URL`           | Redis URL (currently unused; reserved for future cache / queue).         |
| `MEDIA_DIR`           | Where downloaded media blobs land (sha256-deduped).                       |
| `ENCRYPTION_KEY`      | Fernet key for encrypting stored credentials. Blank for tests only.       |
| `API_TOKEN`           | Shared bearer token for the viewing API. Blank = auth disabled (local/intranet trust). |
| `FULL_SYNC_HOUR`      | Scheduler: local-time hour for the daily full sync (default `2`).         |
| `MESSAGE_INTERVAL_HOURS` | Scheduler: hours between incremental message syncs (default `4`).      |

Auth precedence: session-cookie mode wins when both are set (the existing
live session is always preferred over a fresh pcLogin).

## Running the scheduler manually

```bash
cd yunke-server
python -m app.sync.cli scheduler                  # uses FULL_SYNC_HOUR + MESSAGE_INTERVAL_HOURS
python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6
```

> **2026-08-20 起:定时同步全部页面方式(Playwright 浏览器抓取),不走 API 直调**
> —— 每日全量跑 `scripts/full_sync_all.sh`,消息增量跑
> `scripts/scrape_via_page.py --entity messages`。API 直调
> (`sync --entity ...` / `run_once`) 保留为手动调试入口,不再挂任何定时触发器。
> 容器部署需保证镜像内有 Playwright + Chromium。

Entity names accepted by `sync --entity` and `run_once`:
`departments`, `accounts`, `friends`, `groups`, `voice_calls`,
`phone_calls`, `sms`, `messages`, `all_wechat`, `all` (default).

## Current limitations

- **pcLogin deferred.** Programmatic login (`getSecureKey` → `pcLogin`) is
  wired and exercises real RSA encryption, but the final `pcLogin` body
  field names are inferred and the live CRM has been observed rejecting
  them. **Use session-cookie mode** (`YK_USERID` + `YK_COMPANY` + the
  captured `YK_COOKIE`) until that's resolved. Capture the cookie/userid
  from a logged-in browser tab on `crm2.yunkecn.com`.
- **WhatsApp voice / media deferred.** The WhatsApp scraper covers the
  friend list + chat records; voice-call and media-download paths for
  WhatsApp are not yet wired.
- **Rate-limited pacing.** Request cadence is deliberately conservative
  (default ≈ one request every 1.0–1.8 s with jitter; exponential backoff
  on 风控 signals). A full first-time sync is therefore slow (hours for
  thousands of conversations). Tunable via `REQUEST_MIN_INTERVAL` /
  `REQUEST_JITTER` / `MAX_REQUESTS_PER_RUN` in `app/config.py` — loosen
  only after evidence that the CRM tolerates a faster pace.
- **Conversation discovery is local-only.** The incremental message sync
  reads the conversation list from the already-mirrored `customer` table,
  not the upstream `talkerList` endpoint. A first-time-empty mirror will
  therefore not pull any messages until at least one full friend sync has
  populated customers.
- **Redis is provisioned but unused.** The compose stack includes a Redis
  service for future cache / queue use; nothing in the app reads or writes
  it yet.

## Verification

```bash
cd yunke-server && .venv/Scripts/python.exe -m pytest -q   # 574 tests
cd yunkes && npm run build                                  # clean production build
cd yunke-server && python -m app.sync.cli --help            # CLI usage
docker compose config                                       # validates the stack
```