Add the first list scraper for the yunke data-mirror system, establishing
the reusable pagination pattern all later list scrapers will build on.
- app/scraper/base.py: generic `paginate_by_page` async generator that walks
page-numbered endpoints until a page comes back empty (the yunke CRM's
primary "no more rows" signal), with a defensive over-fetch ceiling keyed
off the declared total (strict `>` so an honest source yields exactly
`total`). Plus a tiny `RateLimiter` enforcing a min interval between calls.
- app/scraper/wechat.py: `scrape_wechat_friends` wraps
`POST /wechatweb/wxFriends/friend_list/es`, maps each record to the
`Customer` column shape (snake_case) with `channel="wechat"` and a verbatim
`raw_json` copy. Optional `department_id`/`user_id` filters pass through to
the request body.
- tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend
records + a response-envelope builder, reused by the scraper tests.
- tests/test_scrape_wechat_friends.py: drives the real pagination +
normalization logic through respx (page1 returns both records, page2 empty
-> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2
upstream calls.
- tests/test_scrape_base.py: smoke tests for the RateLimiter spacing.
Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7;
scrapers remain pure producers of normalized dicts.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 1.4 — 会话层. Adds the authenticated HTTP client that every yunke
scraper will use to call the CRM API.
- app/session/login.py: get_secure_key + pc_login wrappers, LoginResult
dataclass, and a clearly-marked Phase-0 swappable encrypt_password
placeholder (real RSA per getSecureKey is a TODO; drop-in contract
documented in the docstring).
- app/session/yunke_client.py: YunkeClient wrapping httpx.AsyncClient.
Constructor takes either (username+password) for auto-login or
(userid+company[+cookies]) for an existing session. Injects the 5
observed auth headers (userid/company/source=yunkepc/channel=4/
content-type=application/json) on every request, auto-logs-in on first
request, and on HTTP 401 (or body code 401) re-logs-in once and
retries — a persistent 401 raises after the single retry so there is
no infinite loop.
- Tests (respx): login populates LoginResult + httpx cookie jar; every
request carries all 5 auth headers + cookie; 401 triggers exactly one
re-login + retry and returns 200; persistent 401 raises after one
retry; first request with no session auto-logs-in.
Session is held in-memory only for now; persistence to encrypted
YkAccount.session_blob_enc is left as a later concern (TODO in
yunke_client.py).
Co-Authored-By: Claude <noreply@anthropic.com>
Implements the 14 core ORM models for the yunke data-mirror system using
SQLAlchemy 2.0 Mapped/mapped_column style, driven by the field lists in
docs/superpowers/specs/2026-08-01-yunke-data-dictionary.md:
Department, Staff, WechatAccount, WhatsappAccount, Customer, ChatGroup,
GroupMember, Message, Media, PhoneCall, Sms, SyncState, YkAccount, SyncRun.
Conventions:
- Natural source PKs where they exist (friend_id, wx_id, msg_id, room_id,
media_hash, call_id, sms_id, dept_id, user_id); surrogate Integer PKs
otherwise (SyncState, YkAccount, SyncRun). GroupMember uses a composite
(channel, room_id, member_id) PK.
- Common audit columns version/raw_json/fetched_at/updated_at on every
source-mirrored table.
- channel only where meaningful: Customer/Message/Media/ChatGroup/
GroupMember/Sms use messaging channels; PhoneCall.channel is call-type;
SyncState.channel scopes the cursor. Omitted on Staff/Department/
WechatAccount/WhatsappAccount/YkAccount/SyncRun.
- Message composite index on (channel, owner_account_id, talker_id,
timestamp) for the hot per-talker time-order query path.
- message_table_name(channel, timestamp) helper documents the future
message_<channel>_<YYYYMM> sharding convention; the Message model still
maps to a single physical table for now (sharding is a later task).
All types are portable generic SQLAlchemy types so tests run on in-memory
SQLite while prod targets MySQL. Tests verify Customer round-trip,
raw_json JSON round-trip, Message natural PK, GroupMember composite PK,
the Message composite index, and full table creation on SQLite.
Alembic wired via migrations/env.py to Base.metadata + get_settings().db_url
with an async->sync driver swap (mysql+asyncmy -> mysql+pymysql,
sqlite+aiosqlite -> sqlite). Initial migration autogenerated against an
empty SQLite DB; server defaults normalized to sa.func.now() so the file
is MySQL-portable. upgrade head is intentionally not run (no MySQL yet).
Co-Authored-By: Claude <noreply@anthropic.com>
Create pyproject.toml (FastAPI + SQLAlchemy + Playwright + APScheduler +
Redis + cryptography stack), .env.example, app/__init__.py and a
pydantic-settings Settings singleton (get_settings), and a root
.gitignore covering Python, Node, env, IDE and the local media dir.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念