Adds a shared-token auth dependency for the viewing API (Task 2.9). Trust
model: when API_TOKEN is empty (the default) auth is disabled — we trust the
network boundary (local/intranet tool). When set, every request must present
the token via `Authorization: Bearer <token>` or `?token=<token>`, else 401.
`/api/health` is exempt so liveness probes work without the token.
- app/config.py: add `api_token: str = ""` to Settings.
- app/api/auth.py: `require_auth` dependency (secrets.compare_digest, reads
get_settings() per-request so token rotation via cache_clear() works).
- app/api/main.py: apply the dependency globally; document the trust model.
- .env.example: document API_TOKEN.
- tests/test_api_auth.py: 8 tests covering disabled/correct/missing/wrong/
malformed-header/query-param/health-exempt paths.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 2.7 — the coordination layer tying scrapers + repos together with
resumable, incremental sync.
app/sync/watermarks.py — SyncState CRUD keyed on (entity_type, channel,
scope_id): get_watermark / get_last_timestamp / get_last_version /
set_watermark (upsert). A partial advance (timestamp-only) does not clobber
the other cursor with NULL.
app/sync/orchestrator.py — the sync brain:
* sync_wechat_messages: time-window incremental. Reads last_timestamp for
(message, wechat, "<wechat_id>:<friend_id>"), queries [last_ts, now] via
scrape_wechat_messages, upserts each, best-effort downloads media, advances
the watermark to max(timestamp). A re-run with no new messages queries an
empty window and fetches nothing — the watermark advanced past.
* sync_{departments,wechat_accounts,wechat_friends,wechat_groups,
wechat_voice_calls}: full re-scrape + idempotent upsert (these list APIs
have no since-param; 'incremental' = re-scrape all, rely on PK idempotency).
Watermark records last_synced_at only.
* sync_all_wechat: coordinator running departments -> accounts -> friends ->
groups -> voice_calls, then per-conversation messages for an explicit
(wechat_id, friend_id) list. Auto-discovery deferred.
Each run logs a SyncRun row (started -> success/failed, counts, error).
Failure policy: a scraper that raises is caught, recorded as failed, and a
failed SyncResult is returned — the orchestrator does not re-raise, so one
broken entity does not abort the batch. Media-download failures are
best-effort. The orchestrator never commits; callers own the transaction.
Tests (TDD red -> green; in-memory SQLite; scrapers/repos monkeypatched):
- watermarks: set/get, upsert (same key twice -> 1 row), partial-advance
preserves the other cursor, defaults when absent.
- orchestrator messages: first run upserts 2 + advances watermark to T2;
second run scrape called with start==T2 and upserts 0 (incremental proof);
media failure does not abort; SyncRun success recorded.
- orchestrator friends: 3 records upserted, SyncRun success, watermark
last_synced_at set.
- failure policy: scraper raises -> SyncRun failed + failed result, no raise
(both friends and message paths).
- sync_all_wechat: 5 entities + 2 message runs in order, all success.
Suite: 149 -> 163 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
M2 Task 2.6 (media downloader) + Task 2.8 (media serve API).
Downloader (app/scraper/media.py):
- download_url(client, url): fetches bytes via the YunkeClient's underlying
httpx client (carries session cookies; respx-mockable; follows redirects).
- download_message_media(client, msg, store, session): URL-driven fetcher that
picks the best http(s) URL (file_url -> file_hd_url -> file_th_url), dedups
by md5 (skips the network when a Media row with local file already exists),
falls back to sha256(bytes) when no md5, stores via ObjectStore.put_by_hash,
and upserts a Media row. Skips type-10 system messages (non-media per
phase0-findings §2) and type-3 opaque-id / text (no http url). Resilient:
download errors are logged and return None (no raise).
- media_type_from_message / guess_mime_type helpers.
Repo (app/storage/repos.py): upsert_media — idempotent Core upsert into Media by
media_hash. Volatile download artifacts (local_path/size_bytes/mime_type) use
COALESCE(incoming, existing) on conflict so a re-scrape that did not re-download
does not clobber previously stored values. Exported from app.storage.
Serve API (app/api/routers/media.py): GET /api/media/{media_hash} streams the
local file via FileResponse (handles HTTP Range / 206). 404 when the row is
absent, has no local file, or the file is missing on disk. Content-type from the
row's mime_type, falling back to ext-derived guess.
Tests (+19, suite 130 -> 149, all green; ruff clean):
- test_media_downloader.py: image/video download + upsert; md5 dedup skips
second network hit; sha256 fallback when no md5; type-3/text/system -> None;
download failure -> None (no raise); upsert_media idempotency +
null-local_path preservation + 3x-same-key-one-row.
- test_api_media.py: 200 + content-type + bytes; 404 unknown hash; 404 missing
file on disk; Range -> 206 partial; ext-derived content-type when mime NULL.
Co-Authored-By: Claude <noreply@anthropic.com>
- scrape_wechat_voice_calls: wraps POST /wechatweb/callRecord/queryWeChatVoiceList
with page-number pagination (reuses paginate_by_page); normalizes records to
the PhoneCall column shape with channel='voice_call'. Field mapping per
phase0-findings §4: id->call_id, duration(str s)->call_duration(int),
startTime(YYYY-MM-DD HH:MM:SS)->start_call_time(ms epoch), callStatus->
call_status, talker->call_number, talkerNickName->plan_customer_name,
ossFileName->record_file (may be null). isSend (0=in/1=out) is inverted to
incoming_call (1=in/0=out) so the column's meaning matches the phone_call
channel.
- upsert_voice_call: idempotent Core upsert into PhoneCall by call_id
(sqlite/mysql dialect-aware). Adds call_duration/start_call_time to the
_INT_FIELDS coercion net (scraper already emits ints; defensive for
hand-built records).
- Group-message test (Task 2.3): locks in that scrape_wechat_messages already
handles chatroom conversations (friendId=@chatroom, roomid set, talker=member
wxid) — yields room_id populated and talker_id = the sending member. No
scraper change required.
Suite: 123 -> 130 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Add scrape_wechat_groups (POST /wechatweb/chatroom/getChatroomList) and
scrape_wechat_talkers (POST /wechatweb/wxFriends/talkerList), both reusing
the page-number paginator. Groups parse the semicolon-joined memberIds into
a member_ids_list so upsert_chat_group can mirror GroupMember rows
(composite PK channel/room_id/member_id) idempotently alongside the
ChatGroup row. Talkers are a sync-planning input (member_count>0 marks a
group conversation); no dedicated table.
Co-Authored-By: Claude <noreply@anthropic.com>
M2 organization scrapers for yunke-server:
- scrape_departments: POST /usercenter/dept/tree. Single request (no
pagination). Defensively unwraps the envelope (data is node list OR an
object wrapping nodes under children), flattens BOTH nested and flat
shapes, and reconstructs the materialized path by walking the pId chain
(ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.
- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
pagination via the existing paginate_by_page helper. Envelope differs
from friend_list (no basicResult wrapper): data.data/data.total. Maps
wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
WechatAccount columns; carries embedded staff pair for the repo layer.
Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
forward phone/sale_phone/is_on_job when present.
Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).
Suite: 97 -> 111 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace the Phase-0 placeholder password encryption with real RSA
(PKCS#1 v1.5). encrypt_password now builds an RSA public key from the
getSecureKey modulus/exponent (RSAPublicNumbers, e=int(hex,16),
n=int(hex,16)), encrypts the UTF-8 password, and returns hex (documented
assumption -- base64 is the one-line fallback noted in the docstring).
get_secure_key now returns the inner data dict to match the new contract;
pc_login keeps {account, password} with a comment on the username/password
and account/pwd fallbacks if pcLogin rejects.
Extend normalize_wechat_type with the Phase-0 confirmed codes:
2 -> image, 3 -> image_legacy, 4 -> video, 10 -> system (1 -> text and
-2 -> divider unchanged; 6/9/13/15/21/22 stay unknown until confirmed).
Tests:
- encrypt_password round-trip: generate an RSA keypair, build secure_key
from its public numbers, encrypt, then decrypt with the private key and
assert the original password comes back. Pins 512-hex output for a
2048-bit key. No network.
- Parser parametrize table extended for the new codes (and a downstream
message-fixture assertion updated from unknown -> image_legacy).
Suite: 87 -> 97 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Wires the existing pieces together into a runnable integration check:
scrape ~10 WeChat friends + ~50 of their messages (last 7 days) from the
real yunke CRM (read-only), upsert into a local SQLite smoke DB, then
print a summary that verifies rows landed.
Two auth modes:
* Existing-session (preferred): YK_USERID/YK_COMPANY[/YK_COOKIE] -> YunkeClient
constructed from a captured session, login bypassed.
* Login mode (Phase-0 blocked): YUNKE_USERNAME/YUNKE_PASSWORD -> attempts
client.login(), which the real CRM will reject until the placeholder
password encryption is replaced. Prints a clear message and exits 3.
build_client_from_env() is split out as a pure, network-free helper so
auth-mode selection is unit-testable. main() is async (asyncio.run in
__main__). HTTPStatusError from a 401/expired session is wrapped into a
helpful message + exit 4 (no traceback). client.aclose() + engine.dispose()
always run in finally.
Defaults to SMOKE_DB_URL=sqlite+aiosqlite:///./smoke.db so no MySQL is
required. Run from yunke-server/ with the venv active:
python -m scripts.smoke_wechat
Tests cover the entrypoint shape and auth-mode selection (no network).
Suite: 77 -> 87 passing.
Co-Authored-By: Claude <noreply@anthropic.com>
Minimal read-only FastAPI surface over the mirrored DB:
- GET /api/customers with pagination (page/page_size) and optional
channel / owner_account_id / department_id filters, ordered by
last_chat_time desc (nulls last) then friend_id.
- GET /api/health liveness probe.
- Pydantic v2 CustomerOut + generic Page[T] envelope.
- CORS configured permissively for the local frontend.
- Tests override get_session to an in-memory async SQLite session
(StaticPool) and drive the app via httpx ASGITransport.
Co-Authored-By: Claude <noreply@anthropic.com>
The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the storage layer that bridges scrapers (which yield normalized dicts)
to the DB and the local media store.
app/storage/repos.py
* upsert_customer / upsert_customers / upsert_messages / upsert_wechat_account
/ upsert_staff_from_account, all taking an AsyncSession + scraper-shaped dicts.
* Dialect-aware Core upsert: SQLite on_conflict_do_update for tests,
MySQL on_duplicate_key_update for prod; dialect detected via session.bind.name.
_build_upsert_stmt(model, values, dialect) is the reusable helper.
* Type coercion: _coerce_bool / _coerce_int (module-level, unit-tested) map
is_send/is_del/is_block/is_on_job int-as-bool to bool and gender/type
numeric strings to int; None preserved; raw_json kept verbatim.
* For M1 the Message model maps to a single physical `message` table;
per-month sharding (message_table_name helper) is a LATER task (YAGNI).
app/storage/object_store.py
* Local-filesystem ObjectStore with put_bytes / put_stream / put_by_hash /
exists / get_path, all dispatched via asyncio.to_thread so disk I/O is
genuinely async.
* put_by_hash computes sha256 and writes only if absent (content-addressed
dedup); layout is base_dir/<first 2 hash chars>/<hash><ext>.
* Default base dir is Settings.media_dir; the interface is the seam a
future MinIO/S3 backend will swap in behind.
tests: 24 new tests (in-memory SQLite for repos; tmp_path for object_store)
covering idempotency (same PK twice -> 1 row, updated), type coercion, batch
dedup (3 messages / 1 dup msg_id -> 2 rows), and hash dedup (corrupt-file
probe proves second put_by_hash does not rewrite). ruff clean; full suite
67/67 green.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type
normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py.
- app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text",
-2 -> "divider", everything else -> "unknown". Total (never raises);
unknown codes pass through so no real message is silently dropped.
- app/scraper/base.py: paginate_by_cursor() generic async generator. Takes
a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the
cursor and stops on has_more=False OR an empty page (defensive). Same
RateLimiter integration as paginate_by_page.
- app/scraper/wechat.py: scrape_wechat_messages() wraps
POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type
is overridden per request to application/x-www-form-urlencoded via the
YunkeClient.post(headers=...) override hook. Pages backward by setting
end = data.end - 1 each batch, repeats until hasLast=false. Drops any
record whose type/messageType is -2 (date divider UI chrome). Yields
normalized dicts (snake_case Message columns + channel="wechat" +
raw_json); the raw type code is preserved on raw_json for the
persistence layer (Task 1.7).
Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked
msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2
divider dropped), full field mapping vs the live-capture names, the
content-type is form-encoded (not JSON), and the 2nd request body's
end == E1 - 1 (proving the backward cursor advance).
Co-Authored-By: Claude <noreply@anthropic.com>
Add the first list scraper for the yunke data-mirror system, establishing
the reusable pagination pattern all later list scrapers will build on.
- app/scraper/base.py: generic `paginate_by_page` async generator that walks
page-numbered endpoints until a page comes back empty (the yunke CRM's
primary "no more rows" signal), with a defensive over-fetch ceiling keyed
off the declared total (strict `>` so an honest source yields exactly
`total`). Plus a tiny `RateLimiter` enforcing a min interval between calls.
- app/scraper/wechat.py: `scrape_wechat_friends` wraps
`POST /wechatweb/wxFriends/friend_list/es`, maps each record to the
`Customer` column shape (snake_case) with `channel="wechat"` and a verbatim
`raw_json` copy. Optional `department_id`/`user_id` filters pass through to
the request body.
- tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend
records + a response-envelope builder, reused by the scraper tests.
- tests/test_scrape_wechat_friends.py: drives the real pagination +
normalization logic through respx (page1 returns both records, page2 empty
-> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2
upstream calls.
- tests/test_scrape_base.py: smoke tests for the RateLimiter spacing.
Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7;
scrapers remain pure producers of normalized dicts.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 1.4 — 会话层. Adds the authenticated HTTP client that every yunke
scraper will use to call the CRM API.
- app/session/login.py: get_secure_key + pc_login wrappers, LoginResult
dataclass, and a clearly-marked Phase-0 swappable encrypt_password
placeholder (real RSA per getSecureKey is a TODO; drop-in contract
documented in the docstring).
- app/session/yunke_client.py: YunkeClient wrapping httpx.AsyncClient.
Constructor takes either (username+password) for auto-login or
(userid+company[+cookies]) for an existing session. Injects the 5
observed auth headers (userid/company/source=yunkepc/channel=4/
content-type=application/json) on every request, auto-logs-in on first
request, and on HTTP 401 (or body code 401) re-logs-in once and
retries — a persistent 401 raises after the single retry so there is
no infinite loop.
- Tests (respx): login populates LoginResult + httpx cookie jar; every
request carries all 5 auth headers + cookie; 401 triggers exactly one
re-login + retry and returns 200; persistent 401 raises after one
retry; first request with no session auto-logs-in.
Session is held in-memory only for now; persistence to encrypted
YkAccount.session_blob_enc is left as a later concern (TODO in
yunke_client.py).
Co-Authored-By: Claude <noreply@anthropic.com>
Implements the 14 core ORM models for the yunke data-mirror system using
SQLAlchemy 2.0 Mapped/mapped_column style, driven by the field lists in
docs/superpowers/specs/2026-08-01-yunke-data-dictionary.md:
Department, Staff, WechatAccount, WhatsappAccount, Customer, ChatGroup,
GroupMember, Message, Media, PhoneCall, Sms, SyncState, YkAccount, SyncRun.
Conventions:
- Natural source PKs where they exist (friend_id, wx_id, msg_id, room_id,
media_hash, call_id, sms_id, dept_id, user_id); surrogate Integer PKs
otherwise (SyncState, YkAccount, SyncRun). GroupMember uses a composite
(channel, room_id, member_id) PK.
- Common audit columns version/raw_json/fetched_at/updated_at on every
source-mirrored table.
- channel only where meaningful: Customer/Message/Media/ChatGroup/
GroupMember/Sms use messaging channels; PhoneCall.channel is call-type;
SyncState.channel scopes the cursor. Omitted on Staff/Department/
WechatAccount/WhatsappAccount/YkAccount/SyncRun.
- Message composite index on (channel, owner_account_id, talker_id,
timestamp) for the hot per-talker time-order query path.
- message_table_name(channel, timestamp) helper documents the future
message_<channel>_<YYYYMM> sharding convention; the Message model still
maps to a single physical table for now (sharding is a later task).
All types are portable generic SQLAlchemy types so tests run on in-memory
SQLite while prod targets MySQL. Tests verify Customer round-trip,
raw_json JSON round-trip, Message natural PK, GroupMember composite PK,
the Message composite index, and full table creation on SQLite.
Alembic wired via migrations/env.py to Base.metadata + get_settings().db_url
with an async->sync driver swap (mysql+asyncmy -> mysql+pymysql,
sqlite+aiosqlite -> sqlite). Initial migration autogenerated against an
empty SQLite DB; server defaults normalized to sa.func.now() so the file
is MySQL-portable. upgrade head is intentionally not run (no MySQL yet).
Co-Authored-By: Claude <noreply@anthropic.com>
Create pyproject.toml (FastAPI + SQLAlchemy + Playwright + APScheduler +
Redis + cryptography stack), .env.example, app/__init__.py and a
pydantic-settings Settings singleton (get_settings), and a root
.gitignore covering Python, Node, env, IDE and the local media dir.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念