WA list endpoints return wall-clock strings ('2026-08-01 16:39') for
lastUpdate; storing raw broke CustomerOut.last_chat_time (int) validation
-> 500 on /api/customers?channel=whatsapp. Now parsed via parse_datetime_to_ms.
Co-Authored-By: Claude <noreply@anthropic.com>
Live validation: businessFriendList uses 'page' (not pageNum) and REQUIRES
departmentId (root) + type or returns empty. Same class of bug as wxUser/list.
Co-Authored-By: Claude <noreply@anthropic.com>
Replicates the Customer all-fields pattern across the remaining three
entities. Backend now exposes `raw_json` on MessageOut / StaffOut /
AccountOut so the frontend can render every 云客 source field without a
second endpoint. Each entity gains a Chinese-label field utility + an
el-drawer surfacing all fields (structured-first, raw_json fallback,
unknown keys under 其他).
Backend:
- schemas.py: add `raw_json: dict | None` to MessageOut / StaffOut /
AccountOut. MessageOut/StaffOut pick the column up via from_attributes;
AccountOut is built explicitly in the router, which now threads
`raw_json` through from both WechatAccount and WhatsappAccount rows.
- tests: new raw_json serialization tests for messages, staff, accounts
(assert populated dict round-trips verbatim + NULL serializes as null).
Frontend (yunkes/):
- New utils: messageFields.ts, staffFields.ts, accountFields.ts — each
with a 中文 label map, ordered groups, structured-column preference,
and an 其他 fallback for unknown raw_json keys.
- Message: clicking a chat bubble opens an all-fields drawer (referMsg
renders as pretty-printed JSON). Image preview + file download links
stopPropagation so their own interactions still work.
- Staff: row click → drawer; columns relabeled (员工ID, 销售手机).
- Accounts: new AccountList.vue at /accounts with a 账号 nav item,
unified WeChat/WhatsApp table (渠道/账号ID/昵称/微信号/归属员工/好友数/
群数/收发/在职/最后更新), channel filter, pagination, row-click drawer.
- api: add Account type + getAccounts(); add raw_json to Message/Staff.
Co-Authored-By: Claude <noreply@anthropic.com>
Backend:
- CustomerOut now exposes raw_json (verbatim upstream friend record) so the
frontend can render every 云客 field without per-field structured columns.
- The verbatim blob round-trips via from_attributes; no model change needed.
Frontend:
- CustomerList table mirrors the 云客 客户列表 column set (头像/微信昵称/微信号/
备注/真实姓名/手机/性别/地区/描述/销售员工/部门/来源/添加方式/添加时间/最近
聊天/消息数/渠道). A readField helper reads the structured column first and
falls back to raw_json.<camelCase>.
- Row click opens an all-fields drawer (right side) rendering EVERY field of
the record with Chinese labels, grouped (基本信息/联系方式/归属员工与部门/时间
与来源/消息统计/好友状态/其他). Timestamps format to local datetime, flags
render 是/否, image URLs render as el-image thumbnails, empty values show —.
- New src/utils/customerFields.ts holds the camelCase→中文 label map, the
structured-column alias map, group definitions, and value-type formatters;
unknown raw_json keys auto-flow into an 其他 section so new upstream fields
surface without code changes.
- Drawer 查看聊天记录 action navigates to /chat/:channel/:ownerId/:talkerId
(owner_account_id + friend_id + channel), preserving the existing path.
Co-Authored-By: Claude <noreply@anthropic.com>
Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
isOnJob / departmentId / startdate / enddate + empty userId/salesname/
weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.
Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.
Co-Authored-By: Claude <noreply@anthropic.com>
Live validation found two real bugs:
- YunkeClient sent 'source: yunkepc' on every request; msg/chats binds
source as Integer -> 500. Removed (cookie auth suffices, per Phase 0).
- WeChat message talker_id used the masked 'talker' field; the timeline
joins on the full peer id. Now: 1:1 -> oriTalker (full), group -> talker
(member). ori_talker keeps the raw oriTalker (peer/chatroom).
Co-Authored-By: Claude <noreply@anthropic.com>
Backend (Part A):
- MessageOut: add file_th_url, file_hd_url, duration, transcription_text, md5
(room_id + sender_head_url already existed) so the chat timeline can render
image thumbnails, voice duration/transcription, and avatars from a single
response.
- PhoneCallOut: add record_file so the Calls view can offer recording playback.
- SmsOut: add content (full body) so the SMS detail dialog shows the whole text.
- Tests: red->green for each new field (media fields default None + populated,
record_file round-trip, content round-trip). Suite 267 -> 271.
Frontend (Part B):
- api/index.ts: wrappers getGroups/getGroupMembers/getCalls/getSms/getSyncStatus
+ Group/GroupMember/PhoneCall/Sms/SyncState/SyncRun/SyncStatus types in
types.ts (mirror the *Out pydantic schemas).
- views/GroupList.vue (/groups): table + channel filter + pagination; row click
opens the group timeline.
- views/CallList.vue (/calls): table + call-type filter (phone_call/voice_call/
all); "播放录音" action opens an el-dialog with <audio controls>.
- views/SmsList.vue (/sms): table; row click opens a detail dialog with the full
content.
- views/SyncStatus.vue (/sync): two tables (states + recent_runs) with a manual
refresh; run status color-coded.
- ChatTimeline.vue: reused for both 1:1 DM (talkerId) and group (roomId) by
branching the query; group title + per-member sender already handled by
MessageBubble.
- router: add /groups /calls /sms /sync + /group/:channel/:ownerId/:roomId.
- layouts/MainLayout.vue: replace the single 客户列表 button with an el-menu
(客户 / 群 / 通话 / 短信 / 同步); org tree kept on the left.
Co-Authored-By: Claude <noreply@anthropic.com>
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.
- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
scrape_whatsapp_friends (page-number pagination over the
data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
(backward-cursor pagination, JSON body {uid,friendUid,start,end} —
NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
1->image, -2->divider, else unknown; never raises) and a shared
coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
also upserts the embedded Staff(user_id, name) via the existing
upsert_staff_from_account helper; WhatsappAccount imported.
Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.
Suite: 182 -> 211 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Cleanup after successful live validation (10 real customers scraped+stored):
- app/config.py: add yk_userid / yk_company / yk_cookie Settings fields so
existing-session creds can be supplied via .env (bypasses programmatic
login for validation with a live browser session).
- app/storage/repos.py: coerce list values to ';'-joined strings in
_build_column_values. Fixes a real bug: the live friends-list API returns
Customer.deptRoute as a LIST, but dept_route is a scalar String(255)
column. raw_json still preserves the original list (no data loss).
- scripts/smoke_wechat.py:
* Revert build_client_from_env to read os.environ ONLY (no get_settings
fallbacks), so the monkeypatch.setenv unit tests stay hermetic and the
real .env stops leaking in. 3 previously-red tests green again.
* Add _push_settings_into_environ() (Settings -> os.environ, real env
wins, only fills gaps) and call it as the first line of main(); so the
smoke run still honors .env without the pure builder depending on
Settings. .env -> Settings -> os.environ (once) -> build_client_from_env.
* Keep the await _print_summary fix from validation.
- tests/test_repos.py: regression test for list dept_route coercion
(stored joined 'A;B', raw_json keeps the list).
Co-Authored-By: Claude <noreply@anthropic.com>
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds a shared-token auth dependency for the viewing API (Task 2.9). Trust
model: when API_TOKEN is empty (the default) auth is disabled — we trust the
network boundary (local/intranet tool). When set, every request must present
the token via `Authorization: Bearer <token>` or `?token=<token>`, else 401.
`/api/health` is exempt so liveness probes work without the token.
- app/config.py: add `api_token: str = ""` to Settings.
- app/api/auth.py: `require_auth` dependency (secrets.compare_digest, reads
get_settings() per-request so token rotation via cache_clear() works).
- app/api/main.py: apply the dependency globally; document the trust model.
- .env.example: document API_TOKEN.
- tests/test_api_auth.py: 8 tests covering disabled/correct/missing/wrong/
malformed-header/query-param/health-exempt paths.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 2.7 — the coordination layer tying scrapers + repos together with
resumable, incremental sync.
app/sync/watermarks.py — SyncState CRUD keyed on (entity_type, channel,
scope_id): get_watermark / get_last_timestamp / get_last_version /
set_watermark (upsert). A partial advance (timestamp-only) does not clobber
the other cursor with NULL.
app/sync/orchestrator.py — the sync brain:
* sync_wechat_messages: time-window incremental. Reads last_timestamp for
(message, wechat, "<wechat_id>:<friend_id>"), queries [last_ts, now] via
scrape_wechat_messages, upserts each, best-effort downloads media, advances
the watermark to max(timestamp). A re-run with no new messages queries an
empty window and fetches nothing — the watermark advanced past.
* sync_{departments,wechat_accounts,wechat_friends,wechat_groups,
wechat_voice_calls}: full re-scrape + idempotent upsert (these list APIs
have no since-param; 'incremental' = re-scrape all, rely on PK idempotency).
Watermark records last_synced_at only.
* sync_all_wechat: coordinator running departments -> accounts -> friends ->
groups -> voice_calls, then per-conversation messages for an explicit
(wechat_id, friend_id) list. Auto-discovery deferred.
Each run logs a SyncRun row (started -> success/failed, counts, error).
Failure policy: a scraper that raises is caught, recorded as failed, and a
failed SyncResult is returned — the orchestrator does not re-raise, so one
broken entity does not abort the batch. Media-download failures are
best-effort. The orchestrator never commits; callers own the transaction.
Tests (TDD red -> green; in-memory SQLite; scrapers/repos monkeypatched):
- watermarks: set/get, upsert (same key twice -> 1 row), partial-advance
preserves the other cursor, defaults when absent.
- orchestrator messages: first run upserts 2 + advances watermark to T2;
second run scrape called with start==T2 and upserts 0 (incremental proof);
media failure does not abort; SyncRun success recorded.
- orchestrator friends: 3 records upserted, SyncRun success, watermark
last_synced_at set.
- failure policy: scraper raises -> SyncRun failed + failed result, no raise
(both friends and message paths).
- sync_all_wechat: 5 entities + 2 message runs in order, all success.
Suite: 149 -> 163 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
M2 Task 2.6 (media downloader) + Task 2.8 (media serve API).
Downloader (app/scraper/media.py):
- download_url(client, url): fetches bytes via the YunkeClient's underlying
httpx client (carries session cookies; respx-mockable; follows redirects).
- download_message_media(client, msg, store, session): URL-driven fetcher that
picks the best http(s) URL (file_url -> file_hd_url -> file_th_url), dedups
by md5 (skips the network when a Media row with local file already exists),
falls back to sha256(bytes) when no md5, stores via ObjectStore.put_by_hash,
and upserts a Media row. Skips type-10 system messages (non-media per
phase0-findings §2) and type-3 opaque-id / text (no http url). Resilient:
download errors are logged and return None (no raise).
- media_type_from_message / guess_mime_type helpers.
Repo (app/storage/repos.py): upsert_media — idempotent Core upsert into Media by
media_hash. Volatile download artifacts (local_path/size_bytes/mime_type) use
COALESCE(incoming, existing) on conflict so a re-scrape that did not re-download
does not clobber previously stored values. Exported from app.storage.
Serve API (app/api/routers/media.py): GET /api/media/{media_hash} streams the
local file via FileResponse (handles HTTP Range / 206). 404 when the row is
absent, has no local file, or the file is missing on disk. Content-type from the
row's mime_type, falling back to ext-derived guess.
Tests (+19, suite 130 -> 149, all green; ruff clean):
- test_media_downloader.py: image/video download + upsert; md5 dedup skips
second network hit; sha256 fallback when no md5; type-3/text/system -> None;
download failure -> None (no raise); upsert_media idempotency +
null-local_path preservation + 3x-same-key-one-row.
- test_api_media.py: 200 + content-type + bytes; 404 unknown hash; 404 missing
file on disk; Range -> 206 partial; ext-derived content-type when mime NULL.
Co-Authored-By: Claude <noreply@anthropic.com>
- scrape_wechat_voice_calls: wraps POST /wechatweb/callRecord/queryWeChatVoiceList
with page-number pagination (reuses paginate_by_page); normalizes records to
the PhoneCall column shape with channel='voice_call'. Field mapping per
phase0-findings §4: id->call_id, duration(str s)->call_duration(int),
startTime(YYYY-MM-DD HH:MM:SS)->start_call_time(ms epoch), callStatus->
call_status, talker->call_number, talkerNickName->plan_customer_name,
ossFileName->record_file (may be null). isSend (0=in/1=out) is inverted to
incoming_call (1=in/0=out) so the column's meaning matches the phone_call
channel.
- upsert_voice_call: idempotent Core upsert into PhoneCall by call_id
(sqlite/mysql dialect-aware). Adds call_duration/start_call_time to the
_INT_FIELDS coercion net (scraper already emits ints; defensive for
hand-built records).
- Group-message test (Task 2.3): locks in that scrape_wechat_messages already
handles chatroom conversations (friendId=@chatroom, roomid set, talker=member
wxid) — yields room_id populated and talker_id = the sending member. No
scraper change required.
Suite: 123 -> 130 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Add scrape_wechat_groups (POST /wechatweb/chatroom/getChatroomList) and
scrape_wechat_talkers (POST /wechatweb/wxFriends/talkerList), both reusing
the page-number paginator. Groups parse the semicolon-joined memberIds into
a member_ids_list so upsert_chat_group can mirror GroupMember rows
(composite PK channel/room_id/member_id) idempotently alongside the
ChatGroup row. Talkers are a sync-planning input (member_count>0 marks a
group conversation); no dedicated table.
Co-Authored-By: Claude <noreply@anthropic.com>
M2 organization scrapers for yunke-server:
- scrape_departments: POST /usercenter/dept/tree. Single request (no
pagination). Defensively unwraps the envelope (data is node list OR an
object wrapping nodes under children), flattens BOTH nested and flat
shapes, and reconstructs the materialized path by walking the pId chain
(ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.
- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
pagination via the existing paginate_by_page helper. Envelope differs
from friend_list (no basicResult wrapper): data.data/data.total. Maps
wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
WechatAccount columns; carries embedded staff pair for the repo layer.
Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
forward phone/sale_phone/is_on_job when present.
Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).
Suite: 97 -> 111 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace the Phase-0 placeholder password encryption with real RSA
(PKCS#1 v1.5). encrypt_password now builds an RSA public key from the
getSecureKey modulus/exponent (RSAPublicNumbers, e=int(hex,16),
n=int(hex,16)), encrypts the UTF-8 password, and returns hex (documented
assumption -- base64 is the one-line fallback noted in the docstring).
get_secure_key now returns the inner data dict to match the new contract;
pc_login keeps {account, password} with a comment on the username/password
and account/pwd fallbacks if pcLogin rejects.
Extend normalize_wechat_type with the Phase-0 confirmed codes:
2 -> image, 3 -> image_legacy, 4 -> video, 10 -> system (1 -> text and
-2 -> divider unchanged; 6/9/13/15/21/22 stay unknown until confirmed).
Tests:
- encrypt_password round-trip: generate an RSA keypair, build secure_key
from its public numbers, encrypt, then decrypt with the private key and
assert the original password comes back. Pins 512-hex output for a
2048-bit key. No network.
- Parser parametrize table extended for the new codes (and a downstream
message-fixture assertion updated from unknown -> image_legacy).
Suite: 87 -> 97 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Wires the existing pieces together into a runnable integration check:
scrape ~10 WeChat friends + ~50 of their messages (last 7 days) from the
real yunke CRM (read-only), upsert into a local SQLite smoke DB, then
print a summary that verifies rows landed.
Two auth modes:
* Existing-session (preferred): YK_USERID/YK_COMPANY[/YK_COOKIE] -> YunkeClient
constructed from a captured session, login bypassed.
* Login mode (Phase-0 blocked): YUNKE_USERNAME/YUNKE_PASSWORD -> attempts
client.login(), which the real CRM will reject until the placeholder
password encryption is replaced. Prints a clear message and exits 3.
build_client_from_env() is split out as a pure, network-free helper so
auth-mode selection is unit-testable. main() is async (asyncio.run in
__main__). HTTPStatusError from a 401/expired session is wrapped into a
helpful message + exit 4 (no traceback). client.aclose() + engine.dispose()
always run in finally.
Defaults to SMOKE_DB_URL=sqlite+aiosqlite:///./smoke.db so no MySQL is
required. Run from yunke-server/ with the venv active:
python -m scripts.smoke_wechat
Tests cover the entrypoint shape and auth-mode selection (no network).
Suite: 77 -> 87 passing.
Co-Authored-By: Claude <noreply@anthropic.com>
Minimal read-only FastAPI surface over the mirrored DB:
- GET /api/customers with pagination (page/page_size) and optional
channel / owner_account_id / department_id filters, ordered by
last_chat_time desc (nulls last) then friend_id.
- GET /api/health liveness probe.
- Pydantic v2 CustomerOut + generic Page[T] envelope.
- CORS configured permissively for the local frontend.
- Tests override get_session to an in-memory async SQLite session
(StaticPool) and drive the app via httpx ASGITransport.
Co-Authored-By: Claude <noreply@anthropic.com>
The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the storage layer that bridges scrapers (which yield normalized dicts)
to the DB and the local media store.
app/storage/repos.py
* upsert_customer / upsert_customers / upsert_messages / upsert_wechat_account
/ upsert_staff_from_account, all taking an AsyncSession + scraper-shaped dicts.
* Dialect-aware Core upsert: SQLite on_conflict_do_update for tests,
MySQL on_duplicate_key_update for prod; dialect detected via session.bind.name.
_build_upsert_stmt(model, values, dialect) is the reusable helper.
* Type coercion: _coerce_bool / _coerce_int (module-level, unit-tested) map
is_send/is_del/is_block/is_on_job int-as-bool to bool and gender/type
numeric strings to int; None preserved; raw_json kept verbatim.
* For M1 the Message model maps to a single physical `message` table;
per-month sharding (message_table_name helper) is a LATER task (YAGNI).
app/storage/object_store.py
* Local-filesystem ObjectStore with put_bytes / put_stream / put_by_hash /
exists / get_path, all dispatched via asyncio.to_thread so disk I/O is
genuinely async.
* put_by_hash computes sha256 and writes only if absent (content-addressed
dedup); layout is base_dir/<first 2 hash chars>/<hash><ext>.
* Default base dir is Settings.media_dir; the interface is the seam a
future MinIO/S3 backend will swap in behind.
tests: 24 new tests (in-memory SQLite for repos; tmp_path for object_store)
covering idempotency (same PK twice -> 1 row, updated), type coercion, batch
dedup (3 messages / 1 dup msg_id -> 2 rows), and hash dedup (corrupt-file
probe proves second put_by_hash does not rewrite). ruff clean; full suite
67/67 green.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type
normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py.
- app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text",
-2 -> "divider", everything else -> "unknown". Total (never raises);
unknown codes pass through so no real message is silently dropped.
- app/scraper/base.py: paginate_by_cursor() generic async generator. Takes
a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the
cursor and stops on has_more=False OR an empty page (defensive). Same
RateLimiter integration as paginate_by_page.
- app/scraper/wechat.py: scrape_wechat_messages() wraps
POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type
is overridden per request to application/x-www-form-urlencoded via the
YunkeClient.post(headers=...) override hook. Pages backward by setting
end = data.end - 1 each batch, repeats until hasLast=false. Drops any
record whose type/messageType is -2 (date divider UI chrome). Yields
normalized dicts (snake_case Message columns + channel="wechat" +
raw_json); the raw type code is preserved on raw_json for the
persistence layer (Task 1.7).
Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked
msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2
divider dropped), full field mapping vs the live-capture names, the
content-type is form-encoded (not JSON), and the 2nd request body's
end == E1 - 1 (proving the backward cursor advance).
Co-Authored-By: Claude <noreply@anthropic.com>
Add the first list scraper for the yunke data-mirror system, establishing
the reusable pagination pattern all later list scrapers will build on.
- app/scraper/base.py: generic `paginate_by_page` async generator that walks
page-numbered endpoints until a page comes back empty (the yunke CRM's
primary "no more rows" signal), with a defensive over-fetch ceiling keyed
off the declared total (strict `>` so an honest source yields exactly
`total`). Plus a tiny `RateLimiter` enforcing a min interval between calls.
- app/scraper/wechat.py: `scrape_wechat_friends` wraps
`POST /wechatweb/wxFriends/friend_list/es`, maps each record to the
`Customer` column shape (snake_case) with `channel="wechat"` and a verbatim
`raw_json` copy. Optional `department_id`/`user_id` filters pass through to
the request body.
- tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend
records + a response-envelope builder, reused by the scraper tests.
- tests/test_scrape_wechat_friends.py: drives the real pagination +
normalization logic through respx (page1 returns both records, page2 empty
-> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2
upstream calls.
- tests/test_scrape_base.py: smoke tests for the RateLimiter spacing.
Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7;
scrapers remain pure producers of normalized dicts.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 1.4 — 会话层. Adds the authenticated HTTP client that every yunke
scraper will use to call the CRM API.
- app/session/login.py: get_secure_key + pc_login wrappers, LoginResult
dataclass, and a clearly-marked Phase-0 swappable encrypt_password
placeholder (real RSA per getSecureKey is a TODO; drop-in contract
documented in the docstring).
- app/session/yunke_client.py: YunkeClient wrapping httpx.AsyncClient.
Constructor takes either (username+password) for auto-login or
(userid+company[+cookies]) for an existing session. Injects the 5
observed auth headers (userid/company/source=yunkepc/channel=4/
content-type=application/json) on every request, auto-logs-in on first
request, and on HTTP 401 (or body code 401) re-logs-in once and
retries — a persistent 401 raises after the single retry so there is
no infinite loop.
- Tests (respx): login populates LoginResult + httpx cookie jar; every
request carries all 5 auth headers + cookie; 401 triggers exactly one
re-login + retry and returns 200; persistent 401 raises after one
retry; first request with no session auto-logs-in.
Session is held in-memory only for now; persistence to encrypted
YkAccount.session_blob_enc is left as a later concern (TODO in
yunke_client.py).
Co-Authored-By: Claude <noreply@anthropic.com>
Implements the 14 core ORM models for the yunke data-mirror system using
SQLAlchemy 2.0 Mapped/mapped_column style, driven by the field lists in
docs/superpowers/specs/2026-08-01-yunke-data-dictionary.md:
Department, Staff, WechatAccount, WhatsappAccount, Customer, ChatGroup,
GroupMember, Message, Media, PhoneCall, Sms, SyncState, YkAccount, SyncRun.
Conventions:
- Natural source PKs where they exist (friend_id, wx_id, msg_id, room_id,
media_hash, call_id, sms_id, dept_id, user_id); surrogate Integer PKs
otherwise (SyncState, YkAccount, SyncRun). GroupMember uses a composite
(channel, room_id, member_id) PK.
- Common audit columns version/raw_json/fetched_at/updated_at on every
source-mirrored table.
- channel only where meaningful: Customer/Message/Media/ChatGroup/
GroupMember/Sms use messaging channels; PhoneCall.channel is call-type;
SyncState.channel scopes the cursor. Omitted on Staff/Department/
WechatAccount/WhatsappAccount/YkAccount/SyncRun.
- Message composite index on (channel, owner_account_id, talker_id,
timestamp) for the hot per-talker time-order query path.
- message_table_name(channel, timestamp) helper documents the future
message_<channel>_<YYYYMM> sharding convention; the Message model still
maps to a single physical table for now (sharding is a later task).
All types are portable generic SQLAlchemy types so tests run on in-memory
SQLite while prod targets MySQL. Tests verify Customer round-trip,
raw_json JSON round-trip, Message natural PK, GroupMember composite PK,
the Message composite index, and full table creation on SQLite.
Alembic wired via migrations/env.py to Base.metadata + get_settings().db_url
with an async->sync driver swap (mysql+asyncmy -> mysql+pymysql,
sqlite+aiosqlite -> sqlite). Initial migration autogenerated against an
empty SQLite DB; server defaults normalized to sa.func.now() so the file
is MySQL-portable. upgrade head is intentionally not run (no MySQL yet).
Co-Authored-By: Claude <noreply@anthropic.com>
Create pyproject.toml (FastAPI + SQLAlchemy + Playwright + APScheduler +
Redis + cryptography stack), .env.example, app/__init__.py and a
pydantic-settings Settings singleton (get_settings), and a root
.gitignore covering Python, Node, env, IDE and the local media dir.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念