Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
isOnJob / departmentId / startdate / enddate + empty userId/salesname/
weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.
Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.
Co-Authored-By: Claude <noreply@anthropic.com>
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.
- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
scrape_whatsapp_friends (page-number pagination over the
data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
(backward-cursor pagination, JSON body {uid,friendUid,start,end} —
NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
1->image, -2->divider, else unknown; never raises) and a shared
coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
also upserts the embedded Staff(user_id, name) via the existing
upsert_staff_from_account helper; WhatsappAccount imported.
Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.
Suite: 182 -> 211 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
- scrape_wechat_voice_calls: wraps POST /wechatweb/callRecord/queryWeChatVoiceList
with page-number pagination (reuses paginate_by_page); normalizes records to
the PhoneCall column shape with channel='voice_call'. Field mapping per
phase0-findings §4: id->call_id, duration(str s)->call_duration(int),
startTime(YYYY-MM-DD HH:MM:SS)->start_call_time(ms epoch), callStatus->
call_status, talker->call_number, talkerNickName->plan_customer_name,
ossFileName->record_file (may be null). isSend (0=in/1=out) is inverted to
incoming_call (1=in/0=out) so the column's meaning matches the phone_call
channel.
- upsert_voice_call: idempotent Core upsert into PhoneCall by call_id
(sqlite/mysql dialect-aware). Adds call_duration/start_call_time to the
_INT_FIELDS coercion net (scraper already emits ints; defensive for
hand-built records).
- Group-message test (Task 2.3): locks in that scrape_wechat_messages already
handles chatroom conversations (friendId=@chatroom, roomid set, talker=member
wxid) — yields room_id populated and talker_id = the sending member. No
scraper change required.
Suite: 123 -> 130 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Add scrape_wechat_groups (POST /wechatweb/chatroom/getChatroomList) and
scrape_wechat_talkers (POST /wechatweb/wxFriends/talkerList), both reusing
the page-number paginator. Groups parse the semicolon-joined memberIds into
a member_ids_list so upsert_chat_group can mirror GroupMember rows
(composite PK channel/room_id/member_id) idempotently alongside the
ChatGroup row. Talkers are a sync-planning input (member_count>0 marks a
group conversation); no dedicated table.
Co-Authored-By: Claude <noreply@anthropic.com>
M2 organization scrapers for yunke-server:
- scrape_departments: POST /usercenter/dept/tree. Single request (no
pagination). Defensively unwraps the envelope (data is node list OR an
object wrapping nodes under children), flattens BOTH nested and flat
shapes, and reconstructs the materialized path by walking the pId chain
(ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.
- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
pagination via the existing paginate_by_page helper. Envelope differs
from friend_list (no basicResult wrapper): data.data/data.total. Maps
wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
WechatAccount columns; carries embedded staff pair for the repo layer.
Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
forward phone/sale_phone/is_on_job when present.
Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).
Suite: 97 -> 111 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type
normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py.
- app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text",
-2 -> "divider", everything else -> "unknown". Total (never raises);
unknown codes pass through so no real message is silently dropped.
- app/scraper/base.py: paginate_by_cursor() generic async generator. Takes
a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the
cursor and stops on has_more=False OR an empty page (defensive). Same
RateLimiter integration as paginate_by_page.
- app/scraper/wechat.py: scrape_wechat_messages() wraps
POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type
is overridden per request to application/x-www-form-urlencoded via the
YunkeClient.post(headers=...) override hook. Pages backward by setting
end = data.end - 1 each batch, repeats until hasLast=false. Drops any
record whose type/messageType is -2 (date divider UI chrome). Yields
normalized dicts (snake_case Message columns + channel="wechat" +
raw_json); the raw type code is preserved on raw_json for the
persistence layer (Task 1.7).
Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked
msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2
divider dropped), full field mapping vs the live-capture names, the
content-type is form-encoded (not JSON), and the 2nd request body's
end == E1 - 1 (proving the backward cursor advance).
Co-Authored-By: Claude <noreply@anthropic.com>
Add the first list scraper for the yunke data-mirror system, establishing
the reusable pagination pattern all later list scrapers will build on.
- app/scraper/base.py: generic `paginate_by_page` async generator that walks
page-numbered endpoints until a page comes back empty (the yunke CRM's
primary "no more rows" signal), with a defensive over-fetch ceiling keyed
off the declared total (strict `>` so an honest source yields exactly
`total`). Plus a tiny `RateLimiter` enforcing a min interval between calls.
- app/scraper/wechat.py: `scrape_wechat_friends` wraps
`POST /wechatweb/wxFriends/friend_list/es`, maps each record to the
`Customer` column shape (snake_case) with `channel="wechat"` and a verbatim
`raw_json` copy. Optional `department_id`/`user_id` filters pass through to
the request body.
- tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend
records + a response-envelope builder, reused by the scraper tests.
- tests/test_scrape_wechat_friends.py: drives the real pagination +
normalization logic through respx (page1 returns both records, page2 empty
-> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2
upstream calls.
- tests/test_scrape_base.py: smoke tests for the RateLimiter spacing.
Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7;
scrapers remain pure producers of normalized dicts.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念