Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
isOnJob / departmentId / startdate / enddate + empty userId/salesname/
weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.
Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.
Co-Authored-By: Claude <noreply@anthropic.com>
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type
normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py.
- app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text",
-2 -> "divider", everything else -> "unknown". Total (never raises);
unknown codes pass through so no real message is silently dropped.
- app/scraper/base.py: paginate_by_cursor() generic async generator. Takes
a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the
cursor and stops on has_more=False OR an empty page (defensive). Same
RateLimiter integration as paginate_by_page.
- app/scraper/wechat.py: scrape_wechat_messages() wraps
POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type
is overridden per request to application/x-www-form-urlencoded via the
YunkeClient.post(headers=...) override hook. Pages backward by setting
end = data.end - 1 each batch, repeats until hasLast=false. Drops any
record whose type/messageType is -2 (date divider UI chrome). Yields
normalized dicts (snake_case Message columns + channel="wechat" +
raw_json); the raw type code is preserved on raw_json for the
persistence layer (Task 1.7).
Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked
msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2
divider dropped), full field mapping vs the live-capture names, the
content-type is form-encoded (not JSON), and the 2nd request body's
end == E1 - 1 (proving the backward cursor advance).
Co-Authored-By: Claude <noreply@anthropic.com>
Add the first list scraper for the yunke data-mirror system, establishing
the reusable pagination pattern all later list scrapers will build on.
- app/scraper/base.py: generic `paginate_by_page` async generator that walks
page-numbered endpoints until a page comes back empty (the yunke CRM's
primary "no more rows" signal), with a defensive over-fetch ceiling keyed
off the declared total (strict `>` so an honest source yields exactly
`total`). Plus a tiny `RateLimiter` enforcing a min interval between calls.
- app/scraper/wechat.py: `scrape_wechat_friends` wraps
`POST /wechatweb/wxFriends/friend_list/es`, maps each record to the
`Customer` column shape (snake_case) with `channel="wechat"` and a verbatim
`raw_json` copy. Optional `department_id`/`user_id` filters pass through to
the request body.
- tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend
records + a response-envelope builder, reused by the scraper tests.
- tests/test_scrape_wechat_friends.py: drives the real pagination +
normalization logic through respx (page1 returns both records, page2 empty
-> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2
upstream calls.
- tests/test_scrape_base.py: smoke tests for the RateLimiter spacing.
Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7;
scrapers remain pure producers of normalized dicts.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念