浅念
4a47e824c7
Add the first list scraper for the yunke data-mirror system, establishing the reusable pagination pattern all later list scrapers will build on. - app/scraper/base.py: generic `paginate_by_page` async generator that walks page-numbered endpoints until a page comes back empty (the yunke CRM's primary "no more rows" signal), with a defensive over-fetch ceiling keyed off the declared total (strict `>` so an honest source yields exactly `total`). Plus a tiny `RateLimiter` enforcing a min interval between calls. - app/scraper/wechat.py: `scrape_wechat_friends` wraps `POST /wechatweb/wxFriends/friend_list/es`, maps each record to the `Customer` column shape (snake_case) with `channel="wechat"` and a verbatim `raw_json` copy. Optional `department_id`/`user_id` filters pass through to the request body. - tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend records + a response-envelope builder, reused by the scraper tests. - tests/test_scrape_wechat_friends.py: drives the real pagination + normalization logic through respx (page1 returns both records, page2 empty -> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2 upstream calls. - tests/test_scrape_base.py: smoke tests for the RateLimiter spacing. Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7; scrapers remain pure producers of normalized dicts. Co-Authored-By: Claude <noreply@anthropic.com>
13 lines
604 B
Python
13 lines
604 B
Python
"""Scraper layer for the yunke data-mirror system.
|
|
|
|
Scrapers sit between the session layer (:mod:`app.session.yunke_client`) and
|
|
the persistence layer (a LATER task). Each scraper is an async generator that
|
|
walks one upstream list endpoint and yields **normalized dicts** —
|
|
snake_case keys matching the target ORM model columns, plus a ``raw_json`` key
|
|
holding the original upstream record verbatim.
|
|
|
|
Persistence (upsert into ORM tables) is intentionally out of scope here: it is
|
|
Task 1.7. Keeping scrapers pure producers lets us test ingestion logic against
|
|
``respx`` fixtures without a database.
|
|
"""
|