feat: wechat friend scraper
Add the first list scraper for the yunke data-mirror system, establishing the reusable pagination pattern all later list scrapers will build on. - app/scraper/base.py: generic `paginate_by_page` async generator that walks page-numbered endpoints until a page comes back empty (the yunke CRM's primary "no more rows" signal), with a defensive over-fetch ceiling keyed off the declared total (strict `>` so an honest source yields exactly `total`). Plus a tiny `RateLimiter` enforcing a min interval between calls. - app/scraper/wechat.py: `scrape_wechat_friends` wraps `POST /wechatweb/wxFriends/friend_list/es`, maps each record to the `Customer` column shape (snake_case) with `channel="wechat"` and a verbatim `raw_json` copy. Optional `department_id`/`user_id` filters pass through to the request body. - tests/cassettes/wxFriends_fixture.py: two realistic, capture-shaped friend records + a response-envelope builder, reused by the scraper tests. - tests/test_scrape_wechat_friends.py: drives the real pagination + normalization logic through respx (page1 returns both records, page2 empty -> stop). Asserts exact field mappings, `raw_json` fidelity, and exactly 2 upstream calls. - tests/test_scrape_base.py: smoke tests for the RateLimiter spacing. Persistence (upsert into `Customer`) is deliberately deferred to Task 1.7; scrapers remain pure producers of normalized dicts. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
@@ -0,0 +1,12 @@
|
||||
"""Scraper layer for the yunke data-mirror system.
|
||||
|
||||
Scrapers sit between the session layer (:mod:`app.session.yunke_client`) and
|
||||
the persistence layer (a LATER task). Each scraper is an async generator that
|
||||
walks one upstream list endpoint and yields **normalized dicts** —
|
||||
snake_case keys matching the target ORM model columns, plus a ``raw_json`` key
|
||||
holding the original upstream record verbatim.
|
||||
|
||||
Persistence (upsert into ORM tables) is intentionally out of scope here: it is
|
||||
Task 1.7. Keeping scrapers pure producers lets us test ingestion logic against
|
||||
``respx`` fixtures without a database.
|
||||
"""
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Reusable building blocks for list scrapers.
|
||||
|
||||
This module establishes the pagination pattern every list scraper in the
|
||||
project reuses (see design doc §6.2 拉取层). It is intentionally small:
|
||||
|
||||
* :func:`paginate_by_page` — a generic async generator that walks page-numbered
|
||||
endpoints until a page comes back empty (the primary stop signal used by the
|
||||
yunke CRM), with a defensive over-fetch ceiling keyed off the declared total.
|
||||
* :class:`RateLimiter` — a tiny asyncio limiter that enforces a minimum
|
||||
interval between successive calls. Scrapers that need to be polite to the
|
||||
upstream construct one and ``await`` :meth:`acquire` before each call.
|
||||
|
||||
The paginator talks to :class:`app.session.yunke_client.YunkeClient` (which
|
||||
already injects auth headers + handles 401 re-login). It is generic over the
|
||||
response shape via three callables:
|
||||
|
||||
* ``parse_total(resp) -> int | None`` — read the declared row count from the
|
||||
envelope (used only for the over-fetch safety; ``None`` disables it).
|
||||
* ``parse_rows(resp) -> list[dict]`` — extract the list of records.
|
||||
* ``build_body(page_num, page_size, **extra) -> dict`` — build the request
|
||||
body for a given page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from typing import Any
|
||||
|
||||
from app.session.yunke_client import YunkeClient
|
||||
|
||||
__all__ = ["RateLimiter", "paginate_by_page"]
|
||||
|
||||
# Informal type aliases for the three response-handling callables. The yunke
|
||||
# response shapes are dict-shaped JSON and do not benefit from finer typing.
|
||||
ParseTotal = Callable[[Any], "int | None"]
|
||||
ParseRows = Callable[[Any], "list[dict[str, Any]]"]
|
||||
BuildBody = Callable[..., "dict[str, Any]"]
|
||||
|
||||
|
||||
async def paginate_by_page(
|
||||
client: YunkeClient,
|
||||
path: str,
|
||||
*,
|
||||
page_size: int,
|
||||
parse_total: ParseTotal,
|
||||
parse_rows: ParseRows,
|
||||
build_body: BuildBody,
|
||||
rate_limiter: RateLimiter | None = None,
|
||||
**extra: Any,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Walk a page-numbered list endpoint, yielding one dict per record.
|
||||
|
||||
Pagination strategy
|
||||
-------------------
|
||||
Start at ``pageNum=1`` and advance while pages are non-empty. The yunke
|
||||
CRM signals "no more rows" by returning an empty ``data`` array on the
|
||||
page *after* the last real record; that empty page is the **primary stop
|
||||
signal**. As a defensive ceiling against a misbehaving upstream that never
|
||||
sends an empty page, we also stop once we have yielded strictly more rows
|
||||
than the declared ``total`` (strict ``>`` so an honest source yields
|
||||
exactly ``total`` rows).
|
||||
|
||||
``total`` is parsed lazily from the first response and may be ``None`` if
|
||||
the envelope does not carry it — in that case only the empty-page signal
|
||||
terminates the loop.
|
||||
|
||||
``rate_limiter`` is an optional :class:`RateLimiter`; when supplied, it is
|
||||
acquired before every upstream call.
|
||||
"""
|
||||
page_num = 1
|
||||
total: int | None = None
|
||||
yielded = 0
|
||||
|
||||
while True:
|
||||
if rate_limiter is not None:
|
||||
await rate_limiter.acquire()
|
||||
|
||||
body = build_body(page_num, page_size, **extra)
|
||||
resp = await client.post(path, json=body)
|
||||
|
||||
# Read the declared total once from the first response. The yunke
|
||||
# envelopes keep it on every page, but reading it once is enough for
|
||||
# the over-fetch ceiling and avoids trusting stale updates.
|
||||
if total is None:
|
||||
total = parse_total(resp)
|
||||
|
||||
rows = parse_rows(resp)
|
||||
if not rows:
|
||||
# Empty page == the upstream's "no more rows" signal. This is the
|
||||
# normal, primary termination path.
|
||||
return
|
||||
|
||||
for row in rows:
|
||||
yield row
|
||||
yielded += 1
|
||||
|
||||
# Over-fetch safety: a well-behaved source sends an empty page next,
|
||||
# but if it never does (or sends duplicates), cap at the declared total.
|
||||
# Strict ``>`` so we yield exactly ``total`` when the count is honest.
|
||||
if total is not None and yielded > total:
|
||||
return
|
||||
|
||||
page_num += 1
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""Minimal asyncio limiter enforcing a minimum interval between calls.
|
||||
|
||||
Call ``await limiter.acquire()`` before each upstream request. The first
|
||||
call never waits. Successive calls wait just long enough that the gap
|
||||
between the previous call's dispatch and this one is at least
|
||||
``min_interval`` seconds.
|
||||
|
||||
This is deliberately tiny: no tokens, no concurrency cap, no skew handling
|
||||
— just a floor on inter-call spacing, which is all the yunke scrapers need
|
||||
to stay polite to the CRM. If you need a semaphore or token bucket, that
|
||||
belongs in a future helper, not here.
|
||||
"""
|
||||
|
||||
def __init__(self, min_interval: float = 0.0) -> None:
|
||||
self._min_interval = float(min_interval)
|
||||
self._last: float = 0.0
|
||||
|
||||
async def acquire(self) -> None:
|
||||
"""Block until at least ``min_interval`` has elapsed since the last call."""
|
||||
if self._min_interval <= 0:
|
||||
self._last = time.monotonic()
|
||||
return
|
||||
|
||||
now = time.monotonic()
|
||||
wait = self._min_interval - (now - self._last)
|
||||
if wait > 0:
|
||||
await asyncio.sleep(wait)
|
||||
|
||||
self._last = time.monotonic()
|
||||
@@ -0,0 +1,171 @@
|
||||
"""WeChat friend-list scraper.
|
||||
|
||||
Wraps ``POST /wechatweb/wxFriends/friend_list/es`` and yields normalized dicts
|
||||
ready to be upserted into :class:`app.db.models.Customer` (persistence is a
|
||||
LATER task — Task 1.7). Each dict carries:
|
||||
|
||||
* snake_case keys matching the ``Customer`` columns (see :func:`_normalize`),
|
||||
* ``channel == "wechat"``,
|
||||
* ``raw_json`` — the original upstream record, verbatim, so nothing captured
|
||||
is lost even when a column has no target field.
|
||||
|
||||
Field names are mapped from the real capture (see Task 1.5 spec). The mapping
|
||||
is a straight ``camelCase -> snake_case`` translation with a couple of
|
||||
disambiguations where the source has two fields for one column:
|
||||
|
||||
* ``phone`` <- ``phone``
|
||||
* ``wechat_phone`` <- ``wechatPhone``
|
||||
* ``dept_name`` <- ``deptName`` (falls back to ``deptRouteName``)
|
||||
* ``head_image_url`` <- ``friendHeadImageUrl`` (falls back to ``headImg``)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from app.scraper.base import paginate_by_page
|
||||
from app.session.yunke_client import YunkeClient
|
||||
|
||||
__all__ = ["scrape_wechat_friends"]
|
||||
|
||||
FRIEND_LIST_PATH = "/wechatweb/wxFriends/friend_list/es"
|
||||
|
||||
|
||||
def _parse_total(resp: Any) -> int | None:
|
||||
"""Read the declared row count from the friend_list envelope.
|
||||
|
||||
Shape: ``data.basicResult.total`` (an int). Returns ``None`` if the path
|
||||
is absent so the paginator falls back to the empty-page stop signal.
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return None
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
basic = data.get("basicResult")
|
||||
if not isinstance(basic, dict):
|
||||
return None
|
||||
total = basic.get("total")
|
||||
return int(total) if isinstance(total, (int, float)) else None
|
||||
|
||||
|
||||
def _parse_rows(resp: Any) -> list[dict[str, Any]]:
|
||||
"""Extract the list of friend records from the envelope.
|
||||
|
||||
Shape: ``data.basicResult.data`` (a list of records). Missing/non-list
|
||||
values collapse to an empty list so the paginator sees "empty page".
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return []
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return []
|
||||
basic = data.get("basicResult")
|
||||
if not isinstance(basic, dict):
|
||||
return []
|
||||
rows = basic.get("data")
|
||||
return rows if isinstance(rows, list) else []
|
||||
|
||||
|
||||
def _build_body(
|
||||
page_num: int,
|
||||
page_size: int,
|
||||
*,
|
||||
department_id: str = "",
|
||||
user_id: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""Build the friend_list request body for a given page.
|
||||
|
||||
The real client sends ``departmentId`` / ``userId`` filters; we accept
|
||||
them as optional kwargs (default empty) so per-staff / per-department
|
||||
scrapes are a drop-in without changing the paginator contract.
|
||||
"""
|
||||
body: dict[str, Any] = {"pageNum": page_num, "pageSize": page_size}
|
||||
if department_id:
|
||||
body["departmentId"] = department_id
|
||||
if user_id:
|
||||
body["userId"] = user_id
|
||||
return body
|
||||
|
||||
|
||||
def _normalize(record: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Map one upstream friend record to the ``Customer`` column shape.
|
||||
|
||||
Adds ``channel="wechat"`` and preserves the original record under
|
||||
``raw_json``. ``None``-safe: missing fields become ``None`` rather than
|
||||
raising, so partial payloads do not break the whole scrape.
|
||||
"""
|
||||
return {
|
||||
"friend_id": record.get("friendWxId"),
|
||||
"channel": "wechat",
|
||||
"owner_account_id": record.get("wxId"),
|
||||
"friend_nickname": record.get("friendWxNickname"),
|
||||
"friend_remark": record.get("friendWxRemark"),
|
||||
"real_name": record.get("realName"),
|
||||
"phone": record.get("phone"),
|
||||
"wechat_phone": record.get("wechatPhone"),
|
||||
"gender": record.get("gender"),
|
||||
"region": record.get("region"),
|
||||
"description": record.get("description"),
|
||||
"sale_name": record.get("saleName"),
|
||||
"user_id": record.get("userId"),
|
||||
"dept_route": record.get("deptRoute"),
|
||||
# Prefer the canonical deptName; fall back to the route-display name.
|
||||
"dept_name": record.get("deptName") or record.get("deptRouteName"),
|
||||
# Prefer friendHeadImageUrl; fall back to the legacy headImg field.
|
||||
"head_image_url": record.get("friendHeadImageUrl") or record.get("headImg"),
|
||||
"last_chat_time": record.get("lastChatTime"),
|
||||
"send_num": record.get("sendNum"),
|
||||
"receive_num": record.get("receiveNum"),
|
||||
"message_count": record.get("messageCount"),
|
||||
"delete_flag": record.get("deleteFlag"),
|
||||
"is_block": record.get("isBlock"),
|
||||
"friend_state": record.get("friendState"),
|
||||
"version": record.get("version"),
|
||||
"raw_json": record,
|
||||
}
|
||||
|
||||
|
||||
async def scrape_wechat_friends(
|
||||
client: YunkeClient,
|
||||
*,
|
||||
department_id: str = "",
|
||||
user_id: str = "",
|
||||
page_size: int = 50,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield normalized WeChat friend records from the CRM.
|
||||
|
||||
Walks ``POST /wechatweb/wxFriends/friend_list/es`` page by page until the
|
||||
upstream reports no more rows, mapping each record to the ``Customer``
|
||||
column shape (snake_case) plus a verbatim ``raw_json`` copy.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
client:
|
||||
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
|
||||
handled by the client; this scraper only concerns itself with the
|
||||
friend-list endpoint contract.
|
||||
department_id, user_id:
|
||||
Optional upstream filters (passed through to the request body as
|
||||
``departmentId`` / ``userId``). Empty by default — scrape everything
|
||||
the logged-in account can see.
|
||||
page_size:
|
||||
Page size to request. Defaults to 50 (a sane middle ground between
|
||||
request count and per-response payload size for this endpoint).
|
||||
"""
|
||||
async for record in paginate_by_page(
|
||||
client,
|
||||
FRIEND_LIST_PATH,
|
||||
page_size=page_size,
|
||||
parse_total=_parse_total,
|
||||
parse_rows=_parse_rows,
|
||||
build_body=_build_body,
|
||||
department_id=department_id,
|
||||
user_id=user_id,
|
||||
):
|
||||
yield _normalize(record)
|
||||
Reference in New Issue
Block a user