浅念
903286f379
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py. - app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text", -2 -> "divider", everything else -> "unknown". Total (never raises); unknown codes pass through so no real message is silently dropped. - app/scraper/base.py: paginate_by_cursor() generic async generator. Takes a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the cursor and stops on has_more=False OR an empty page (defensive). Same RateLimiter integration as paginate_by_page. - app/scraper/wechat.py: scrape_wechat_messages() wraps POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type is overridden per request to application/x-www-form-urlencoded via the YunkeClient.post(headers=...) override hook. Pages backward by setting end = data.end - 1 each batch, repeats until hasLast=false. Drops any record whose type/messageType is -2 (date divider UI chrome). Yields normalized dicts (snake_case Message columns + channel="wechat" + raw_json); the raw type code is preserved on raw_json for the persistence layer (Task 1.7). Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2 divider dropped), full field mapping vs the live-capture names, the content-type is form-encoded (not JSON), and the 2nd request body's end == E1 - 1 (proving the backward cursor advance). Co-Authored-By: Claude <noreply@anthropic.com>
52 lines
2.2 KiB
Python
52 lines
2.2 KiB
Python
"""Field / type-code normalization helpers for upstream record parsers.
|
|
|
|
The yunke CRM uses opaque integer type codes (``type`` / ``messageType``) on
|
|
WeChat message records. The storage layer prefers stable enum strings so that
|
|
media dispatch (image / voice / video / file) and presentation (text vs.
|
|
date divider) can be keyed off a single normalized value rather than the
|
|
raw code, whose exact mapping for non-text codes is a Phase-0 unknown.
|
|
|
|
This module is intentionally tiny: one total function that maps the
|
|
documented codes and falls through to ``"unknown"`` for everything else.
|
|
It MUST NOT raise — every record pulled during a scrape passes through here,
|
|
and one unexpected code must not abort the whole pull. The raw code is
|
|
preserved on the record's ``raw_json`` so the persistence layer can recover
|
|
it when needed.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
__all__ = ["normalize_wechat_type"]
|
|
|
|
# Documented WeChat message type codes (see Task 1.6 spec).
|
|
# * ``1`` — plain text (and sticker text). The only non-media type we map
|
|
# today; everything else falls through to ``"unknown"``.
|
|
# * ``-2`` — date divider. NOT a real message; the scraper uses this label as
|
|
# its drop signal (filtered out, never yielded, never persisted).
|
|
# Image / voice / video / file / transfer / redpacket / system codes are
|
|
# observed in the wild but their exact integer values are a Phase-0 unknown.
|
|
# When confirmed, extend this table — for now they pass through as
|
|
# ``"unknown"`` so the message is still ingested rather than dropped.
|
|
_WECHAT_TYPE_NAMES: dict[int, str] = {
|
|
1: "text",
|
|
-2: "divider",
|
|
}
|
|
|
|
|
|
def normalize_wechat_type(raw: Any) -> str:
|
|
"""Map an upstream WeChat ``type`` / ``messageType`` code to a stable string.
|
|
|
|
Returns ``"text"`` for code ``1``, ``"divider"`` for code ``-2``, and
|
|
``"unknown"`` for any other / missing / unparsable value.
|
|
|
|
Never raises: an unexpected type must not break the scrape. The raw code
|
|
is preserved on ``raw_json`` so the persistence layer can recover it.
|
|
"""
|
|
try:
|
|
code = int(raw)
|
|
except (TypeError, ValueError):
|
|
return "unknown"
|
|
return _WECHAT_TYPE_NAMES.get(code, "unknown")
|