浅念
903286f379
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py. - app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text", -2 -> "divider", everything else -> "unknown". Total (never raises); unknown codes pass through so no real message is silently dropped. - app/scraper/base.py: paginate_by_cursor() generic async generator. Takes a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the cursor and stops on has_more=False OR an empty page (defensive). Same RateLimiter integration as paginate_by_page. - app/scraper/wechat.py: scrape_wechat_messages() wraps POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type is overridden per request to application/x-www-form-urlencoded via the YunkeClient.post(headers=...) override hook. Pages backward by setting end = data.end - 1 each batch, repeats until hasLast=false. Drops any record whose type/messageType is -2 (date divider UI chrome). Yields normalized dicts (snake_case Message columns + channel="wechat" + raw_json); the raw type code is preserved on raw_json for the persistence layer (Task 1.7). Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2 divider dropped), full field mapping vs the live-capture names, the content-type is form-encoded (not JSON), and the 2nd request body's end == E1 - 1 (proving the backward cursor advance). Co-Authored-By: Claude <noreply@anthropic.com>
45 lines
1.5 KiB
Python
45 lines
1.5 KiB
Python
"""Tests for the message-type normalization parser (Task 1.6).
|
|
|
|
The parser must be total — it never raises on weird input — because every
|
|
incoming WeChat record runs through it during a scrape, and one malformed
|
|
type code must not abort the whole pull. Unknown codes pass through as
|
|
``"unknown"`` so the message is still ingested (and recoverable via
|
|
``raw_json``); only the ``-2`` date-divider code maps to ``"divider"``,
|
|
which the scraper uses as its drop signal.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from app.scraper.parsers import normalize_wechat_type
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"raw,expected",
|
|
[
|
|
# Documented codes.
|
|
(1, "text"),
|
|
(-2, "divider"),
|
|
# Phase-0 unknown codes — pass through as "unknown", never dropped.
|
|
(3, "unknown"), # image
|
|
(34, "unknown"), # voice
|
|
(49, "unknown"), # file / link / transfer
|
|
(10000, "unknown"),
|
|
# Stringified ints (the upstream API may send either form).
|
|
("1", "text"),
|
|
("-2", "divider"),
|
|
("34", "unknown"),
|
|
],
|
|
)
|
|
def test_normalize_wechat_type_maps_known_and_unknown(raw, expected):
|
|
assert normalize_wechat_type(raw) == expected
|
|
|
|
|
|
def test_normalize_wechat_type_never_raises():
|
|
"""Whatever we throw at it, it returns a string — never raises."""
|
|
for raw in (None, "", "junk", "1.5", [], {}, object(), float("nan")):
|
|
out = normalize_wechat_type(raw)
|
|
assert isinstance(out, str)
|
|
assert out == "unknown"
|