Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.
- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
scrape_whatsapp_friends (page-number pagination over the
data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
(backward-cursor pagination, JSON body {uid,friendUid,start,end} —
NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
1->image, -2->divider, else unknown; never raises) and a shared
coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
also upserts the embedded Staff(user_id, name) via the existing
upsert_staff_from_account helper; WhatsappAccount imported.
Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.
Suite: 182 -> 211 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace the Phase-0 placeholder password encryption with real RSA
(PKCS#1 v1.5). encrypt_password now builds an RSA public key from the
getSecureKey modulus/exponent (RSAPublicNumbers, e=int(hex,16),
n=int(hex,16)), encrypts the UTF-8 password, and returns hex (documented
assumption -- base64 is the one-line fallback noted in the docstring).
get_secure_key now returns the inner data dict to match the new contract;
pc_login keeps {account, password} with a comment on the username/password
and account/pwd fallbacks if pcLogin rejects.
Extend normalize_wechat_type with the Phase-0 confirmed codes:
2 -> image, 3 -> image_legacy, 4 -> video, 10 -> system (1 -> text and
-2 -> divider unchanged; 6/9/13/15/21/22 stay unknown until confirmed).
Tests:
- encrypt_password round-trip: generate an RSA keypair, build secure_key
from its public numbers, encrypt, then decrypt with the private key and
assert the original password comes back. Pins 512-hex output for a
2048-bit key. No network.
- Parser parametrize table extended for the new codes (and a downstream
message-fixture assertion updated from unknown -> image_legacy).
Suite: 87 -> 97 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds Task 1.6: a backward-cursor paginator in base.py, a message-type
normalizer in parsers.py, and the WeChat chat-message scraper in wechat.py.
- app/scraper/parsers.py: normalize_wechat_type() maps 1 -> "text",
-2 -> "divider", everything else -> "unknown". Total (never raises);
unknown codes pass through so no real message is silently dropped.
- app/scraper/base.py: paginate_by_cursor() generic async generator. Takes
a fetch(cursor) -> (rows, next_cursor, has_more) coroutine; advances the
cursor and stops on has_more=False OR an empty page (defensive). Same
RateLimiter integration as paginate_by_page.
- app/scraper/wechat.py: scrape_wechat_messages() wraps
POST /wechatweb/msg/chats. Body is form-encoded (NOT JSON) — content-type
is overridden per request to application/x-www-form-urlencoded via the
YunkeClient.post(headers=...) override hook. Pages backward by setting
end = data.end - 1 each batch, repeats until hasLast=false. Drops any
record whose type/messageType is -2 (date divider UI chrome). Yields
normalized dicts (snake_case Message columns + channel="wechat" +
raw_json); the raw type code is preserved on raw_json for the
persistence layer (Task 1.7).
Tests: 13 new (10 parser parametrized + 3 scraper). respx-mocked
msg/chats with two backward pages — asserts [m1, m2, m3] yielded (the -2
divider dropped), full field mapping vs the live-capture names, the
content-type is form-encoded (not JSON), and the 2nd request body's
end == E1 - 1 (proving the backward cursor advance).
Co-Authored-By: Claude <noreply@anthropic.com>
浅念