feat: whatsapp account/friend/message scrapers
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.
- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
scrape_whatsapp_friends (page-number pagination over the
data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
(backward-cursor pagination, JSON body {uid,friendUid,start,end} —
NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
1->image, -2->divider, else unknown; never raises) and a shared
coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
also upserts the embedded Staff(user_id, name) via the existing
upsert_staff_from_account helper; WhatsappAccount imported.
Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.
Suite: 182 -> 211 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,24 +1,29 @@
|
||||
"""Field / type-code normalization helpers for upstream record parsers.
|
||||
|
||||
The yunke CRM uses opaque integer type codes (``type`` / ``messageType``) on
|
||||
WeChat message records. The storage layer prefers stable enum strings so that
|
||||
media dispatch (image / voice / video / file) and presentation (text vs.
|
||||
date divider) can be keyed off a single normalized value rather than the
|
||||
raw code.
|
||||
The yunke CRM uses opaque integer type codes (``type`` / ``messageType`` on
|
||||
WeChat, ``type`` / ``msgType`` on WhatsApp) on message records. The storage
|
||||
layer prefers stable enum strings so that media dispatch (image / voice /
|
||||
video / file) and presentation (text vs. date divider) can be keyed off a
|
||||
single normalized value rather than the raw code.
|
||||
|
||||
This module is intentionally tiny: one total function that maps the
|
||||
Phase-0 confirmed codes and falls through to ``"unknown"`` for everything
|
||||
else. It MUST NOT raise — every record pulled during a scrape passes through
|
||||
This module is intentionally tiny: one total function per channel that maps
|
||||
the Phase-0 confirmed codes and falls through to ``"unknown"`` for everything
|
||||
else. Each MUST NOT raise — every record pulled during a scrape passes through
|
||||
here, and one unexpected code must not abort the whole pull. The raw code is
|
||||
preserved on the record's ``raw_json`` so the persistence layer can recover
|
||||
it when needed.
|
||||
preserved on the record's ``raw_json`` so the persistence layer can recover it
|
||||
when needed.
|
||||
|
||||
:func:`coerce_int_or_none` is the shared scraper-layer int coercion (mirrors
|
||||
:func:`app.storage.repos._coerce_int`) used to emit a ready-to-store
|
||||
``type_raw`` int and to test raw type codes (e.g. the ``-2`` divider drop)
|
||||
without the scraper taking a DB dependency.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
__all__ = ["normalize_wechat_type"]
|
||||
__all__ = ["coerce_int_or_none", "normalize_wechat_type", "normalize_whatsapp_type"]
|
||||
|
||||
# Phase-0 confirmed WeChat message type codes (see
|
||||
# docs/superpowers/specs/phase0-findings.md §2). Captured by live inspection
|
||||
@@ -63,3 +68,68 @@ def normalize_wechat_type(raw: Any) -> str:
|
||||
except (TypeError, ValueError):
|
||||
return "unknown"
|
||||
return _WECHAT_TYPE_NAMES.get(code, "unknown")
|
||||
|
||||
|
||||
# Phase-0 confirmed WhatsApp message type codes (see data dictionary §3 +
|
||||
# phase0-findings.md §5). Captured by live inspection of the yunke CRM.
|
||||
# * ``0`` — plain text.
|
||||
# * ``1`` — image / media. ``mediaName`` is a deferred-download FILENAME
|
||||
# (not a URL); the media layer resolves it via a separate endpoint.
|
||||
# * ``-2`` — date divider. NOT a real message; the scraper uses this label as
|
||||
# its drop signal (filtered out, never yielded, never persisted).
|
||||
# Other codes pass through as ``"unknown"`` so the message is still ingested
|
||||
# (raw code kept on ``raw_json``) rather than dropped. Extend this table as
|
||||
# each is confirmed.
|
||||
_WHATSAPP_TYPE_NAMES: dict[int, str] = {
|
||||
0: "text",
|
||||
1: "image",
|
||||
-2: "divider",
|
||||
}
|
||||
|
||||
|
||||
def normalize_whatsapp_type(raw: Any) -> str:
|
||||
"""Map an upstream WhatsApp ``type`` / ``msgType`` code to a stable string.
|
||||
|
||||
Confirmed mappings: ``0→text``, ``1→image``, ``-2→divider``. Any other /
|
||||
missing / unparsable value returns ``"unknown"``.
|
||||
|
||||
Never raises: an unexpected type must not break the scrape. The raw code
|
||||
is preserved on ``raw_json`` so the persistence layer can recover it.
|
||||
"""
|
||||
try:
|
||||
code = int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return "unknown"
|
||||
return _WHATSAPP_TYPE_NAMES.get(code, "unknown")
|
||||
|
||||
|
||||
def coerce_int_or_none(value: Any) -> int | None:
|
||||
"""Coerce a possibly-string numeric value to ``int``; ``None`` preserved.
|
||||
|
||||
Mirrors :func:`app.storage.repos._coerce_int` so the scraper can emit a
|
||||
ready-to-store ``type_raw`` int and check raw type codes (e.g. the ``-2``
|
||||
divider drop) without depending on the storage layer (the scraper has no
|
||||
DB dependency by design). A non-numeric value (``"text"`` / ``None`` /
|
||||
missing) returns ``None``.
|
||||
|
||||
Kept in this module (rather than per-scraper) so both the WeChat and
|
||||
WhatsApp scrapers share one well-tested coercion instead of each carrying
|
||||
a private copy.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
return int(value)
|
||||
if isinstance(value, str):
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user