feat: whatsapp account/friend/message scrapers
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.
- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
scrape_whatsapp_friends (page-number pagination over the
data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
(backward-cursor pagination, JSON body {uid,friendUid,start,end} —
NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
1->image, -2->divider, else unknown; never raises) and a shared
coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
also upserts the embedded Staff(user_id, name) via the existing
upsert_staff_from_account helper; WhatsappAccount imported.
Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.
Suite: 182 -> 211 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -63,6 +63,7 @@ from app.db.models import (
|
||||
PhoneCall,
|
||||
Staff,
|
||||
WechatAccount,
|
||||
WhatsappAccount,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -76,6 +77,7 @@ __all__ = [
|
||||
"upsert_staff_from_account",
|
||||
"upsert_voice_call",
|
||||
"upsert_wechat_account",
|
||||
"upsert_whatsapp_account",
|
||||
]
|
||||
|
||||
|
||||
@@ -411,6 +413,25 @@ async def upsert_staff_from_account(session: AsyncSession, rec: dict[str, Any])
|
||||
await upsert_staff(session, staff_rec)
|
||||
|
||||
|
||||
async def upsert_whatsapp_account(session: AsyncSession, rec: dict[str, Any]) -> None:
|
||||
"""Idempotently upsert a :class:`WhatsappAccount` AND its embedded Staff row.
|
||||
|
||||
PK = ``uid`` (of the form ``<phone>@s.whatsapp.net``). Mirrors
|
||||
:func:`upsert_wechat_account`: the companion :func:`upsert_staff_from_account`
|
||||
covers the embedded staff row (``user_id`` + ``user_name``); this helper
|
||||
calls it automatically so an account scrape that lands accounts also lands
|
||||
the owning staff — callers do not need to upsert staff separately.
|
||||
|
||||
The WA account payload (unlike the WeChat one) DOES carry ``sale_phone``,
|
||||
so the embedded staff row picks it up via the shared staff-from-account
|
||||
helper. The staff upsert is best-effort: an account record with no
|
||||
``user_id`` still writes the account row; the staff upsert silently
|
||||
no-ops.
|
||||
"""
|
||||
await _upsert_one(session, WhatsappAccount, rec)
|
||||
await upsert_staff_from_account(session, rec)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ChatGroup + GroupMember upsert (M2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user