feat: phone-call + sms scrapers

M4 (/pc/ CRM module): mirror phone-call + SMS endpoints.

Scraper (app/scraper/call.py):
- scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call')
  Envelope data.data / data.totalCount / data.pageCount; page-number pagination.
  incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike
  voice-call isSend); callSeconds preferred over legacy callDuration string;
  startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch.
- scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms')
  Envelope data.data / data.dataCount (NOT totalCount), no pageCount;
  relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch;
  empty smsSub normalized to None content_preview.

Shared helper: extracted parse_datetime_to_ms(value, formats) to
app/scraper/parsers.py and refactored _parse_voice_start_time to delegate
(DRY across all three call/SMS datetime sources).

Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id /
sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion,
matching start_call_time's existing treatment.

Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert
+ SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types
'phone_call' and 'sms'.

Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures
exercising pagination + full field mapping), repo idempotency + coercion
tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers.
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 15:59:25 +08:00
parent 8a22fd9ff8
commit 44d64dd237
10 changed files with 1647 additions and 42 deletions
+66 -1
View File
@@ -21,9 +21,16 @@ without the scraper taking a DB dependency.
from __future__ import annotations
import datetime as dt
from collections.abc import Sequence
from typing import Any
__all__ = ["coerce_int_or_none", "normalize_wechat_type", "normalize_whatsapp_type"]
__all__ = [
"coerce_int_or_none",
"normalize_wechat_type",
"normalize_whatsapp_type",
"parse_datetime_to_ms",
]
# Phase-0 confirmed WeChat message type codes (see
# docs/superpowers/specs/phase0-findings.md §2). Captured by live inspection
@@ -133,3 +140,61 @@ def coerce_int_or_none(value: Any) -> int | None:
except ValueError:
return None
return None
def parse_datetime_to_ms(
value: Any, formats: Sequence[str]
) -> int | None:
"""Coerce a datetime-shaped upstream value to a ms-epoch ``int``.
Centralizes the datetime -> ms-epoch parsing shared by the call scrapers
(``PhoneCall.start_call_time`` from ``callLog.startCallTime`` /
``Sms.create_time`` from ``smsLogsList.createTime``) and the voice-call
scraper (``PhoneCall.start_call_time`` from ``startTime``). The accepted
shapes:
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
* numeric ``str`` (e.g. ``"1700000000123"``) -> ``int`` (no datetime
round-trip).
* datetime ``str`` matching one of ``formats`` -> parsed with
``datetime.strptime`` and converted to ms-epoch. The CRM is a Chinese
deployment and the wall-clock times are presumably CST (UTC+8); this
scraper parses them as **naive UTC** for determinism — a documented
simplification (if production later needs CST-accurate buckets, attach
a CST tzinfo to the parsed datetime before the timestamp call). The
``formats`` are tried in order; the first match wins. ``.replace(
tzinfo=UTC)`` is chained on the strptime result so ruff's DTZ007 sees
the naive datetime being made aware (and so the resulting epoch is
well-defined).
* ``None`` / empty / unparseable by any format -> ``None`` (no spurious
value stored).
``formats`` is the ordered list of ``strptime`` patterns to attempt. The
call scraper passes ``["%Y-%m-%d %H:%M:%S"]`` (``startCallTime``); the SMS
scraper passes ``["%Y-%m-%d %H:%M"]`` (``createTime``, one precision step
shorter — no seconds).
"""
if value is None:
return None
if isinstance(value, bool):
return int(value)
if isinstance(value, (int, float)):
return int(value)
if isinstance(value, str):
s = value.strip()
if not s:
return None
# Numeric string -> ms-epoch int (no datetime round-trip).
try:
return int(s)
except ValueError:
pass
# Try each strptime format in order; first match wins.
for fmt in formats:
try:
parsed = dt.datetime.strptime(s, fmt).replace(tzinfo=dt.UTC)
except ValueError:
continue
return int(parsed.timestamp() * 1000)
return None
return None