feat: phone-call + sms scrapers
M4 (/pc/ CRM module): mirror phone-call + SMS endpoints. Scraper (app/scraper/call.py): - scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call') Envelope data.data / data.totalCount / data.pageCount; page-number pagination. incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike voice-call isSend); callSeconds preferred over legacy callDuration string; startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch. - scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms') Envelope data.data / data.dataCount (NOT totalCount), no pageCount; relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch; empty smsSub normalized to None content_preview. Shared helper: extracted parse_datetime_to_ms(value, formats) to app/scraper/parsers.py and refactored _parse_voice_start_time to delegate (DRY across all three call/SMS datetime sources). Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id / sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion, matching start_call_time's existing treatment. Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert + SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types 'phone_call' and 'sms'. Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures exercising pagination + full field mapping), repo idempotency + coercion tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers. ruff clean. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -21,9 +21,16 @@ without the scraper taking a DB dependency.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from collections.abc import Sequence
|
||||
from typing import Any
|
||||
|
||||
__all__ = ["coerce_int_or_none", "normalize_wechat_type", "normalize_whatsapp_type"]
|
||||
__all__ = [
|
||||
"coerce_int_or_none",
|
||||
"normalize_wechat_type",
|
||||
"normalize_whatsapp_type",
|
||||
"parse_datetime_to_ms",
|
||||
]
|
||||
|
||||
# Phase-0 confirmed WeChat message type codes (see
|
||||
# docs/superpowers/specs/phase0-findings.md §2). Captured by live inspection
|
||||
@@ -133,3 +140,61 @@ def coerce_int_or_none(value: Any) -> int | None:
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def parse_datetime_to_ms(
|
||||
value: Any, formats: Sequence[str]
|
||||
) -> int | None:
|
||||
"""Coerce a datetime-shaped upstream value to a ms-epoch ``int``.
|
||||
|
||||
Centralizes the datetime -> ms-epoch parsing shared by the call scrapers
|
||||
(``PhoneCall.start_call_time`` from ``callLog.startCallTime`` /
|
||||
``Sms.create_time`` from ``smsLogsList.createTime``) and the voice-call
|
||||
scraper (``PhoneCall.start_call_time`` from ``startTime``). The accepted
|
||||
shapes:
|
||||
|
||||
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
|
||||
* numeric ``str`` (e.g. ``"1700000000123"``) -> ``int`` (no datetime
|
||||
round-trip).
|
||||
* datetime ``str`` matching one of ``formats`` -> parsed with
|
||||
``datetime.strptime`` and converted to ms-epoch. The CRM is a Chinese
|
||||
deployment and the wall-clock times are presumably CST (UTC+8); this
|
||||
scraper parses them as **naive UTC** for determinism — a documented
|
||||
simplification (if production later needs CST-accurate buckets, attach
|
||||
a CST tzinfo to the parsed datetime before the timestamp call). The
|
||||
``formats`` are tried in order; the first match wins. ``.replace(
|
||||
tzinfo=UTC)`` is chained on the strptime result so ruff's DTZ007 sees
|
||||
the naive datetime being made aware (and so the resulting epoch is
|
||||
well-defined).
|
||||
* ``None`` / empty / unparseable by any format -> ``None`` (no spurious
|
||||
value stored).
|
||||
|
||||
``formats`` is the ordered list of ``strptime`` patterns to attempt. The
|
||||
call scraper passes ``["%Y-%m-%d %H:%M:%S"]`` (``startCallTime``); the SMS
|
||||
scraper passes ``["%Y-%m-%d %H:%M"]`` (``createTime``, one precision step
|
||||
shorter — no seconds).
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
if isinstance(value, str):
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return None
|
||||
# Numeric string -> ms-epoch int (no datetime round-trip).
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
pass
|
||||
# Try each strptime format in order; first match wins.
|
||||
for fmt in formats:
|
||||
try:
|
||||
parsed = dt.datetime.strptime(s, fmt).replace(tzinfo=dt.UTC)
|
||||
except ValueError:
|
||||
continue
|
||||
return int(parsed.timestamp() * 1000)
|
||||
return None
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user