feat: phone-call + sms scrapers

M4 (/pc/ CRM module): mirror phone-call + SMS endpoints.

Scraper (app/scraper/call.py):
- scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call')
  Envelope data.data / data.totalCount / data.pageCount; page-number pagination.
  incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike
  voice-call isSend); callSeconds preferred over legacy callDuration string;
  startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch.
- scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms')
  Envelope data.data / data.dataCount (NOT totalCount), no pageCount;
  relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch;
  empty smsSub normalized to None content_preview.

Shared helper: extracted parse_datetime_to_ms(value, formats) to
app/scraper/parsers.py and refactored _parse_voice_start_time to delegate
(DRY across all three call/SMS datetime sources).

Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id /
sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion,
matching start_call_time's existing treatment.

Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert
+ SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types
'phone_call' and 'sms'.

Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures
exercising pagination + full field mapping), repo idempotency + coercion
tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers.
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 15:59:25 +08:00
parent 8a22fd9ff8
commit 44d64dd237
10 changed files with 1647 additions and 42 deletions
+11 -41
View File
@@ -58,12 +58,11 @@ disambiguations where the source has two fields for one column:
from __future__ import annotations
import datetime as dt
from collections.abc import AsyncIterator
from typing import Any
from app.scraper.base import paginate_by_cursor, paginate_by_page
from app.scraper.parsers import normalize_wechat_type
from app.scraper.parsers import normalize_wechat_type, parse_datetime_to_ms
from app.session.yunke_client import YunkeClient
__all__ = [
@@ -1035,47 +1034,18 @@ def _build_voice_body(page_num: int, page_size: int) -> dict[str, Any]:
def _parse_voice_start_time(value: Any) -> int | None:
"""Coerce a voice-call ``startTime`` to a ms-epoch int (the PhoneCall column type).
The voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
datetime string (confirmed by the Task 2.4 spec). To be robust against the
upstream also emitting a numeric ms-epoch (as other call endpoints do), the
following shapes are accepted:
Thin wrapper over the shared :func:`app.scraper.parsers.parse_datetime_to_ms`
so the voice-call scraper speaks the same datetime -> ms-epoch dialect as
the phone-call + SMS scrapers while keeping its capture-specific name. The
voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
datetime string (confirmed by the Task 2.4 spec); the shared helper also
accepts a numeric ms-epoch (int / float / numeric str) for robustness.
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
* numeric ``str`` (e.g. ``"1700000000123"``) -> ``int``.
* ``"YYYY-MM-DD HH:MM:SS"`` datetime string -> parsed and converted to
ms-epoch. The CRM is a Chinese deployment and the wall-clock times are
presumably CST (UTC+8); this scraper parses them as **naive UTC** for
determinism — a documented simplification. If production later needs
CST-accurate buckets, swap the ``tzinfo`` attached here.
* ``None`` / empty / unparseable -> ``None`` (no spurious value stored).
See :func:`parse_datetime_to_ms` for the full accepted-shapes contract and
the documented naive-UTC simplification (the CRM wall-clock times are
presumably CST; this parses them as UTC for determinism).
"""
if value is None:
return None
if isinstance(value, bool):
return int(value)
if isinstance(value, (int, float)):
return int(value)
if isinstance(value, str):
s = value.strip()
if not s:
return None
# Numeric string -> ms-epoch int (no datetime round-trip).
try:
return int(s)
except ValueError:
pass
# Documented "YYYY-MM-DD HH:MM:SS" datetime string -> ms-epoch.
# ``.replace(tzinfo=...)`` is chained on the strptime so ruff's
# DTZ007 sees the naive datetime being made aware (and so the
# resulting epoch is well-defined).
try:
parsed = dt.datetime.strptime(s, "%Y-%m-%d %H:%M:%S").replace(
tzinfo=dt.UTC
)
except ValueError:
return None
return int(parsed.timestamp() * 1000)
return None
return parse_datetime_to_ms(value, ["%Y-%m-%d %H:%M:%S"])
def _normalize_voice_call(record: dict[str, Any]) -> dict[str, Any]: