feat: phone-call + sms scrapers
M4 (/pc/ CRM module): mirror phone-call + SMS endpoints. Scraper (app/scraper/call.py): - scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call') Envelope data.data / data.totalCount / data.pageCount; page-number pagination. incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike voice-call isSend); callSeconds preferred over legacy callDuration string; startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch. - scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms') Envelope data.data / data.dataCount (NOT totalCount), no pageCount; relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch; empty smsSub normalized to None content_preview. Shared helper: extracted parse_datetime_to_ms(value, formats) to app/scraper/parsers.py and refactored _parse_voice_start_time to delegate (DRY across all three call/SMS datetime sources). Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id / sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion, matching start_call_time's existing treatment. Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert + SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types 'phone_call' and 'sms'. Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures exercising pagination + full field mapping), repo idempotency + coercion tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers. ruff clean. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -58,12 +58,11 @@ disambiguations where the source has two fields for one column:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from app.scraper.base import paginate_by_cursor, paginate_by_page
|
||||
from app.scraper.parsers import normalize_wechat_type
|
||||
from app.scraper.parsers import normalize_wechat_type, parse_datetime_to_ms
|
||||
from app.session.yunke_client import YunkeClient
|
||||
|
||||
__all__ = [
|
||||
@@ -1035,47 +1034,18 @@ def _build_voice_body(page_num: int, page_size: int) -> dict[str, Any]:
|
||||
def _parse_voice_start_time(value: Any) -> int | None:
|
||||
"""Coerce a voice-call ``startTime`` to a ms-epoch int (the PhoneCall column type).
|
||||
|
||||
The voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
|
||||
datetime string (confirmed by the Task 2.4 spec). To be robust against the
|
||||
upstream also emitting a numeric ms-epoch (as other call endpoints do), the
|
||||
following shapes are accepted:
|
||||
Thin wrapper over the shared :func:`app.scraper.parsers.parse_datetime_to_ms`
|
||||
so the voice-call scraper speaks the same datetime -> ms-epoch dialect as
|
||||
the phone-call + SMS scrapers while keeping its capture-specific name. The
|
||||
voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
|
||||
datetime string (confirmed by the Task 2.4 spec); the shared helper also
|
||||
accepts a numeric ms-epoch (int / float / numeric str) for robustness.
|
||||
|
||||
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
|
||||
* numeric ``str`` (e.g. ``"1700000000123"``) -> ``int``.
|
||||
* ``"YYYY-MM-DD HH:MM:SS"`` datetime string -> parsed and converted to
|
||||
ms-epoch. The CRM is a Chinese deployment and the wall-clock times are
|
||||
presumably CST (UTC+8); this scraper parses them as **naive UTC** for
|
||||
determinism — a documented simplification. If production later needs
|
||||
CST-accurate buckets, swap the ``tzinfo`` attached here.
|
||||
* ``None`` / empty / unparseable -> ``None`` (no spurious value stored).
|
||||
See :func:`parse_datetime_to_ms` for the full accepted-shapes contract and
|
||||
the documented naive-UTC simplification (the CRM wall-clock times are
|
||||
presumably CST; this parses them as UTC for determinism).
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
if isinstance(value, str):
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return None
|
||||
# Numeric string -> ms-epoch int (no datetime round-trip).
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
pass
|
||||
# Documented "YYYY-MM-DD HH:MM:SS" datetime string -> ms-epoch.
|
||||
# ``.replace(tzinfo=...)`` is chained on the strptime so ruff's
|
||||
# DTZ007 sees the naive datetime being made aware (and so the
|
||||
# resulting epoch is well-defined).
|
||||
try:
|
||||
parsed = dt.datetime.strptime(s, "%Y-%m-%d %H:%M:%S").replace(
|
||||
tzinfo=dt.UTC
|
||||
)
|
||||
except ValueError:
|
||||
return None
|
||||
return int(parsed.timestamp() * 1000)
|
||||
return None
|
||||
return parse_datetime_to_ms(value, ["%Y-%m-%d %H:%M:%S"])
|
||||
|
||||
|
||||
def _normalize_voice_call(record: dict[str, Any]) -> dict[str, Any]:
|
||||
|
||||
Reference in New Issue
Block a user