feat: phone-call + sms scrapers
M4 (/pc/ CRM module): mirror phone-call + SMS endpoints. Scraper (app/scraper/call.py): - scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call') Envelope data.data / data.totalCount / data.pageCount; page-number pagination. incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike voice-call isSend); callSeconds preferred over legacy callDuration string; startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch. - scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms') Envelope data.data / data.dataCount (NOT totalCount), no pageCount; relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch; empty smsSub normalized to None content_preview. Shared helper: extracted parse_datetime_to_ms(value, formats) to app/scraper/parsers.py and refactored _parse_voice_start_time to delegate (DRY across all three call/SMS datetime sources). Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id / sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion, matching start_call_time's existing treatment. Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert + SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types 'phone_call' and 'sms'. Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures exercising pagination + full field mapping), repo idempotency + coercion tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers. ruff clean. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
@@ -0,0 +1,376 @@
|
||||
"""Phone-call + SMS scrapers (M4, ``/pc/`` CRM module).
|
||||
|
||||
Endpoints covered:
|
||||
|
||||
* :func:`scrape_phone_calls` — wraps ``POST /pc/callLog/list`` and yields
|
||||
normalized phone-call dicts mapped to :class:`app.db.models.PhoneCall` with
|
||||
``channel="phone_call"``. Page-number pagination.
|
||||
* :func:`scrape_sms` — wraps ``POST /pc/customer/smsLogsList`` and yields
|
||||
normalized SMS dicts mapped to :class:`app.db.models.Sms` with
|
||||
``channel="sms"`` (the model's default). Page-number pagination.
|
||||
|
||||
Both reuse :func:`app.scraper.base.paginate_by_page` and emit snake_case dicts
|
||||
plus a verbatim ``raw_json`` copy, mirroring :mod:`app.scraper.wechat`.
|
||||
|
||||
Envelope shapes (Phase-0 confirmed, data dictionary §4):
|
||||
|
||||
* ``callLog/list``::
|
||||
|
||||
{"success": true, "data": {"data": [...], "page": .., "pageCount": ..,
|
||||
"totalCount": .., "phoneHidden": ..,
|
||||
"hideRecordPlay": ..}}
|
||||
|
||||
Rows under ``data.data``; declared row count on ``data.totalCount``; page
|
||||
count on ``data.pageCount``. The paginator consumes the row-count ceiling
|
||||
via ``parse_total`` and otherwise relies on the empty-page stop signal.
|
||||
|
||||
* ``smsLogsList``::
|
||||
|
||||
{"success": true, "data": {"data": [...], "dataCount": .., "power": ..}}
|
||||
|
||||
Rows under ``data.data``; declared row count on ``data.dataCount`` (NOT
|
||||
``totalCount``); **no** ``pageCount``. The paginator relies solely on the
|
||||
empty-page stop signal (the over-fetch ceiling keyed off ``dataCount``
|
||||
still applies).
|
||||
|
||||
Field mapping notes:
|
||||
|
||||
* Phone-call ``incomingCall`` is ``1=incoming / 0=outgoing`` — the SAME
|
||||
polarity as ``PhoneCall.incoming_call``. It is therefore stored DIRECTLY,
|
||||
with NO polarity inversion (unlike the voice-call channel's ``isSend``,
|
||||
which is inverted in :func:`app.scraper.wechat._normalize_voice_call`).
|
||||
* Phone-call ``callSeconds`` (int) is preferred over the legacy
|
||||
``callDuration`` ``"M'SS\""`` string for ``call_duration``.
|
||||
* Phone-call ``startCallTime`` (``YYYY-MM-DD HH:MM:SS``) and SMS
|
||||
``createTime`` (``YYYY-MM-DD HH:MM``, one precision step shorter) are both
|
||||
parsed to ms-epoch via the shared
|
||||
:func:`app.scraper.parsers.parse_datetime_to_ms`, treated as naive UTC for
|
||||
determinism (documented simplification — see the helper's docstring).
|
||||
* SMS ``smsSub`` (a short preview) is carried on ``content_preview``; an
|
||||
empty / falsy preview is normalized to ``None`` so the column reads as
|
||||
"no preview" rather than a literal empty string.
|
||||
|
||||
Auth is the same crm2 session cookie every other scraper uses (Phase-0 §1
|
||||
confirmed cookie-only auth suffices for every data endpoint).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from app.scraper.base import paginate_by_page
|
||||
from app.scraper.parsers import coerce_int_or_none, parse_datetime_to_ms
|
||||
from app.session.yunke_client import YunkeClient
|
||||
|
||||
__all__ = ["scrape_phone_calls", "scrape_sms"]
|
||||
|
||||
CALL_LOG_PATH = "/pc/callLog/list"
|
||||
SMS_LOG_PATH = "/pc/customer/smsLogsList"
|
||||
|
||||
# Datetime strptime patterns for the two endpoints. The callLog's
|
||||
# ``startCallTime`` carries seconds; the smsLogsList's ``createTime`` is one
|
||||
# precision step shorter (no seconds — default to :00 via the pattern).
|
||||
_CALL_DATETIME_FORMATS = ["%Y-%m-%d %H:%M:%S"]
|
||||
_SMS_DATETIME_FORMATS = ["%Y-%m-%d %H:%M"]
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Phone-call scraper (M4) — POST /pc/callLog/list
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
def _parse_call_total(resp: Any) -> int | None:
|
||||
"""Read the declared row count from the ``callLog/list`` envelope.
|
||||
|
||||
Shape: ``data.totalCount`` (an int). Returns ``None`` if the path is
|
||||
absent so the paginator falls back to the empty-page stop signal.
|
||||
(Live ``totalCount`` is 6047 per the data dictionary §4.)
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return None
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
total = data.get("totalCount")
|
||||
if isinstance(total, bool):
|
||||
return None
|
||||
return int(total) if isinstance(total, (int, float)) else None
|
||||
|
||||
|
||||
def _parse_call_rows(resp: Any) -> list[dict[str, Any]]:
|
||||
"""Extract the call list from the ``callLog/list`` envelope.
|
||||
|
||||
Shape: ``data.data`` (a list of records). Missing / non-list values
|
||||
collapse to an empty list so the paginator sees "empty page".
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return []
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return []
|
||||
rows = data.get("data")
|
||||
return rows if isinstance(rows, list) else []
|
||||
|
||||
|
||||
def _build_call_body(page_num: int, page_size: int) -> dict[str, Any]:
|
||||
"""Build the ``callLog/list`` request body for a given page.
|
||||
|
||||
The live capture carries no filter parameters — just the page slice.
|
||||
"""
|
||||
return {"pageNum": page_num, "pageSize": page_size}
|
||||
|
||||
|
||||
def _normalize_phone_call(record: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Map one upstream ``callLog/list`` record to the ``PhoneCall`` column shape.
|
||||
|
||||
Adds ``channel="phone_call"`` and preserves the original record under
|
||||
``raw_json``. Field mapping per data dictionary §4 + the PhoneCall column
|
||||
set:
|
||||
|
||||
* ``id`` -> ``call_id`` (PK)
|
||||
* ``startCallTime`` -> ``start_call_time`` (``YYYY-MM-DD HH:MM:SS`` ->
|
||||
ms epoch; see :func:`parse_datetime_to_ms`)
|
||||
* ``callSeconds`` -> ``call_duration`` (int seconds, PREFERRED over
|
||||
the legacy ``callDuration`` ``"M'SS\""`` string)
|
||||
* ``callStatus`` -> ``call_status`` (0=未接 / 2=接通)
|
||||
* ``incomingCall`` -> ``incoming_call`` (1=呼入 incoming / 0=呼出
|
||||
outgoing — stored DIRECTLY, NO inversion; this API's polarity already
|
||||
matches ``PhoneCall.incoming_call``)
|
||||
* ``userId`` / ``userIdName`` -> owning staff pair
|
||||
* ``departmentId`` -> ``department_id``
|
||||
* ``callNumber`` -> ``call_number`` (the customer phone)
|
||||
* ``simPhone`` / ``simNumber`` / ``imei`` -> SIM/IMEI metadata
|
||||
* ``recordFile`` -> ``record_file`` (OSS recording URL; may be null)
|
||||
* ``isUpload`` -> ``is_upload`` (bool field; repo coerces)
|
||||
* ``downloadCount`` -> ``download_count``
|
||||
* ``aiAsr`` -> ``ai_asr`` (ASR transcription text; may be null)
|
||||
* ``qcStatus`` -> ``qc_status``
|
||||
* ``planCustomerName`` -> ``plan_customer_name``
|
||||
|
||||
``call_duration`` prefers ``callSeconds`` (int) and falls back to
|
||||
``callDuration`` only when ``callSeconds`` is missing — the live capture
|
||||
sends both, but ``callSeconds`` is the cleaner (already-int) source.
|
||||
Coerced to ``int | None`` so a numeric string still lands cleanly.
|
||||
|
||||
The non-applicable ``ring_time`` column is left ``None`` — the callLog
|
||||
payload carries only ``callSeconds`` (talk time), not a separate ring
|
||||
duration. ``None``-safe on every mapped field.
|
||||
"""
|
||||
# Prefer callSeconds (int seconds) over callDuration ("M'SS\"" string).
|
||||
# Fall back to callDuration only when callSeconds is missing; coerce both
|
||||
# via the shared int coercion so a numeric string still lands cleanly.
|
||||
call_seconds = record.get("callSeconds")
|
||||
if call_seconds is None:
|
||||
call_seconds = record.get("callDuration")
|
||||
|
||||
return {
|
||||
"call_id": record.get("id"),
|
||||
"channel": "phone_call",
|
||||
"start_call_time": parse_datetime_to_ms(
|
||||
record.get("startCallTime"), _CALL_DATETIME_FORMATS
|
||||
),
|
||||
"call_duration": coerce_int_or_none(call_seconds),
|
||||
"call_status": coerce_int_or_none(record.get("callStatus")),
|
||||
# NOTE: stored DIRECTLY — this API's incomingCall (1=incoming/0=outgoing)
|
||||
# already matches PhoneCall.incoming_call polarity (no inversion).
|
||||
"incoming_call": coerce_int_or_none(record.get("incomingCall")),
|
||||
"user_id": record.get("userId"),
|
||||
"user_id_name": record.get("userIdName"),
|
||||
"department_id": record.get("departmentId"),
|
||||
"call_number": record.get("callNumber"),
|
||||
"sim_phone": record.get("simPhone"),
|
||||
"sim_number": record.get("simNumber"),
|
||||
"imei": record.get("imei"),
|
||||
"record_file": record.get("recordFile"),
|
||||
"is_upload": record.get("isUpload"),
|
||||
"download_count": coerce_int_or_none(record.get("downloadCount")),
|
||||
"ai_asr": record.get("aiAsr"),
|
||||
"qc_status": coerce_int_or_none(record.get("qcStatus")),
|
||||
"plan_customer_name": record.get("planCustomerName"),
|
||||
"version": record.get("version"),
|
||||
"raw_json": record,
|
||||
}
|
||||
|
||||
|
||||
async def scrape_phone_calls(
|
||||
client: YunkeClient,
|
||||
*,
|
||||
page_size: int = 50,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield normalized phone-call records from ``POST /pc/callLog/list``.
|
||||
|
||||
Walks the endpoint page by page until the upstream reports no more rows
|
||||
(empty ``data.data`` array — the primary stop signal), mapping each record
|
||||
to the ``PhoneCall`` column shape with ``channel="phone_call"`` plus a
|
||||
verbatim ``raw_json`` copy.
|
||||
|
||||
The envelope keeps the row list under ``data.data`` (same as ``wxUser/
|
||||
list``) but the declared row count is on ``data.totalCount`` (NOT
|
||||
``data.total``) and the page count is on ``data.pageCount``. The paginator
|
||||
consumes ``totalCount`` for its over-fetch ceiling and otherwise relies on
|
||||
the empty-page stop signal.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
client:
|
||||
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
|
||||
handled by the client.
|
||||
page_size:
|
||||
Page size to request. Defaults to 50 (a sane middle ground between
|
||||
request count and per-response payload size for this endpoint).
|
||||
"""
|
||||
async for record in paginate_by_page(
|
||||
client,
|
||||
CALL_LOG_PATH,
|
||||
page_size=page_size,
|
||||
parse_total=_parse_call_total,
|
||||
parse_rows=_parse_call_rows,
|
||||
build_body=_build_call_body,
|
||||
):
|
||||
yield _normalize_phone_call(record)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# SMS scraper (M4) — POST /pc/customer/smsLogsList
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
def _parse_sms_total(resp: Any) -> int | None:
|
||||
"""Read the declared row count from the ``smsLogsList`` envelope.
|
||||
|
||||
Shape: ``data.dataCount`` (an int) — note ``dataCount``, NOT
|
||||
``totalCount`` (and there is no ``pageCount`` on this envelope). Returns
|
||||
``None`` if the path is absent so the paginator falls back to the
|
||||
empty-page stop signal. (Live ``dataCount`` is 53 per data dictionary §4.)
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return None
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
total = data.get("dataCount")
|
||||
if isinstance(total, bool):
|
||||
return None
|
||||
return int(total) if isinstance(total, (int, float)) else None
|
||||
|
||||
|
||||
def _parse_sms_rows(resp: Any) -> list[dict[str, Any]]:
|
||||
"""Extract the SMS list from the ``smsLogsList`` envelope.
|
||||
|
||||
Shape: ``data.data`` (a list of records). Missing / non-list values
|
||||
collapse to an empty list so the paginator sees "empty page".
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return []
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return []
|
||||
rows = data.get("data")
|
||||
return rows if isinstance(rows, list) else []
|
||||
|
||||
|
||||
def _build_sms_body(page_num: int, page_size: int) -> dict[str, Any]:
|
||||
"""Build the ``smsLogsList`` request body for a given page.
|
||||
|
||||
The live capture carries no filter parameters — just the page slice.
|
||||
"""
|
||||
return {"pageNum": page_num, "pageSize": page_size}
|
||||
|
||||
|
||||
def _normalize_sms(record: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Map one upstream ``smsLogsList`` record to the ``Sms`` column shape.
|
||||
|
||||
Adds ``channel="sms"`` (the model's default, but carried explicitly so the
|
||||
repo upsert is independent of the column-default application order) and
|
||||
preserves the original record under ``raw_json``. Field mapping per data
|
||||
dictionary §4 + the Sms column set:
|
||||
|
||||
* ``id`` -> ``sms_id`` (PK)
|
||||
* ``customerPhone`` -> ``customer_phone``
|
||||
* ``sms`` -> ``content`` (full SMS text)
|
||||
* ``smsSub`` -> ``content_preview`` (short preview; an EMPTY
|
||||
/ falsy preview is normalized to ``None`` so the column reads as "no
|
||||
preview" rather than a literal empty string)
|
||||
* ``sender`` -> ``sender``
|
||||
* ``direction`` -> ``direction`` (0=收 incoming / 1=发 outgoing)
|
||||
* ``userId`` -> ``user_id`` (owning staff)
|
||||
* ``senderDepartment`` / ``senderDepartmentId`` -> dept name / id
|
||||
* ``receiveName`` -> ``receive_name``
|
||||
* ``createTime`` -> ``create_time`` (``YYYY-MM-DD HH:MM`` -> ms
|
||||
epoch; seconds default to :00 via the strptime pattern)
|
||||
* ``simPhone`` / ``simNumber`` -> SIM metadata
|
||||
* ``source`` -> ``source``
|
||||
|
||||
``None``-safe on every mapped field.
|
||||
"""
|
||||
return {
|
||||
"sms_id": record.get("id"),
|
||||
"channel": "sms",
|
||||
"customer_phone": record.get("customerPhone"),
|
||||
"content": record.get("sms"),
|
||||
# Empty preview -> None (so the column reads "no preview" rather than
|
||||
# a literal empty string). The repo does not normalize this; the
|
||||
# scraper owns the falsy->None collapse.
|
||||
"content_preview": record.get("smsSub") or None,
|
||||
"sender": record.get("sender"),
|
||||
"direction": coerce_int_or_none(record.get("direction")),
|
||||
"user_id": record.get("userId"),
|
||||
"sender_department": record.get("senderDepartment"),
|
||||
"sender_department_id": record.get("senderDepartmentId"),
|
||||
"receive_name": record.get("receiveName"),
|
||||
"create_time": parse_datetime_to_ms(
|
||||
record.get("createTime"), _SMS_DATETIME_FORMATS
|
||||
),
|
||||
"sim_phone": record.get("simPhone"),
|
||||
"sim_number": record.get("simNumber"),
|
||||
"source": record.get("source"),
|
||||
"version": record.get("version"),
|
||||
"raw_json": record,
|
||||
}
|
||||
|
||||
|
||||
async def scrape_sms(
|
||||
client: YunkeClient,
|
||||
*,
|
||||
page_size: int = 50,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield normalized SMS records from ``POST /pc/customer/smsLogsList``.
|
||||
|
||||
Walks the endpoint page by page until the upstream reports no more rows
|
||||
(empty ``data.data`` array — the primary stop signal), mapping each record
|
||||
to the ``Sms`` column shape with ``channel="sms"`` plus a verbatim
|
||||
``raw_json`` copy.
|
||||
|
||||
The envelope keeps the row list under ``data.data`` but the declared row
|
||||
count is on ``data.dataCount`` (NOT ``totalCount`` and NOT ``total``) and
|
||||
there is **no** ``pageCount``. The paginator therefore relies on
|
||||
``dataCount`` only for the over-fetch ceiling and otherwise on the
|
||||
empty-page stop signal — the same shape as the talker-list endpoint.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
client:
|
||||
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
|
||||
handled by the client.
|
||||
page_size:
|
||||
Page size to request. Defaults to 50 (a sane middle ground between
|
||||
request count and per-response payload size for this endpoint).
|
||||
"""
|
||||
async for record in paginate_by_page(
|
||||
client,
|
||||
SMS_LOG_PATH,
|
||||
page_size=page_size,
|
||||
parse_total=_parse_sms_total,
|
||||
parse_rows=_parse_sms_rows,
|
||||
build_body=_build_sms_body,
|
||||
):
|
||||
yield _normalize_sms(record)
|
||||
@@ -21,9 +21,16 @@ without the scraper taking a DB dependency.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from collections.abc import Sequence
|
||||
from typing import Any
|
||||
|
||||
__all__ = ["coerce_int_or_none", "normalize_wechat_type", "normalize_whatsapp_type"]
|
||||
__all__ = [
|
||||
"coerce_int_or_none",
|
||||
"normalize_wechat_type",
|
||||
"normalize_whatsapp_type",
|
||||
"parse_datetime_to_ms",
|
||||
]
|
||||
|
||||
# Phase-0 confirmed WeChat message type codes (see
|
||||
# docs/superpowers/specs/phase0-findings.md §2). Captured by live inspection
|
||||
@@ -133,3 +140,61 @@ def coerce_int_or_none(value: Any) -> int | None:
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def parse_datetime_to_ms(
|
||||
value: Any, formats: Sequence[str]
|
||||
) -> int | None:
|
||||
"""Coerce a datetime-shaped upstream value to a ms-epoch ``int``.
|
||||
|
||||
Centralizes the datetime -> ms-epoch parsing shared by the call scrapers
|
||||
(``PhoneCall.start_call_time`` from ``callLog.startCallTime`` /
|
||||
``Sms.create_time`` from ``smsLogsList.createTime``) and the voice-call
|
||||
scraper (``PhoneCall.start_call_time`` from ``startTime``). The accepted
|
||||
shapes:
|
||||
|
||||
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
|
||||
* numeric ``str`` (e.g. ``"1700000000123"``) -> ``int`` (no datetime
|
||||
round-trip).
|
||||
* datetime ``str`` matching one of ``formats`` -> parsed with
|
||||
``datetime.strptime`` and converted to ms-epoch. The CRM is a Chinese
|
||||
deployment and the wall-clock times are presumably CST (UTC+8); this
|
||||
scraper parses them as **naive UTC** for determinism — a documented
|
||||
simplification (if production later needs CST-accurate buckets, attach
|
||||
a CST tzinfo to the parsed datetime before the timestamp call). The
|
||||
``formats`` are tried in order; the first match wins. ``.replace(
|
||||
tzinfo=UTC)`` is chained on the strptime result so ruff's DTZ007 sees
|
||||
the naive datetime being made aware (and so the resulting epoch is
|
||||
well-defined).
|
||||
* ``None`` / empty / unparseable by any format -> ``None`` (no spurious
|
||||
value stored).
|
||||
|
||||
``formats`` is the ordered list of ``strptime`` patterns to attempt. The
|
||||
call scraper passes ``["%Y-%m-%d %H:%M:%S"]`` (``startCallTime``); the SMS
|
||||
scraper passes ``["%Y-%m-%d %H:%M"]`` (``createTime``, one precision step
|
||||
shorter — no seconds).
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
if isinstance(value, str):
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return None
|
||||
# Numeric string -> ms-epoch int (no datetime round-trip).
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
pass
|
||||
# Try each strptime format in order; first match wins.
|
||||
for fmt in formats:
|
||||
try:
|
||||
parsed = dt.datetime.strptime(s, fmt).replace(tzinfo=dt.UTC)
|
||||
except ValueError:
|
||||
continue
|
||||
return int(parsed.timestamp() * 1000)
|
||||
return None
|
||||
return None
|
||||
|
||||
@@ -58,12 +58,11 @@ disambiguations where the source has two fields for one column:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from app.scraper.base import paginate_by_cursor, paginate_by_page
|
||||
from app.scraper.parsers import normalize_wechat_type
|
||||
from app.scraper.parsers import normalize_wechat_type, parse_datetime_to_ms
|
||||
from app.session.yunke_client import YunkeClient
|
||||
|
||||
__all__ = [
|
||||
@@ -1035,47 +1034,18 @@ def _build_voice_body(page_num: int, page_size: int) -> dict[str, Any]:
|
||||
def _parse_voice_start_time(value: Any) -> int | None:
|
||||
"""Coerce a voice-call ``startTime`` to a ms-epoch int (the PhoneCall column type).
|
||||
|
||||
The voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
|
||||
datetime string (confirmed by the Task 2.4 spec). To be robust against the
|
||||
upstream also emitting a numeric ms-epoch (as other call endpoints do), the
|
||||
following shapes are accepted:
|
||||
Thin wrapper over the shared :func:`app.scraper.parsers.parse_datetime_to_ms`
|
||||
so the voice-call scraper speaks the same datetime -> ms-epoch dialect as
|
||||
the phone-call + SMS scrapers while keeping its capture-specific name. The
|
||||
voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
|
||||
datetime string (confirmed by the Task 2.4 spec); the shared helper also
|
||||
accepts a numeric ms-epoch (int / float / numeric str) for robustness.
|
||||
|
||||
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
|
||||
* numeric ``str`` (e.g. ``"1700000000123"``) -> ``int``.
|
||||
* ``"YYYY-MM-DD HH:MM:SS"`` datetime string -> parsed and converted to
|
||||
ms-epoch. The CRM is a Chinese deployment and the wall-clock times are
|
||||
presumably CST (UTC+8); this scraper parses them as **naive UTC** for
|
||||
determinism — a documented simplification. If production later needs
|
||||
CST-accurate buckets, swap the ``tzinfo`` attached here.
|
||||
* ``None`` / empty / unparseable -> ``None`` (no spurious value stored).
|
||||
See :func:`parse_datetime_to_ms` for the full accepted-shapes contract and
|
||||
the documented naive-UTC simplification (the CRM wall-clock times are
|
||||
presumably CST; this parses them as UTC for determinism).
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
if isinstance(value, str):
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return None
|
||||
# Numeric string -> ms-epoch int (no datetime round-trip).
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
pass
|
||||
# Documented "YYYY-MM-DD HH:MM:SS" datetime string -> ms-epoch.
|
||||
# ``.replace(tzinfo=...)`` is chained on the strptime so ruff's
|
||||
# DTZ007 sees the naive datetime being made aware (and so the
|
||||
# resulting epoch is well-defined).
|
||||
try:
|
||||
parsed = dt.datetime.strptime(s, "%Y-%m-%d %H:%M:%S").replace(
|
||||
tzinfo=dt.UTC
|
||||
)
|
||||
except ValueError:
|
||||
return None
|
||||
return int(parsed.timestamp() * 1000)
|
||||
return None
|
||||
return parse_datetime_to_ms(value, ["%Y-%m-%d %H:%M:%S"])
|
||||
|
||||
|
||||
def _normalize_voice_call(record: dict[str, Any]) -> dict[str, Any]:
|
||||
|
||||
Reference in New Issue
Block a user