xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b51b32a14b3aecaee3ba5c7013dfe1d409ec026f
yunke/yunke-server/app/scraper/wechat.py
T
浅念 6ff288f1d3 feat(groups): chat_group 补 6 群主/建群列 + 复合PK迁移修复 + 存量回填 
对齐云客员工聊天窗口·微信群 tab(2026-08-25 真机验证 raw_json 键):
- ChatGroup 加列 owner_name/owner_wxid/owner_alias/owner_avatar/
  friend_count/create_time(全 nullable)
- scraper _normalize_chat_group 同名提取;createTime 兼容 ms-int 与
  'YYYY-MM-DD HH:mm:ss'(smoke.db 实测 1105 行全为后者),复用
  _parse_add_time 归一 ms-epoch
- 迁移 099ac2097fa2:加 6 列 + 修复 Task1 评审遗留 drift——PK 由单列
  room_id 变更为复合 (room_id, channel, owner_account_id) 且
  owner_account_id NOT NULL(对齐模型 8a99940;smoke.db 1105 行无 NULL,
  未兜底编造数据);SQLite 走 batch 重建(copy_from 显式旧结构),
  MySQL 渲染为直接 ALTER(离线 --sql 已验证)
- 回填脚本 scripts/backfill_chat_group_fields.py(幂等,逐列只补空);
  smoke.db 实跑 backfilled 1105/1105,重跑 0/1105
- GroupOut 加 6 字段(model_validate 自动输出,路由无改)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-25 14:32:12 +08:00

1292 lines
52 KiB
Python
Raw Blame History

"""WeChat + organization scrapers.
Endpoints covered today:
* :func:`scrape_wechat_friends` — wraps ``POST /wechatweb/wxFriends/friend_list/es``
and yields normalized dicts ready to be upserted into
:class:`app.db.models.Customer`.
* :func:`scrape_wechat_messages` — wraps ``POST /wechatweb/msg/chats`` and
yields normalized chat-message dicts (see Task 1.6).
* :func:`scrape_departments` — wraps ``POST /usercenter/dept/tree`` and
yields normalized :class:`app.db.models.Department` dicts. Single request,
no pagination; the tree (flat or nested) is flattened and the materialized
``path`` is reconstructed from parent links (Task 2.0).
* :func:`scrape_wechat_accounts` — wraps ``POST /wechatweb/wxUser/list`` and
yields normalized :class:`app.db.models.WechatAccount` dicts (plus the
embedded ``userId``/``userName`` staff pair). The endpoint binds
``@RequestParam`` from a **form-urlencoded** body (NOT JSON) and REQUIRES
``isOnJob`` / ``departmentId`` (the company-root dept id) / ``startdate`` /
``enddate`` or it answers ``code -90000`` "缺少..."; a recent ~7-day
window works (the date affects per-account stats, not which accounts are
returned). Page-number pagination (Task 2.1).
* :func:`scrape_wechat_groups` — wraps ``POST /wechatweb/wxFriends/friend_list/es``
with ``friendTypeYunke="2"`` in the body (the SAME endpoint friends use;
``"1"`` selects friends, ``"2"`` selects groups) and yields normalized
:class:`app.db.models.ChatGroup` dicts. The response envelope is identical
to the friend scraper's, so the shared ``_parse_total`` / ``_parse_rows``
helpers are reused. Member details are NOT available from this endpoint —
``member_ids`` / ``member_ids_list`` are emitted as ``None`` / ``[]``
(group list works; member mirroring is deferred). Page-number pagination.
* :func:`scrape_wechat_talkers` — wraps ``POST /wechatweb/wxFriends/talkerList``
and yields the per-staff-wechat conversation list (M2). Used by the sync
orchestrator to decide which conversations have recent traffic to pull.
Page-number pagination; the owning ``wechatId`` is sent in every request.
* :func:`scrape_wechat_voice_calls` — wraps ``POST /wechatweb/callRecord/queryWeChatVoiceList``
and yields normalized WeChat voice-call records (M2). Each record is mapped
to the ``PhoneCall`` column shape with ``channel="voice_call"``; the
recording URL (``ossFileName``) lands on ``record_file``. Page-number
pagination.
All yield dicts (not ORM). Persistence is the caller's job (the
:mod:`app.storage.repos` module bridges these dicts to the ORM tables). Each
dict carries:
* snake_case keys matching the target ORM columns (see :func:`_normalize`
for friends, :func:`_normalize_message` for messages,
:func:`_normalize_department` for departments,
:func:`_normalize_wxuser_account` for accounts,
:func:`_normalize_chat_group` for groups,
:func:`_normalize_talker` for talkers, and
:func:`_normalize_voice_call` for voice calls),
* ``channel == "wechat"`` (only on communication-channel entities; the
organization entities ``Department`` / ``WechatAccount`` carry no channel),
* ``raw_json`` — the original upstream record, verbatim, so nothing captured
is lost even when a column has no target field.
Friend field names are mapped from the real capture (see Task 1.5 spec). The
mapping is a straight ``camelCase -> snake_case`` translation with a couple of
disambiguations where the source has two fields for one column:
* ``phone`` <- ``phone``
* ``wechat_phone`` <- ``wechatPhone``
* ``dept_name`` <- ``deptName`` (falls back to ``deptRouteName``)
* ``head_image_url`` <- ``friendHeadImageUrl`` (falls back to ``headImg``)
"""
from __future__ import annotations
import datetime as dt
from collections.abc import AsyncIterator
from typing import Any
from app.scraper.base import paginate_by_cursor, paginate_by_page
from app.scraper.parsers import normalize_wechat_type, parse_datetime_to_ms
from app.session.yunke_client import YunkeClient
__all__ = [
"scrape_departments",
"scrape_wechat_accounts",
"scrape_wechat_friends",
"scrape_wechat_groups",
"scrape_wechat_messages",
"scrape_wechat_talkers",
"scrape_wechat_voice_calls",
]
FRIEND_LIST_PATH = "/wechatweb/wxFriends/friend_list/es"
MSG_CHATS_PATH = "/wechatweb/msg/chats"
DEPT_TREE_PATH = "/usercenter/dept/tree"
WXUSER_LIST_PATH = "/wechatweb/wxUser/list"
TALKER_LIST_PATH = "/wechatweb/wxFriends/talkerList"
VOICE_LIST_PATH = "/wechatweb/callRecord/queryWeChatVoiceList"
# Upstream sends form-encoded bodies for ``msg/chats`` (NOT JSON), so we have
# to override the client's default JSON content-type on every request.
_FORM_CONTENT_TYPE = "application/x-www-form-urlencoded"
def _parse_total(resp: Any) -> int | None:
"""Read the declared row count from the friend_list envelope.
Shape: ``data.basicResult.total`` (an int). Returns ``None`` if the path
is absent so the paginator falls back to the empty-page stop signal.
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return None
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return None
basic = data.get("basicResult")
if not isinstance(basic, dict):
return None
total = basic.get("total")
return int(total) if isinstance(total, (int, float)) else None
def _parse_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the list of friend records from the envelope.
Shape: ``data.basicResult.data`` (a list of records). Missing/non-list
values collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
basic = data.get("basicResult")
if not isinstance(basic, dict):
return []
rows = basic.get("data")
return rows if isinstance(rows, list) else []
def _build_body(
page_num: int,
page_size: int,
*,
department_id: str = "",
user_id: str = "",
) -> dict[str, Any]:
"""Build the friend_list request body for a given page.
The real client sends ``departmentId`` / ``userId`` filters; we accept
them as optional kwargs (default empty) so per-staff / per-department
scrapes are a drop-in without changing the paginator contract.
"""
body: dict[str, Any] = {"pageNum": page_num, "pageSize": page_size}
if department_id:
body["departmentId"] = department_id
if user_id:
body["userId"] = user_id
return body
# WeChat ``addTime`` is captured as ``'YYYY-MM-DD HH:mm:ss'`` (the common live
# shape; confirmed in smoke.db raw_json). Some captures instead carry a numeric
# ms-epoch. Both formats are accepted; first match wins.
_ADD_TIME_FORMATS = ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M")
def _parse_add_time(value: Any) -> int | None:
"""Coerce a wechat ``addTime`` to a ms-epoch ``int`` (naive UTC).
Accepted shapes:
* ``int`` / ``float`` — treated as ms-epoch already, returned as ``int``.
* ``str`` matching one of :data:`_ADD_TIME_FORMATS` — parsed via the shared
:func:`app.scraper.parsers.parse_datetime_to_ms` (naive UTC, same
convention as ``lastUpdate`` / ``startTime`` elsewhere in this layer).
* numeric ``str`` (e.g. ``"1690000000000"``) -> ``int`` (no datetime
round-trip).
* ``None`` / empty / unparseable -> ``None`` (None-safe field).
"""
return parse_datetime_to_ms(value, _ADD_TIME_FORMATS)
def _normalize(record: dict[str, Any]) -> dict[str, Any]:
"""Map one upstream friend record to the ``Customer`` column shape.
Adds ``channel="wechat"`` and preserves the original record under
``raw_json``. ``None``-safe: missing fields become ``None`` rather than
raising, so partial payloads do not break the whole scrape.
"""
return {
"friend_id": record.get("friendWxId"),
"channel": "wechat",
"owner_account_id": record.get("wxId"),
"friend_nickname": record.get("friendWxNickname"),
"friend_remark": record.get("friendWxRemark"),
"real_name": record.get("realName"),
"phone": record.get("phone"),
"wechat_phone": record.get("wechatPhone"),
"gender": record.get("gender"),
"region": record.get("region"),
"description": record.get("description"),
"sale_name": record.get("saleName"),
"user_id": record.get("userId"),
"dept_route": record.get("deptRoute"),
# Prefer the canonical deptName; fall back to the route-display name.
"dept_name": record.get("deptName") or record.get("deptRouteName"),
# Prefer friendHeadImageUrl; fall back to the legacy headImg field.
"head_image_url": record.get("friendHeadImageUrl") or record.get("headImg"),
"last_chat_time": record.get("lastChatTime"),
# addTime is delivered as 'YYYY-MM-DD HH:mm:ss' (confirmed in smoke.db
# raw_json); some captures carry a numeric ms-epoch instead. Both shapes
# collapse to a ms-epoch int via _parse_add_time; None when missing/empty.
"add_time": _parse_add_time(record.get("addTime")),
"send_num": record.get("sendNum"),
"receive_num": record.get("receiveNum"),
"message_count": record.get("messageCount"),
"delete_flag": record.get("deleteFlag"),
"is_block": record.get("isBlock"),
"friend_state": record.get("friendState"),
"version": record.get("version"),
"raw_json": record,
}
async def scrape_wechat_friends(
client: YunkeClient,
*,
department_id: str = "",
user_id: str = "",
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat friend records from the CRM.
Walks ``POST /wechatweb/wxFriends/friend_list/es`` page by page until the
upstream reports no more rows, mapping each record to the ``Customer``
column shape (snake_case) plus a verbatim ``raw_json`` copy.
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client; this scraper only concerns itself with the
friend-list endpoint contract.
department_id, user_id:
Optional upstream filters (passed through to the request body as
``departmentId`` / ``userId``). Empty by default — scrape everything
the logged-in account can see.
page_size:
Page size to request. Defaults to 50 (a sane middle ground between
request count and per-response payload size for this endpoint).
"""
async for record in paginate_by_page(
client,
FRIEND_LIST_PATH,
page_size=page_size,
parse_total=_parse_total,
parse_rows=_parse_rows,
build_body=_build_body,
department_id=department_id,
user_id=user_id,
):
yield _normalize(record)
# ===========================================================================
# Message scraper (Task 1.6) — POST /wechatweb/msg/chats
# ===========================================================================
def _coerce_int_or_none(value: Any) -> int | None:
"""Coerce a possibly-string numeric value to ``int``; ``None`` preserved.
Mirrors :func:`app.storage.repos._coerce_int` so the scraper can emit a
ready-to-store ``type_raw`` int and check raw type codes (e.g. the ``-2``
divider drop) without depending on the storage layer (the scraper has no
DB dependency by design). A non-numeric value (``"text"`` / ``None`` /
missing) returns ``None``.
"""
if value is None:
return None
if isinstance(value, bool):
return int(value)
if isinstance(value, int):
return value
if isinstance(value, float):
return int(value)
if isinstance(value, str):
s = value.strip()
if not s:
return None
try:
return int(s)
except ValueError:
return None
return None
async def _fetch_msg_page(
client: YunkeClient,
*,
wechat_id: str,
friend_id: str,
types: str,
start: int,
end_cursor: int,
) -> tuple[list[dict[str, Any]], int, bool]:
"""Fetch one backward page of ``msg/chats``.
Returns the ``(rows, next_cursor, has_more)`` triple expected by
:func:`paginate_by_cursor`. ``next_cursor`` is ``data.end - 1`` so the
next call's window is ``(end-1, ...]`` — i.e. strictly older than the
oldest message in this batch.
"""
resp = await client.post(
MSG_CHATS_PATH,
data={
"wechatId": wechat_id,
"friendId": friend_id,
"types": types,
"start": str(start),
"end": str(end_cursor),
},
headers={"content-type": _FORM_CONTENT_TYPE},
)
try:
body = resp.json()
except (AttributeError, ValueError):
# Malformed envelope — treat as empty + no-more so the paginator stops.
return [], end_cursor, False
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return [], end_cursor, False
rows = data.get("messages")
if not isinstance(rows, list):
rows = []
end = data.get("end")
next_cursor = (
int(end) - 1 if isinstance(end, (int, float)) and not isinstance(end, bool)
else end_cursor
)
has_more = bool(data.get("hasLast"))
return rows, next_cursor, has_more
def _normalize_message(record: dict[str, Any]) -> dict[str, Any]:
"""Map one upstream ``msg/chats`` record to the ``Message`` column shape.
Adds ``channel="wechat"`` and preserves the original record under
``raw_json``. Per design §4.3 the message type is carried as TWO fields:
* ``type`` — the normalized enum label (``"text"`` / ``"unknown"`` / ...)
via :func:`normalize_wechat_type`.
* ``type_raw`` — the original upstream type code (``messageType`` falling
back to ``type``), coerced to ``int | None`` so the raw code survives
end-to-end without round-tripping through ``raw_json``.
``None``-safe: missing fields become ``None`` rather than raising, so a
partial payload does not break the whole scrape. Value coercion
(e.g. ``isSend``/``mine``/``isDel`` ``int`` -> ``bool``) is deferred to
the persistence layer — here we carry the raw values through, except for
``is_send`` which is pulled from ``mine`` (falling back to ``isSend``)
since that disambiguation is straightforward.
"""
# `mine` and `isSend` carry the same semantic; the live capture uses both.
# Prefer `mine`, fall back to `isSend`. Missing-key .get() returns None,
# but `mine` may legitimately be the int 0 (falsy but present) — guard
# explicitly with `is None`.
mine = record.get("mine")
is_send = mine if mine is not None else record.get("isSend")
raw_type = record.get("messageType")
if raw_type is None:
raw_type = record.get("type")
# `timestamp` is the canonical field; fall back to `createTime`.
timestamp = record.get("timestamp")
if timestamp is None:
timestamp = record.get("createTime")
# talker_id = the message's sender/peer FULL id (the timeline join key).
# 1:1 (empty roomid): upstream `talker` is masked, the full peer is
# `oriTalker`. group (roomid set): `talker` is the sending member, while
# `oriTalker` is the chatroom id — so the member is in `talker`.
roomid = record.get("roomid") or ""
if roomid:
talker_id = record.get("talker")
else:
talker_id = record.get("oriTalker") or record.get("talker")
return {
"msg_id": record.get("msgSvrId"),
"channel": "wechat",
"owner_account_id": record.get("wechatId"),
"talker_id": talker_id,
"ori_talker": record.get("oriTalker"),
"room_id": record.get("roomid"),
"is_send": is_send,
"type": normalize_wechat_type(raw_type),
"type_raw": _coerce_int_or_none(raw_type),
"text": record.get("text"),
"file_url": record.get("file"),
"file_th_url": record.get("fileTh"),
"file_hd_url": record.get("fileHd"),
"md5": record.get("md5"),
"md5_th": record.get("md5Th"),
"md5_hd": record.get("md5Hd"),
"duration": record.get("duration"),
"transcription_text": record.get("transcriptionText"),
"img_path": record.get("imgPath"),
"original_image": record.get("originalImage"),
"pic_type": record.get("picType"),
"pay_msg_id": record.get("paymsgid"),
"refer_msg": record.get("referMsg"),
"sender_name": record.get("name"),
"sender_head_url": record.get("headUrl"),
"timestamp": timestamp,
"is_del": record.get("isDel"),
"version": record.get("version"),
"raw_json": record,
}
def _is_divider(record: dict[str, Any]) -> bool:
"""True iff this record is a date-divider (raw type code ``-2``).
These are not real messages — the upstream inserts them as visual date
bars in the chat UI — and must never be yielded or persisted. Per the
Task 1.6 spec we drop a record when EITHER ``messageType`` or ``type`` is
``-2``; all other unknown codes pass through.
The check is on the RAW code (not the normalized label) so the drop
decision is robust against any future change to
:func:`normalize_wechat_type`'s label vocabulary.
"""
return _coerce_int_or_none(record.get("messageType")) == -2 or (
_coerce_int_or_none(record.get("type")) == -2
)
async def scrape_wechat_messages(
client: YunkeClient,
wechat_id: str,
friend_id: str,
start: int,
end: int,
*,
types: str = "",
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat chat messages, paging backward by timestamp.
Wraps ``POST /wechatweb/msg/chats`` (form-encoded body, NOT JSON). The
endpoint pages BACKWARD: each response carries ``data.end`` (oldest
timestamp in this batch) and ``data.hasLast`` (older messages exist?).
The scraper advances the cursor to ``end - 1`` and repeats until
``hasLast`` is false, then stops.
Date dividers (``type == -2``) are dropped — they are UI chrome, not real
messages. All other type codes pass through (mapped to ``"unknown"`` when
not in our small known-table) so no real message is silently lost.
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
wechat_id:
Staff WeChat id (``wechatId`` form field) — maps to
``owner_account_id`` on each yielded record.
friend_id:
Friend ``wxid`` OR chatroom id (``friendId`` form field) — selects
the conversation to pull.
start, end:
Time window in ms epoch. ``end`` is also the initial backward cursor;
the first request queries the ``(start, end]`` window and the cursor
walks toward ``start`` from there.
types:
Optional upstream message-type filter (passed through verbatim to the
form body). Empty by default — pull all types.
"""
async for record in paginate_by_cursor(
lambda end_cursor: _fetch_msg_page(
client,
wechat_id=wechat_id,
friend_id=friend_id,
types=types,
start=start,
end_cursor=end_cursor,
),
initial_cursor=end,
):
if _is_divider(record):
continue
yield _normalize_message(record)
# ===========================================================================
# Department tree scraper (Task 2.0) — POST /usercenter/dept/tree
# ===========================================================================
def _extract_dept_node_list(resp: Any) -> list[dict[str, Any]]:
"""Extract the raw node list from a ``dept/tree`` response, defensively.
The envelope's ``data`` may be:
* a list of nodes (the common case), OR
* an object wrapping the node list under ``children`` (or ``list`` /
``tree`` / ``data`` — be tolerant of wrapper-key variance).
Returns ``[]`` on any malformed shape so the scraper yields nothing rather
than raising: a broken upstream should not abort the whole sync. Nodes are
returned AS-IS (possibly still nested via ``children``); flattening is the
next step (:func:`_flatten_dept_tree`).
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if isinstance(data, list):
return data
if isinstance(data, dict):
# Walk the plausible wrapper keys in priority order. ``children`` is
# the documented yunke tree convention; the others are defensive.
for key in ("children", "list", "tree", "data"):
val = data.get(key)
if isinstance(val, list):
return val
return []
def _flatten_dept_tree(nodes: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Flatten a (possibly nested) dept-tree node list depth-first.
Each node MAY carry a ``children`` array (the nested shape); when absent,
the tree is already flat and parents are carried via ``pId``. Both shapes
flatten to the same depth-first list.
Each emitted node is a shallow copy of the source node with the
``children`` key stripped, so the ``raw_json`` we store per-dept does not
carry a recursive subtree (only that node's own attributes). Non-dict
entries are dropped defensively.
"""
flat: list[dict[str, Any]] = []
# Iterative DFS via a stack. Nodes are pushed in reverse so the first
# child is popped first (preserves source order in the output).
stack: list[dict[str, Any]] = [
n for n in reversed(nodes) if isinstance(n, dict)
]
while stack:
node = stack.pop()
children = node.get("children")
# Shallow copy without ``children`` so raw_json stored per-dept is clean.
flat_node = {k: v for k, v in node.items() if k != "children"}
flat.append(flat_node)
if isinstance(children, list) and children:
stack.extend(
reversed([c for c in children if isinstance(c, dict)])
)
return flat
def _build_dept_paths(nodes: list[dict[str, Any]]) -> dict[str, str]:
"""Compute the materialized path (ancestor names joined by ``/``) per node.
Returns ``{id: path}``. The path is built by walking up the ``pId`` chain;
the walk terminates when it reaches a node whose ``pId`` is falsy
(``None`` / ``""`` / ``0`` / ``"0"``) OR whose parent is not in the tree
(handles the common yunke convention where root carries ``pId="0"`` and no
node with ``id="0"`` exists). A visited set caps the walk so a cyclic
``pId`` chain cannot loop forever.
A node whose own ``id`` is missing is skipped (no path entry) — the caller
treats a missing path as empty string.
"""
by_id: dict[str, dict[str, Any]] = {}
for n in nodes:
nid = n.get("id")
# First occurrence wins so duplicate ids cannot shadow the canonical
# node (defensive against an upstream that emits a node twice).
if nid is not None and nid not in by_id:
by_id[nid] = n
paths: dict[str, str] = {}
for nid, node in by_id.items():
parts: list[str] = []
seen: set[str] = set()
cur: dict[str, Any] | None = node
while cur is not None:
cid = cur.get("id")
if cid is None or cid in seen:
break
seen.add(cid)
name = cur.get("name")
if name is not None:
parts.append(name)
pid = cur.get("pId")
# Falsy pId OR unknown parent terminates the walk.
cur = by_id.get(pid) if pid else None
paths[nid] = "/".join(reversed(parts))
return paths
def _normalize_department(
node: dict[str, Any], path: str
) -> dict[str, Any]:
"""Map one dept-tree node to the ``Department`` column shape.
``parent_id`` is carried verbatim from ``pId`` — the scraper does NOT
synthesize ``NULL`` for root nodes; downstream code treats falsy
``parent_id`` (``"0"`` / ``""`` / ``None``) as "no parent" consistently
with the materialized-path builder. ``path`` is the precomputed
slash-joined ancestor-name chain from :func:`_build_dept_paths`.
"""
return {
"dept_id": node.get("id"),
"parent_id": node.get("pId"),
"name": node.get("name"),
"path": path,
"raw_json": node,
}
async def scrape_departments(
client: YunkeClient,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized department records from ``POST /usercenter/dept/tree``.
Single request (no pagination). The endpoint follows yunke's tree
convention: nodes carry ``id`` / ``pId`` / ``name`` and MAY be delivered
either as a flat list (parents via ``pId``) or as a nested tree (each node
optionally carrying ``children``). Both shapes are flattened to one flat
list of records, each carrying the materialized ``path`` (ancestor names
joined with ``/``).
Defensive envelope: ``data`` may be the node list directly OR an object
wrapping the node list under ``children``; both are unwrapped.
The async-generator signature mirrors the other scrapers so callers can
consume every entity stream with the same ``async for`` pattern, even
though this endpoint always resolves in one round-trip.
"""
resp = await client.post(DEPT_TREE_PATH, json={})
nodes = _extract_dept_node_list(resp)
flat = _flatten_dept_tree(nodes)
paths = _build_dept_paths(flat)
for node in flat:
nid = node.get("id")
path = paths.get(nid, "") if nid is not None else ""
yield _normalize_department(node, path)
# ===========================================================================
# WeChat staff-account scraper (Task 2.1) — POST /wechatweb/wxUser/list
# ===========================================================================
def _parse_wxuser_total(resp: Any) -> int | None:
"""Read the declared row count from the ``wxUser/list`` envelope.
Shape: ``data.total`` (an int). Returns ``None`` if the path is absent so
the paginator falls back to the empty-page stop signal.
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return None
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return None
total = data.get("total")
return int(total) if isinstance(total, (int, float)) and not isinstance(
total, bool
) else None
def _parse_wxuser_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the account list from the ``wxUser/list`` envelope.
Shape: ``data.data`` (a list of records). Missing / non-list values
collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
rows = data.get("data")
return rows if isinstance(rows, list) else []
def _build_wxuser_body(
page_num: int,
page_size: int,
*,
department_id: str,
is_on_job: str = "1",
start: str,
end: str,
) -> dict[str, Any]:
"""Build the form-urlencoded ``wxUser/list`` body for a given page.
Phase-0 live probing established the real contract: the endpoint binds
Spring ``@RequestParam`` from a form body (NOT JSON) and REQUIRES the
filters below or it answers ``code -90000`` ("缺少..."). Empty
``userId`` / ``salesname`` / ``weChatNickName`` are sent explicitly because
the real client does, and the upstream treats missing keys differently
from empty strings for some filters.
"""
return {
"pageNum": page_num,
"pageSize": page_size,
# REQUIRED — empty fails with code -90000. "1" => on-job accounts only
# (the 55 live rows). "0" would pull departed staff instead.
"isOnJob": is_on_job,
# REQUIRED — the company ROOT dept id (caller obtains it from
# scrape_departments). NOT hardcoded here.
"departmentId": department_id,
# REQUIRED — "缺少时间参数" if absent. YYYY-MM-DD strings. The window
# affects per-account stats, NOT which accounts are returned.
"startdate": start,
"enddate": end,
# Always-empty free-text filters (sent to mirror the real client).
"userId": "",
"salesname": "",
"weChatNickName": "",
}
def _default_date_window(now: dt.datetime | None = None) -> tuple[str, str]:
"""A recent 7-day window as ``(start, end)`` ``YYYY-MM-DD`` strings.
The ``wxUser/list`` stats window is cosmetic (it affects per-account
counters, not which accounts are returned), so any recent window works.
We default to the last 7 days measured from ``now`` (UTC by default to
stay deterministic and match the rest of the scraper layer's naive-UTC
convention). Both bounds are inclusive calendar days.
"""
now = now or dt.datetime.now(dt.UTC)
end_date = now.date()
start_date = end_date - dt.timedelta(days=7)
return start_date.isoformat(), end_date.isoformat()
def _normalize_wxuser_account(record: dict[str, Any]) -> dict[str, Any]:
"""Map one upstream ``wxUser/list`` record to the ``WechatAccount`` shape.
Carries the embedded staff pair (``user_id`` <- ``userId``,
``user_name`` <- ``userName``) so the persistence layer can upsert the
owning :class:`app.db.models.Staff` row alongside the account. Adds
``raw_json`` (the verbatim upstream record). ``None``-safe: missing fields
become ``None`` rather than raising, so partial payloads do not break the
scrape.
"""
return {
"wx_id": record.get("wechatId"),
"nick_name": record.get("nickName"),
"alias": record.get("alias"),
"phone": record.get("phone"),
"user_id": record.get("userId"),
"user_name": record.get("userName"),
"contacts_num": record.get("contactsNum"),
"group_num": record.get("groupNum"),
"send_num": record.get("sendNum"),
"receive_num": record.get("receiveNum"),
"is_on_job": record.get("isOnJob"),
"db_size": record.get("dbSize"),
"online_status": record.get("onlineStatus"),
"last_update": record.get("lastUpdate"),
"version": record.get("version"),
"raw_json": record,
}
async def scrape_wechat_accounts(
client: YunkeClient,
*,
department_id: str,
page_size: int = 50,
is_on_job: str = "1",
start: str | None = None,
end: str | None = None,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat staff-account records from ``wxUser/list``.
Walks ``POST /wechatweb/wxUser/list`` page by page until the upstream
reports no more rows (empty ``data.data`` array — the primary stop signal),
mapping each record to the ``WechatAccount`` column shape plus a verbatim
``raw_json`` copy.
The request is **form-urlencoded** (NOT JSON): the endpoint binds Spring
``@RequestParam`` from the form body and REQUIRES ``isOnJob`` /
``departmentId`` / ``startdate`` / ``enddate``. ``department_id`` is the
company-root dept id (the caller resolves it from
:func:`scrape_departments`); it is NOT hardcoded here so this scraper
stays account-agnostic.
The envelope differs from ``friend_list/es``: there is **no**
``basicResult`` wrapper. The pagination fields sit directly under ``data``
(``data.data`` / ``data.total`` / ``data.pageNum`` / ``data.pageSize``).
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
department_id:
Company-root department id (REQUIRED upstream filter). The caller
obtains it from :func:`scrape_departments` (the root node whose
``parent_id`` is falsy).
page_size:
Page size to request. Defaults to 50 (matches the live capture's
observed page size for this endpoint).
is_on_job:
``"1"`` (default) for on-job staff, ``"0"`` for departed. Required
upstream filter — empty fails with ``code -90000``.
start, end:
Optional ``YYYY-MM-DD`` bounds for the per-account stats window. The
date affects per-account counters, NOT which accounts are returned
(live total is 55 regardless), so any recent window works. Defaults
to a recent 7-day window computed from ``datetime.now(UTC)``.
"""
default_start, default_end = _default_date_window()
window_start = start or default_start
window_end = end or default_end
async for record in paginate_by_page(
client,
WXUSER_LIST_PATH,
page_size=page_size,
parse_total=_parse_wxuser_total,
parse_rows=_parse_wxuser_rows,
build_body=_build_wxuser_body,
form=True,
department_id=department_id,
is_on_job=is_on_job,
start=window_start,
end=window_end,
):
yield _normalize_wxuser_account(record)
# ===========================================================================
# Chat-group scraper (M2) — POST /wechatweb/wxFriends/friend_list/es
# (friendTypeYunke="2" selects groups; "1" selects friends)
# ===========================================================================
def _build_group_body(page_num: int, page_size: int) -> dict[str, Any]:
"""Build the ``friend_list/es`` body for chat-groups (friendTypeYunke=2).
Groups share the friend-list endpoint; the ``friendTypeYunke`` body
parameter is the switch (``"1"`` = friends, ``"2"`` = groups). Verified
against the live CRM (2026-08-10 Playwright capture) — the previous
``POST /wechatweb/chatroom/getChatroomList`` endpoint returned 0 rows.
"""
return {"pageNum": page_num, "pageSize": page_size, "friendTypeYunke": "2"}
def _normalize_chat_group(record: dict[str, Any]) -> dict[str, Any]:
"""Map one ``friend_list/es`` group record (friendTypeYunke=2) to ChatGroup.
Field mapping (verified 2026-08-10 Playwright capture):
* ``friendWxId`` -> ``room_id`` (the ``@chatroom`` id)
* ``friendWxNickname`` -> ``room_name`` (the group name)
* ``friendNum`` -> ``member_count``
* ``friendHeadImageUrl`` -> ``head_image_url`` (group avatar)
* ``wxId`` -> ``owner_account_id`` (staff wechat account)
* ``ownerName`` -> ``owner_account_nickname`` (group owner nickname)
* ``userId`` -> ``owner_user_id`` (owning staff id)
* ``phone`` / ``wechatPhone`` -> ``owner_user_phone`` (phone preferred)
* ``messageCount`` -> ``message_count``
* ``sendNum`` -> ``own_message_count`` (this account's sends)
* ``lastChatTime`` -> ``last_chat_time`` (ms-epoch)
* ``version`` -> ``version``
Member ids are NOT surfaced by ``friend_list/es`` (only the ``friendNum``
count is); ``member_ids`` / ``member_ids_list`` are emitted as ``None`` /
``[]`` so the column shape stays stable. Group-member mirroring is
deferred to a future endpoint. Likewise ``loss_members`` / ``add_members``
/ ``update_time`` are not carried by this endpoint and are emitted as
``None``.
Adds ``channel="wechat"`` and preserves the original record under
``raw_json``. ``None``-safe on every field: missing fields become ``None``
rather than raising, so a partial payload does not break the scrape.
2026-08-25 补充列（对齐云客 员工聊天窗口·微信群 tab，真机验证 raw_json 键）：
* ``ownerName`` -> ``owner_name`` (群主昵称，上游脱敏)
* ``ownerWchatId`` -> ``owner_wxid`` (群主微信号，上游脱敏)
* ``ownerAlias`` -> ``owner_alias`` (群主微信号别名)
* ``ownerImage`` -> ``owner_avatar`` (群主头像)
* ``friendCount`` -> ``friend_count`` (群成员中员工好友数)
* ``createTime`` -> ``create_time`` (建群时间;实测 smoke.db 全量为
``'YYYY-MM-DD HH:mm:ss'`` 字符串,另兼容 ms-int/数字串,统一归一为
ms-epoch int,复用 ``_parse_add_time`` 的双形态解析)
"""
return {
"room_id": record.get("friendWxId"),
"channel": "wechat",
"room_name": record.get("friendWxNickname"),
"owner_account_id": record.get("wxId"),
"owner_account_nickname": record.get("ownerName"),
"member_count": record.get("friendNum"),
"member_ids": None,
"member_ids_list": [],
"head_image_url": record.get("friendHeadImageUrl"),
"owner_user_id": record.get("userId"),
"owner_user_phone": record.get("phone") or record.get("wechatPhone"),
"message_count": record.get("messageCount"),
"own_message_count": record.get("sendNum"),
"loss_members": None,
"add_members": None,
"last_chat_time": record.get("lastChatTime"),
"update_time": None,
# 群主/建群补充列(2026-08-25;raw_json 键 -> 同名列)
"owner_name": record.get("ownerName"),
"owner_wxid": record.get("ownerWchatId"),
"owner_alias": record.get("ownerAlias"),
"owner_avatar": record.get("ownerImage"),
"friend_count": _coerce_int_or_none(record.get("friendCount")),
"create_time": _parse_add_time(record.get("createTime")),
"version": record.get("version"),
"raw_json": record,
}
async def scrape_wechat_groups(
client: YunkeClient,
*,
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat chat-group records from ``friend_list/es``.
Walks ``POST /wechatweb/wxFriends/friend_list/es`` page by page (the SAME
endpoint :func:`scrape_wechat_friends` uses) with ``friendTypeYunke="2"``
in the body to select groups (``"1"`` selects friends). The response
envelope is identical to the friend scraper's (``data.basicResult.data``
+ ``data.basicResult.total``), so the shared ``_parse_total`` /
``_parse_rows`` helpers are reused.
Each record is mapped to the ``ChatGroup`` column shape plus a verbatim
``raw_json`` copy. Member ids are NOT available from this endpoint —
``member_ids`` / ``member_ids_list`` are emitted as ``None`` / ``[]``.
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
page_size:
Page size to request. Defaults to 50 (a sane middle ground between
request count and per-response payload size for this endpoint).
"""
async for record in paginate_by_page(
client,
FRIEND_LIST_PATH,
page_size=page_size,
parse_total=_parse_total,
parse_rows=_parse_rows,
build_body=_build_group_body,
):
yield _normalize_chat_group(record)
# ===========================================================================
# Talker (conversation) list scraper (M2) — POST /wechatweb/wxFriends/talkerList
# ===========================================================================
def _parse_talker_total(resp: Any) -> int | None:
"""Read the declared row count from the ``talkerList`` envelope.
The talker endpoint signals "more pages" via ``data.hasMore`` (a boolean)
rather than a declared ``total``. There is no honest row count to read,
so this always returns ``None`` — the paginator then relies solely on the
empty-page stop signal (a final page whose ``data.page`` is empty).
"""
return None
def _parse_talker_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the talker list from the ``talkerList`` envelope.
Shape: ``data.page`` (a list of records). Missing / non-list values
collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
rows = data.get("page")
return rows if isinstance(rows, list) else []
def _build_talker_body(
page_num: int, page_size: int, *, wechat_id: str
) -> dict[str, Any]:
"""Build the ``talkerList`` request body for a given page.
The endpoint is per-staff-wechat: the owning ``wechatId`` MUST be sent in
every request so the upstream returns that account's conversation list.
"""
return {
"wechatId": wechat_id,
"pageNum": page_num,
"pageSize": page_size,
}
def _normalize_talker(record: dict[str, Any], wechat_id: str) -> dict[str, Any]:
"""Map one upstream ``talkerList`` record to a talker dict.
The talker list is primarily a sync-planning input: it tells the
orchestrator which conversations (direct chats AND group chats) have
recent traffic. ``member_count > 0`` marks a group conversation (the
``friend_wechat_id`` is then a ``@chatroom`` id); ``member_count == 0``
marks a 1:1 friend chat. The owning staff wechat is stamped on every
record as ``owner_account_id``. ``raw_json`` preserves the upstream
record verbatim. ``None``-safe on every field.
"""
return {
"friend_wechat_id": record.get("friendWechatId"),
"owner_account_id": wechat_id,
"nick_name": record.get("nickName"),
"remark_name": record.get("remarkName"),
"display_name": record.get("displayName"),
"message_count": record.get("messageCount"),
"last_chat_time": record.get("lastChatTime"),
"member_count": record.get("memberCount"),
"head_url": record.get("headUrl"),
"raw_json": record,
}
async def scrape_wechat_talkers(
client: YunkeClient,
wechat_id: str,
*,
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield the per-staff-wechat conversation list from ``talkerList``.
Wraps ``POST /wechatweb/wxFriends/talkerList``. The endpoint is scoped to
one staff wechat account (``wechat_id`` → the ``wechatId`` request
field); each response carries a slice of that account's recent
conversations under ``data.page`` with a boolean ``data.hasMore``.
The response envelope carries no declared ``total`` (only ``hasMore``),
so this scraper relies solely on the empty-page stop signal: it advances
``pageNum`` until a page comes back with an empty ``data.page`` array.
A well-behaved upstream flips ``hasMore`` to false on the last real page
and then serves an empty page; the paginator stops on whichever signal
fires first.
Talkers are NOT persisted to a dedicated table — the sync orchestrator
consumes this stream to decide which conversations have messages to pull
(``member_count > 0`` ⇒ group chat whose ``friend_wechat_id`` is a
``@chatroom`` id).
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
wechat_id:
Staff WeChat id (``wechatId`` request field) — the owning account
whose conversation list is being scraped.
page_size:
Page size to request. Defaults to 50.
"""
async for record in paginate_by_page(
client,
TALKER_LIST_PATH,
page_size=page_size,
parse_total=_parse_talker_total,
parse_rows=_parse_talker_rows,
build_body=_build_talker_body,
wechat_id=wechat_id,
):
yield _normalize_talker(record, wechat_id)
# ===========================================================================
# Voice-call scraper (M2, Task 2.4) — POST /wechatweb/callRecord/queryWeChatVoiceList
# ===========================================================================
def _parse_voice_total(resp: Any) -> int | None:
"""Read the declared row count from the ``queryWeChatVoiceList`` envelope.
Shape: ``data.total`` (an int). Returns ``None`` if the path is absent so
the paginator falls back to the empty-page stop signal. (Live ``total``
is 333 per phase0-findings.md §4.)
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return None
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return None
total = data.get("total")
return int(total) if isinstance(total, (int, float)) and not isinstance(
total, bool
) else None
def _parse_voice_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the voice-call list from the ``queryWeChatVoiceList`` envelope.
Shape: ``data.data`` (a list of records). Missing / non-list values
collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
rows = data.get("data")
return rows if isinstance(rows, list) else []
def _build_voice_body(
page_num: int,
page_size: int,
*,
department_id: str = "",
call_source: str = "0",
start: str = "2000-01-01 00:00:00",
end: str = "2099-12-31 23:59:59",
) -> dict[str, Any]:
"""Build the ``queryWeChatVoiceList`` request body for a given page.
Live diagnosis (2026-08-07 browser capture) confirmed the endpoint
REQUIRES ``callSource`` + ``departmentId`` + ``isRoom`` + ``startTime`` +
``endTime`` — omit any one and the HTTP layer returns 200 with an EMPTY
``data.data`` array (0 rows). The WeChat variant uses
``call_source="0"`` (the WA variant in :mod:`app.scraper.whatsapp`
passes ``"4"``); a wide time window (2000-01-01 .. 2099-12-31) pulls
the full history in one scrape. ``userId`` / ``sort`` / ``ids`` may
stay empty. ``departmentId`` is omitted when blank so a test/fixture
call without one still forms a valid body.
"""
body: dict[str, Any] = {
"pageNum": page_num,
"pageSize": page_size,
"callSource": call_source,
"isRoom": "0",
"startTime": start,
"endTime": end,
"sort": "",
"ids": [],
"userId": "",
}
if department_id:
body["departmentId"] = department_id
return body
def _parse_voice_start_time(value: Any) -> int | None:
"""Coerce a voice-call ``startTime`` to a ms-epoch int (the PhoneCall column type).
Thin wrapper over the shared :func:`app.scraper.parsers.parse_datetime_to_ms`
so the voice-call scraper speaks the same datetime -> ms-epoch dialect as
the phone-call + SMS scrapers while keeping its capture-specific name. The
voice-call endpoint reports ``startTime`` as a ``YYYY-MM-DD HH:MM:SS``
datetime string (confirmed by the Task 2.4 spec); the shared helper also
accepts a numeric ms-epoch (int / float / numeric str) for robustness.
See :func:`parse_datetime_to_ms` for the full accepted-shapes contract and
the documented naive-UTC simplification (the CRM wall-clock times are
presumably CST; this parses them as UTC for determinism).
"""
return parse_datetime_to_ms(value, ["%Y-%m-%d %H:%M:%S"])
def _normalize_voice_call(record: dict[str, Any]) -> dict[str, Any]:
"""Map one upstream ``queryWeChatVoiceList`` record to the ``PhoneCall`` shape.
Adds ``channel="voice_call"`` and preserves the original record under
``raw_json``. Field mapping per phase0-findings.md §4 + the PhoneCall
column set:
* ``id`` -> ``call_id`` (PK)
* ``duration`` -> ``call_duration`` (string seconds coerced to int)
* ``startTime`` -> ``start_call_time`` (datetime string -> ms epoch;
see :func:`_parse_voice_start_time`)
* ``callStatus`` -> ``call_status`` (1=connected w/ recording,
2=no answer)
* ``isSend`` -> ``incoming_call`` with INVERTED polarity.
Voice-call ``isSend`` is ``0=incoming / 1=outgoing``; ``PhoneCall.
incoming_call`` (matching the ``phone_call`` channel) is
``1=incoming / 0=outgoing``. To keep the column's meaning identical
across both call channels we store ``incoming_call = 1 - isSend``
(``None`` preserved when ``isSend`` is missing/non-numeric).
* ``userId`` -> ``user_id`` (owning staff)
* ``userName`` -> ``user_id_name``
* ``talker`` -> ``call_number`` (the customer wxid; for a room
call this is the ``@chatroom`` id — there is no separate member-sender
field on the voice-call record)
* ``talkerNickName`` -> ``plan_customer_name``
* ``ossFileName`` -> ``record_file`` (the recording URL; may be null)
Non-applicable PhoneCall columns (``ring_time`` / ``sim_phone`` /
``sim_number`` / ``imei`` / ``is_upload`` / ``download_count`` / ``ai_asr``
/ ``qc_status`` / ``department_id``) are left ``None`` — the voice-call
payload carries none of them. The owning ``weChatId`` has no matching
PhoneCall column (there is no ``owner_account_id`` on this model), so it
is preserved only in ``raw_json``. ``None``-safe on every mapped field.
"""
is_send = _coerce_int_or_none(record.get("isSend"))
# Flip isSend (0=incoming/1=outgoing) to incoming_call (1=incoming/0=outgoing).
incoming_call = None if is_send is None else 1 - is_send
return {
"call_id": record.get("id"),
"channel": "voice_call",
"start_call_time": _parse_voice_start_time(record.get("startTime")),
"call_duration": _coerce_int_or_none(record.get("duration")),
"call_status": _coerce_int_or_none(record.get("callStatus")),
"incoming_call": incoming_call,
"user_id": record.get("userId"),
"user_id_name": record.get("userName"),
"call_number": record.get("talker"),
"plan_customer_name": record.get("talkerNickName"),
"record_file": record.get("ossFileName"),
"version": record.get("version"),
"raw_json": record,
}
async def scrape_wechat_voice_calls(
client: YunkeClient,
*,
department_id: str = "",
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat voice-call records from ``queryWeChatVoiceList``.
Walks ``POST /wechatweb/callRecord/queryWeChatVoiceList`` page by page
until the upstream reports no more rows (empty ``data.data`` array — the
primary stop signal), mapping each record to the ``PhoneCall`` column
shape with ``channel="voice_call"`` plus a verbatim ``raw_json`` copy.
The endpoint REQUIRES ``callSource`` + ``departmentId`` + ``isRoom`` +
a ``startTime``/``endTime`` window — without them it returns 200 + an
empty ``data.data`` (0 rows). The orchestrator resolves the
company-root ``department_id`` from the mirrored ``Department`` table
(see :func:`app.sync.orchestrator.sync_wechat_voice_calls`); the
default wide time window ``2000-01-01 .. 2099-12-31`` pulls the full
history in one scrape and is not exposed as a parameter.
The envelope is the same flat shape as ``wxUser/list`` (no
``basicResult`` wrapper): the row list and pagination fields sit directly
under ``data`` (``data.data`` / ``data.total`` / ``data.pageNum`` /
``data.pageSize``).
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
department_id:
Company-root department id (the ``departmentId`` form field).
Required by the live endpoint for non-empty results; resolved by
the orchestrator when omitted (empty skips the field in the body).
page_size:
Page size to request. Defaults to 50 (a sane middle ground between
request count and per-response payload size for this endpoint).
"""
async for record in paginate_by_page(
client,
VOICE_LIST_PATH,
page_size=page_size,
parse_total=_parse_voice_total,
parse_rows=_parse_voice_rows,
build_body=_build_voice_body,
department_id=department_id,
):
yield _normalize_voice_call(record)
Reference in New Issue View Git Blame Copy Permalink