fix(scraper): 群抓取改用 friend_list/es+friendTypeYunke=2(替换 getChatroomList)
getChatroomList 在云客真机返回 0 行(已 Playwright 2026-08-10 验证)。 群与好友共用 /wechatweb/wxFriends/friend_list/es,仅以 friendTypeYunke 区分 (1=好友 / 2=群),响应信封一致(basicResult),复用 _parse_total/_parse_rows。 - 删除 CHATROOM_LIST_PATH / _parse_chatroom_total / _parse_chatroom_rows / _build_chatroom_body / _parse_member_ids(改写后已无引用) - 新增 _build_group_body(pageNum/pageSize/friendTypeYunke="2") - 重写 _normalize_chat_group 映射 friend_list/es 字段 (friendWxId->room_id, friendWxNickname->room_name, friendNum->member_count, wxId/ownerName/userId, phone||wechatPhone, messageCount/sendNum/lastChatTime) - friend_list/es 不返回成员明细,member_ids/member_ids_list 固定 None/[] (群列表可用;成员镜像延后) - 删除 tests/cassettes/chatroom_fixture.py,改复用 wxFriends_fixture 信封 - 重写 test_scrape_wechat_groups:断言新端点 URL + friendTypeYunke="2" body + 新字段映射;member_ids 边界用例改为"始终 None/[]" scrape_wechat_groups 签名不变(client + page_size),orchestrator/scheduler 未改。 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
@@ -19,11 +19,14 @@ Endpoints covered today:
|
||||
``enddate`` or it answers ``code -90000`` "缺少..."; a recent ~7-day
|
||||
window works (the date affects per-account stats, not which accounts are
|
||||
returned). Page-number pagination (Task 2.1).
|
||||
* :func:`scrape_wechat_groups` — wraps ``POST /wechatweb/chatroom/getChatroomList``
|
||||
and yields normalized :class:`app.db.models.ChatGroup` dicts, parsing the
|
||||
semicolon-joined ``memberIds`` into a ``member_ids_list`` so the
|
||||
persistence layer can mirror :class:`app.db.models.GroupMember` rows
|
||||
(M2). Page-number pagination.
|
||||
* :func:`scrape_wechat_groups` — wraps ``POST /wechatweb/wxFriends/friend_list/es``
|
||||
with ``friendTypeYunke="2"`` in the body (the SAME endpoint friends use;
|
||||
``"1"`` selects friends, ``"2"`` selects groups) and yields normalized
|
||||
:class:`app.db.models.ChatGroup` dicts. The response envelope is identical
|
||||
to the friend scraper's, so the shared ``_parse_total`` / ``_parse_rows``
|
||||
helpers are reused. Member details are NOT available from this endpoint —
|
||||
``member_ids`` / ``member_ids_list`` are emitted as ``None`` / ``[]``
|
||||
(group list works; member mirroring is deferred). Page-number pagination.
|
||||
* :func:`scrape_wechat_talkers` — wraps ``POST /wechatweb/wxFriends/talkerList``
|
||||
and yields the per-staff-wechat conversation list (M2). Used by the sync
|
||||
orchestrator to decide which conversations have recent traffic to pull.
|
||||
@@ -84,7 +87,6 @@ FRIEND_LIST_PATH = "/wechatweb/wxFriends/friend_list/es"
|
||||
MSG_CHATS_PATH = "/wechatweb/msg/chats"
|
||||
DEPT_TREE_PATH = "/usercenter/dept/tree"
|
||||
WXUSER_LIST_PATH = "/wechatweb/wxUser/list"
|
||||
CHATROOM_LIST_PATH = "/wechatweb/chatroom/getChatroomList"
|
||||
TALKER_LIST_PATH = "/wechatweb/wxFriends/talkerList"
|
||||
VOICE_LIST_PATH = "/wechatweb/callRecord/queryWeChatVoiceList"
|
||||
|
||||
@@ -843,98 +845,69 @@ async def scrape_wechat_accounts(
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Chat-group scraper (M2) — POST /wechatweb/chatroom/getChatroomList
|
||||
# Chat-group scraper (M2) — POST /wechatweb/wxFriends/friend_list/es
|
||||
# (friendTypeYunke="2" selects groups; "1" selects friends)
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
def _parse_chatroom_total(resp: Any) -> int | None:
|
||||
"""Read the declared row count from the ``getChatroomList`` envelope.
|
||||
def _build_group_body(page_num: int, page_size: int) -> dict[str, Any]:
|
||||
"""Build the ``friend_list/es`` body for chat-groups (friendTypeYunke=2).
|
||||
|
||||
Shape: ``data.total`` (an int). Returns ``None`` if the path is absent so
|
||||
the paginator falls back to the empty-page stop signal.
|
||||
Groups share the friend-list endpoint; the ``friendTypeYunke`` body
|
||||
parameter is the switch (``"1"`` = friends, ``"2"`` = groups). Verified
|
||||
against the live CRM (2026-08-10 Playwright capture) — the previous
|
||||
``POST /wechatweb/chatroom/getChatroomList`` endpoint returned 0 rows.
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return None
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
total = data.get("total")
|
||||
return int(total) if isinstance(total, (int, float)) and not isinstance(
|
||||
total, bool
|
||||
) else None
|
||||
|
||||
|
||||
def _parse_chatroom_rows(resp: Any) -> list[dict[str, Any]]:
|
||||
"""Extract the chatroom list from the ``getChatroomList`` envelope.
|
||||
|
||||
Shape: ``data.data`` (a list of records). Missing / non-list values
|
||||
collapse to an empty list so the paginator sees "empty page".
|
||||
"""
|
||||
try:
|
||||
body = resp.json()
|
||||
except (AttributeError, ValueError):
|
||||
return []
|
||||
data = body.get("data") if isinstance(body, dict) else None
|
||||
if not isinstance(data, dict):
|
||||
return []
|
||||
rows = data.get("data")
|
||||
return rows if isinstance(rows, list) else []
|
||||
|
||||
|
||||
def _build_chatroom_body(page_num: int, page_size: int) -> dict[str, Any]:
|
||||
"""Build the ``getChatroomList`` request body for a given page.
|
||||
|
||||
The live capture carries no filter parameters — just the page slice.
|
||||
"""
|
||||
return {"pageNum": page_num, "pageSize": page_size}
|
||||
|
||||
|
||||
def _parse_member_ids(member_ids: Any) -> list[str]:
|
||||
"""Split a semicolon-joined member id list, dropping empty segments.
|
||||
|
||||
Upstream ``memberIds`` looks like ``"wxid_a;wxid_b;csf7169708"`` and may
|
||||
carry a trailing ``";"`` (yielding an empty segment). Empties are dropped
|
||||
so each entry is a real wxid; a missing/None/non-string value yields ``[]``.
|
||||
"""
|
||||
if not isinstance(member_ids, str):
|
||||
return []
|
||||
return [seg for seg in member_ids.split(";") if seg]
|
||||
return {"pageNum": page_num, "pageSize": page_size, "friendTypeYunke": "2"}
|
||||
|
||||
|
||||
def _normalize_chat_group(record: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Map one upstream ``getChatroomList`` record to the ``ChatGroup`` shape.
|
||||
"""Map one ``friend_list/es`` group record (friendTypeYunke=2) to ChatGroup.
|
||||
|
||||
Field mapping (verified 2026-08-10 Playwright capture):
|
||||
|
||||
* ``friendWxId`` -> ``room_id`` (the ``@chatroom`` id)
|
||||
* ``friendWxNickname`` -> ``room_name`` (the group name)
|
||||
* ``friendNum`` -> ``member_count``
|
||||
* ``friendHeadImageUrl`` -> ``head_image_url`` (group avatar)
|
||||
* ``wxId`` -> ``owner_account_id`` (staff wechat account)
|
||||
* ``ownerName`` -> ``owner_account_nickname`` (group owner nickname)
|
||||
* ``userId`` -> ``owner_user_id`` (owning staff id)
|
||||
* ``phone`` / ``wechatPhone`` -> ``owner_user_phone`` (phone preferred)
|
||||
* ``messageCount`` -> ``message_count``
|
||||
* ``sendNum`` -> ``own_message_count`` (this account's sends)
|
||||
* ``lastChatTime`` -> ``last_chat_time`` (ms-epoch)
|
||||
* ``version`` -> ``version``
|
||||
|
||||
Member ids are NOT surfaced by ``friend_list/es`` (only the ``friendNum``
|
||||
count is); ``member_ids`` / ``member_ids_list`` are emitted as ``None`` /
|
||||
``[]`` so the column shape stays stable. Group-member mirroring is
|
||||
deferred to a future endpoint. Likewise ``loss_members`` / ``add_members``
|
||||
/ ``update_time`` are not carried by this endpoint and are emitted as
|
||||
``None``.
|
||||
|
||||
Adds ``channel="wechat"`` and preserves the original record under
|
||||
``raw_json``. The raw ``memberIds`` string is carried verbatim as
|
||||
``member_ids`` (matching the ``ChatGroup.member_ids`` Text column) AND
|
||||
parsed into ``member_ids_list`` (a plain list of wxids with empties
|
||||
dropped) so :func:`app.storage.repos.upsert_chat_group` can mirror one
|
||||
:class:`app.db.models.GroupMember` row per entry without re-parsing.
|
||||
|
||||
``None``-safe: missing fields become ``None`` rather than raising, so a
|
||||
partial payload does not break the whole scrape.
|
||||
``raw_json``. ``None``-safe on every field: missing fields become ``None``
|
||||
rather than raising, so a partial payload does not break the scrape.
|
||||
"""
|
||||
member_ids = record.get("memberIds")
|
||||
return {
|
||||
"room_id": record.get("chatRoomId"),
|
||||
"room_id": record.get("friendWxId"),
|
||||
"channel": "wechat",
|
||||
"room_name": record.get("chatRoomName"),
|
||||
"owner_account_id": record.get("wechatId"),
|
||||
"owner_account_nickname": record.get("wechatNickName"),
|
||||
"member_count": record.get("memberCount"),
|
||||
"member_ids": member_ids,
|
||||
"member_ids_list": _parse_member_ids(member_ids),
|
||||
"head_image_url": record.get("headImageUrl"),
|
||||
"room_name": record.get("friendWxNickname"),
|
||||
"owner_account_id": record.get("wxId"),
|
||||
"owner_account_nickname": record.get("ownerName"),
|
||||
"member_count": record.get("friendNum"),
|
||||
"member_ids": None,
|
||||
"member_ids_list": [],
|
||||
"head_image_url": record.get("friendHeadImageUrl"),
|
||||
"owner_user_id": record.get("userId"),
|
||||
"owner_user_phone": record.get("userPhone"),
|
||||
"owner_user_phone": record.get("phone") or record.get("wechatPhone"),
|
||||
"message_count": record.get("messageCount"),
|
||||
"own_message_count": record.get("ownMessageCount"),
|
||||
"loss_members": record.get("lossMembers"),
|
||||
"add_members": record.get("addMembers"),
|
||||
"own_message_count": record.get("sendNum"),
|
||||
"loss_members": None,
|
||||
"add_members": None,
|
||||
"last_chat_time": record.get("lastChatTime"),
|
||||
"update_time": record.get("updateTime"),
|
||||
"update_time": None,
|
||||
"version": record.get("version"),
|
||||
"raw_json": record,
|
||||
}
|
||||
@@ -945,17 +918,18 @@ async def scrape_wechat_groups(
|
||||
*,
|
||||
page_size: int = 50,
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Yield normalized WeChat chat-group records from ``getChatroomList``.
|
||||
"""Yield normalized WeChat chat-group records from ``friend_list/es``.
|
||||
|
||||
Walks ``POST /wechatweb/chatroom/getChatroomList`` page by page until the
|
||||
upstream reports no more rows (empty ``data.data`` array — the primary
|
||||
stop signal), mapping each record to the ``ChatGroup`` column shape plus
|
||||
a verbatim ``raw_json`` copy.
|
||||
Walks ``POST /wechatweb/wxFriends/friend_list/es`` page by page (the SAME
|
||||
endpoint :func:`scrape_wechat_friends` uses) with ``friendTypeYunke="2"``
|
||||
in the body to select groups (``"1"`` selects friends). The response
|
||||
envelope is identical to the friend scraper's (``data.basicResult.data``
|
||||
+ ``data.basicResult.total``), so the shared ``_parse_total`` /
|
||||
``_parse_rows`` helpers are reused.
|
||||
|
||||
The semicolon-joined ``memberIds`` field is carried both raw
|
||||
(``member_ids``) and parsed (``member_ids_list``, empties dropped) so the
|
||||
persistence layer can mirror :class:`app.db.models.GroupMember` rows
|
||||
idempotently.
|
||||
Each record is mapped to the ``ChatGroup`` column shape plus a verbatim
|
||||
``raw_json`` copy. Member ids are NOT available from this endpoint —
|
||||
``member_ids`` / ``member_ids_list`` are emitted as ``None`` / ``[]``.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
@@ -968,11 +942,11 @@ async def scrape_wechat_groups(
|
||||
"""
|
||||
async for record in paginate_by_page(
|
||||
client,
|
||||
CHATROOM_LIST_PATH,
|
||||
FRIEND_LIST_PATH,
|
||||
page_size=page_size,
|
||||
parse_total=_parse_chatroom_total,
|
||||
parse_rows=_parse_chatroom_rows,
|
||||
build_body=_build_chatroom_body,
|
||||
parse_total=_parse_total,
|
||||
parse_rows=_parse_rows,
|
||||
build_body=_build_group_body,
|
||||
):
|
||||
yield _normalize_chat_group(record)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user