xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: wechat group+member and talker scrapers

Browse Source 
Add scrape_wechat_groups (POST /wechatweb/chatroom/getChatroomList) and
scrape_wechat_talkers (POST /wechatweb/wxFriends/talkerList), both reusing
the page-number paginator. Groups parse the semicolon-joined memberIds into
a member_ids_list so upsert_chat_group can mirror GroupMember rows
(composite PK channel/room_id/member_id) idempotently alongside the
ChatGroup row. Talkers are a sync-planning input (member_count>0 marks a
group conversation); no dedicated table.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 13:54:16 +08:00
parent 2b79e7a232
commit 2a74d3888d
7 changed files with 1163 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
yunke-server/app/scraper/wechat.py
+273 -2
View File
@@ -15,6 +15,15 @@ Endpoints covered today:
yields normalized :class:`app.db.models.WechatAccount` dicts (plus the
embedded ``userId``/``userName`` staff pair). Page-number pagination
(Task 2.1).
* :func:`scrape_wechat_groups` — wraps ``POST /wechatweb/chatroom/getChatroomList``
and yields normalized :class:`app.db.models.ChatGroup` dicts, parsing the
semicolon-joined ``memberIds`` into a ``member_ids_list`` so the
persistence layer can mirror :class:`app.db.models.GroupMember` rows
(M2). Page-number pagination.
* :func:`scrape_wechat_talkers` — wraps ``POST /wechatweb/wxFriends/talkerList``
and yields the per-staff-wechat conversation list (M2). Used by the sync
orchestrator to decide which conversations have recent traffic to pull.
Page-number pagination; the owning ``wechatId`` is sent in every request.
All yield dicts (not ORM). Persistence is the caller's job (the
:mod:`app.storage.repos` module bridges these dicts to the ORM tables). Each
@@ -22,8 +31,10 @@ dict carries:
* snake_case keys matching the target ORM columns (see :func:`_normalize`
for friends, :func:`_normalize_message` for messages,
:func:`_normalize_department` for departments, and
:func:`_normalize_wxuser_account` for accounts),
:func:`_normalize_department` for departments,
:func:`_normalize_wxuser_account` for accounts,
:func:`_normalize_chat_group` for groups, and
:func:`_normalize_talker` for talkers),
* ``channel == "wechat"`` (only on communication-channel entities; the
organization entities ``Department`` / ``WechatAccount`` carry no channel),
* ``raw_json`` — the original upstream record, verbatim, so nothing captured
@@ -52,13 +63,17 @@ __all__ = [
"scrape_departments",
"scrape_wechat_accounts",
"scrape_wechat_friends",
"scrape_wechat_groups",
"scrape_wechat_messages",
"scrape_wechat_talkers",
]
FRIEND_LIST_PATH = "/wechatweb/wxFriends/friend_list/es"
MSG_CHATS_PATH = "/wechatweb/msg/chats"
DEPT_TREE_PATH = "/usercenter/dept/tree"
WXUSER_LIST_PATH = "/wechatweb/wxUser/list"
CHATROOM_LIST_PATH = "/wechatweb/chatroom/getChatroomList"
TALKER_LIST_PATH = "/wechatweb/wxFriends/talkerList"
# Upstream sends form-encoded bodies for ``msg/chats`` (NOT JSON), so we have
# to override the client's default JSON content-type on every request.
@@ -700,3 +715,259 @@ async def scrape_wechat_accounts(
build_body=_build_wxuser_body,
):
yield _normalize_wxuser_account(record)
# ===========================================================================
# Chat-group scraper (M2) — POST /wechatweb/chatroom/getChatroomList
# ===========================================================================
def _parse_chatroom_total(resp: Any) -> int | None:
"""Read the declared row count from the ``getChatroomList`` envelope.
Shape: ``data.total`` (an int). Returns ``None`` if the path is absent so
the paginator falls back to the empty-page stop signal.
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return None
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return None
total = data.get("total")
return int(total) if isinstance(total, (int, float)) and not isinstance(
total, bool
) else None
def _parse_chatroom_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the chatroom list from the ``getChatroomList`` envelope.
Shape: ``data.data`` (a list of records). Missing / non-list values
collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
rows = data.get("data")
return rows if isinstance(rows, list) else []
def _build_chatroom_body(page_num: int, page_size: int) -> dict[str, Any]:
"""Build the ``getChatroomList`` request body for a given page.
The live capture carries no filter parameters — just the page slice.
"""
return {"pageNum": page_num, "pageSize": page_size}
def _parse_member_ids(member_ids: Any) -> list[str]:
"""Split a semicolon-joined member id list, dropping empty segments.
Upstream ``memberIds`` looks like ``"wxid_a;wxid_b;csf7169708"`` and may
carry a trailing ``";"`` (yielding an empty segment). Empties are dropped
so each entry is a real wxid; a missing/None/non-string value yields ``[]``.
"""
if not isinstance(member_ids, str):
return []
return [seg for seg in member_ids.split(";") if seg]
def _normalize_chat_group(record: dict[str, Any]) -> dict[str, Any]:
"""Map one upstream ``getChatroomList`` record to the ``ChatGroup`` shape.
Adds ``channel="wechat"`` and preserves the original record under
``raw_json``. The raw ``memberIds`` string is carried verbatim as
``member_ids`` (matching the ``ChatGroup.member_ids`` Text column) AND
parsed into ``member_ids_list`` (a plain list of wxids with empties
dropped) so :func:`app.storage.repos.upsert_chat_group` can mirror one
:class:`app.db.models.GroupMember` row per entry without re-parsing.
``None``-safe: missing fields become ``None`` rather than raising, so a
partial payload does not break the whole scrape.
"""
member_ids = record.get("memberIds")
return {
"room_id": record.get("chatRoomId"),
"channel": "wechat",
"room_name": record.get("chatRoomName"),
"owner_account_id": record.get("wechatId"),
"owner_account_nickname": record.get("wechatNickName"),
"member_count": record.get("memberCount"),
"member_ids": member_ids,
"member_ids_list": _parse_member_ids(member_ids),
"head_image_url": record.get("headImageUrl"),
"owner_user_id": record.get("userId"),
"owner_user_phone": record.get("userPhone"),
"message_count": record.get("messageCount"),
"own_message_count": record.get("ownMessageCount"),
"loss_members": record.get("lossMembers"),
"add_members": record.get("addMembers"),
"last_chat_time": record.get("lastChatTime"),
"update_time": record.get("updateTime"),
"version": record.get("version"),
"raw_json": record,
}
async def scrape_wechat_groups(
client: YunkeClient,
*,
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat chat-group records from ``getChatroomList``.
Walks ``POST /wechatweb/chatroom/getChatroomList`` page by page until the
upstream reports no more rows (empty ``data.data`` array — the primary
stop signal), mapping each record to the ``ChatGroup`` column shape plus
a verbatim ``raw_json`` copy.
The semicolon-joined ``memberIds`` field is carried both raw
(``member_ids``) and parsed (``member_ids_list``, empties dropped) so the
persistence layer can mirror :class:`app.db.models.GroupMember` rows
idempotently.
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
page_size:
Page size to request. Defaults to 50 (a sane middle ground between
request count and per-response payload size for this endpoint).
"""
async for record in paginate_by_page(
client,
CHATROOM_LIST_PATH,
page_size=page_size,
parse_total=_parse_chatroom_total,
parse_rows=_parse_chatroom_rows,
build_body=_build_chatroom_body,
):
yield _normalize_chat_group(record)
# ===========================================================================
# Talker (conversation) list scraper (M2) — POST /wechatweb/wxFriends/talkerList
# ===========================================================================
def _parse_talker_total(resp: Any) -> int | None:
"""Read the declared row count from the ``talkerList`` envelope.
The talker endpoint signals "more pages" via ``data.hasMore`` (a boolean)
rather than a declared ``total``. There is no honest row count to read,
so this always returns ``None`` — the paginator then relies solely on the
empty-page stop signal (a final page whose ``data.page`` is empty).
"""
return None
def _parse_talker_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the talker list from the ``talkerList`` envelope.
Shape: ``data.page`` (a list of records). Missing / non-list values
collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
rows = data.get("page")
return rows if isinstance(rows, list) else []
def _build_talker_body(
page_num: int, page_size: int, *, wechat_id: str
) -> dict[str, Any]:
"""Build the ``talkerList`` request body for a given page.
The endpoint is per-staff-wechat: the owning ``wechatId`` MUST be sent in
every request so the upstream returns that account's conversation list.
"""
return {
"wechatId": wechat_id,
"pageNum": page_num,
"pageSize": page_size,
}
def _normalize_talker(record: dict[str, Any], wechat_id: str) -> dict[str, Any]:
"""Map one upstream ``talkerList`` record to a talker dict.
The talker list is primarily a sync-planning input: it tells the
orchestrator which conversations (direct chats AND group chats) have
recent traffic. ``member_count > 0`` marks a group conversation (the
``friend_wechat_id`` is then a ``@chatroom`` id); ``member_count == 0``
marks a 1:1 friend chat. The owning staff wechat is stamped on every
record as ``owner_account_id``. ``raw_json`` preserves the upstream
record verbatim. ``None``-safe on every field.
"""
return {
"friend_wechat_id": record.get("friendWechatId"),
"owner_account_id": wechat_id,
"nick_name": record.get("nickName"),
"remark_name": record.get("remarkName"),
"display_name": record.get("displayName"),
"message_count": record.get("messageCount"),
"last_chat_time": record.get("lastChatTime"),
"member_count": record.get("memberCount"),
"head_url": record.get("headUrl"),
"raw_json": record,
}
async def scrape_wechat_talkers(
client: YunkeClient,
wechat_id: str,
*,
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield the per-staff-wechat conversation list from ``talkerList``.
Wraps ``POST /wechatweb/wxFriends/talkerList``. The endpoint is scoped to
one staff wechat account (``wechat_id`` → the ``wechatId`` request
field); each response carries a slice of that account's recent
conversations under ``data.page`` with a boolean ``data.hasMore``.
The response envelope carries no declared ``total`` (only ``hasMore``),
so this scraper relies solely on the empty-page stop signal: it advances
``pageNum`` until a page comes back with an empty ``data.page`` array.
A well-behaved upstream flips ``hasMore`` to false on the last real page
and then serves an empty page; the paginator stops on whichever signal
fires first.
Talkers are NOT persisted to a dedicated table — the sync orchestrator
consumes this stream to decide which conversations have messages to pull
(``member_count > 0`` ⇒ group chat whose ``friend_wechat_id`` is a
``@chatroom`` id).
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
wechat_id:
Staff WeChat id (``wechatId`` request field) — the owning account
whose conversation list is being scraped.
page_size:
Page size to request. Defaults to 50.
"""
async for record in paginate_by_page(
client,
TALKER_LIST_PATH,
page_size=page_size,
parse_total=_parse_talker_total,
parse_rows=_parse_talker_rows,
build_body=_build_talker_body,
wechat_id=wechat_id,
):
yield _normalize_talker(record, wechat_id)