xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: department + wechat account/staff scrapers

Browse Source 
M2 organization scrapers for yunke-server:

- scrape_departments: POST /usercenter/dept/tree. Single request (no
  pagination). Defensively unwraps the envelope (data is node list OR an
  object wrapping nodes under children), flattens BOTH nested and flat
  shapes, and reconstructs the materialized path by walking the pId chain
  (ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.

- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
  pagination via the existing paginate_by_page helper. Envelope differs
  from friend_list (no basicResult wrapper): data.data/data.total. Maps
  wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
  sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
  WechatAccount columns; carries embedded staff pair for the repo layer.

Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
  coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
  upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
  forward phone/sale_phone/is_on_job when present.

Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).

Suite: 97 -> 111 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 13:46:23 +08:00
parent eebff9f786
commit 2b79e7a232
7 changed files with 1267 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
yunke-server/app/scraper/wechat.py
+299 -7
View File
@@ -1,19 +1,31 @@
"""WeChat scrapers.
"""WeChat + organization scrapers.
Two endpoints today:
Endpoints covered today:
* :func:`scrape_wechat_friends` — wraps ``POST /wechatweb/wxFriends/friend_list/es``
and yields normalized dicts ready to be upserted into
:class:`app.db.models.Customer`.
* :func:`scrape_wechat_messages` — wraps ``POST /wechatweb/msg/chats`` and
yields normalized chat-message dicts (see Task 1.6).
* :func:`scrape_departments` — wraps ``POST /usercenter/dept/tree`` and
yields normalized :class:`app.db.models.Department` dicts. Single request,
no pagination; the tree (flat or nested) is flattened and the materialized
``path`` is reconstructed from parent links (Task 2.0).
* :func:`scrape_wechat_accounts` — wraps ``POST /wechatweb/wxUser/list`` and
yields normalized :class:`app.db.models.WechatAccount` dicts (plus the
embedded ``userId``/``userName`` staff pair). Page-number pagination
(Task 2.1).
Both yield dicts (not ORM). Persistence is a LATER task — Task 1.7. Each dict
carries:
All yield dicts (not ORM). Persistence is the caller's job (the
:mod:`app.storage.repos` module bridges these dicts to the ORM tables). Each
dict carries:
* snake_case keys matching the target ORM columns (see :func:`_normalize`
for friends and :func:`_normalize_message` for messages),
* ``channel == "wechat"``,
for friends, :func:`_normalize_message` for messages,
:func:`_normalize_department` for departments, and
:func:`_normalize_wxuser_account` for accounts),
* ``channel == "wechat"`` (only on communication-channel entities; the
organization entities ``Department`` / ``WechatAccount`` carry no channel),
* ``raw_json`` — the original upstream record, verbatim, so nothing captured
is lost even when a column has no target field.
@@ -36,10 +48,17 @@ from app.scraper.base import paginate_by_cursor, paginate_by_page
from app.scraper.parsers import normalize_wechat_type
from app.session.yunke_client import YunkeClient
__all__ = ["scrape_wechat_friends", "scrape_wechat_messages"]
__all__ = [
"scrape_departments",
"scrape_wechat_accounts",
"scrape_wechat_friends",
"scrape_wechat_messages",
]
FRIEND_LIST_PATH = "/wechatweb/wxFriends/friend_list/es"
MSG_CHATS_PATH = "/wechatweb/msg/chats"
DEPT_TREE_PATH = "/usercenter/dept/tree"
WXUSER_LIST_PATH = "/wechatweb/wxUser/list"
# Upstream sends form-encoded bodies for ``msg/chats`` (NOT JSON), so we have
# to override the client's default JSON content-type on every request.
@@ -408,3 +427,276 @@ async def scrape_wechat_messages(
if _is_divider(record):
continue
yield _normalize_message(record)
# ===========================================================================
# Department tree scraper (Task 2.0) — POST /usercenter/dept/tree
# ===========================================================================
def _extract_dept_node_list(resp: Any) -> list[dict[str, Any]]:
"""Extract the raw node list from a ``dept/tree`` response, defensively.
The envelope's ``data`` may be:
* a list of nodes (the common case), OR
* an object wrapping the node list under ``children`` (or ``list`` /
``tree`` / ``data`` — be tolerant of wrapper-key variance).
Returns ``[]`` on any malformed shape so the scraper yields nothing rather
than raising: a broken upstream should not abort the whole sync. Nodes are
returned AS-IS (possibly still nested via ``children``); flattening is the
next step (:func:`_flatten_dept_tree`).
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if isinstance(data, list):
return data
if isinstance(data, dict):
# Walk the plausible wrapper keys in priority order. ``children`` is
# the documented yunke tree convention; the others are defensive.
for key in ("children", "list", "tree", "data"):
val = data.get(key)
if isinstance(val, list):
return val
return []
def _flatten_dept_tree(nodes: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Flatten a (possibly nested) dept-tree node list depth-first.
Each node MAY carry a ``children`` array (the nested shape); when absent,
the tree is already flat and parents are carried via ``pId``. Both shapes
flatten to the same depth-first list.
Each emitted node is a shallow copy of the source node with the
``children`` key stripped, so the ``raw_json`` we store per-dept does not
carry a recursive subtree (only that node's own attributes). Non-dict
entries are dropped defensively.
"""
flat: list[dict[str, Any]] = []
# Iterative DFS via a stack. Nodes are pushed in reverse so the first
# child is popped first (preserves source order in the output).
stack: list[dict[str, Any]] = [
n for n in reversed(nodes) if isinstance(n, dict)
]
while stack:
node = stack.pop()
children = node.get("children")
# Shallow copy without ``children`` so raw_json stored per-dept is clean.
flat_node = {k: v for k, v in node.items() if k != "children"}
flat.append(flat_node)
if isinstance(children, list) and children:
stack.extend(
reversed([c for c in children if isinstance(c, dict)])
)
return flat
def _build_dept_paths(nodes: list[dict[str, Any]]) -> dict[str, str]:
"""Compute the materialized path (ancestor names joined by ``/``) per node.
Returns ``{id: path}``. The path is built by walking up the ``pId`` chain;
the walk terminates when it reaches a node whose ``pId`` is falsy
(``None`` / ``""`` / ``0`` / ``"0"``) OR whose parent is not in the tree
(handles the common yunke convention where root carries ``pId="0"`` and no
node with ``id="0"`` exists). A visited set caps the walk so a cyclic
``pId`` chain cannot loop forever.
A node whose own ``id`` is missing is skipped (no path entry) — the caller
treats a missing path as empty string.
"""
by_id: dict[str, dict[str, Any]] = {}
for n in nodes:
nid = n.get("id")
# First occurrence wins so duplicate ids cannot shadow the canonical
# node (defensive against an upstream that emits a node twice).
if nid is not None and nid not in by_id:
by_id[nid] = n
paths: dict[str, str] = {}
for nid, node in by_id.items():
parts: list[str] = []
seen: set[str] = set()
cur: dict[str, Any] | None = node
while cur is not None:
cid = cur.get("id")
if cid is None or cid in seen:
break
seen.add(cid)
name = cur.get("name")
if name is not None:
parts.append(name)
pid = cur.get("pId")
# Falsy pId OR unknown parent terminates the walk.
cur = by_id.get(pid) if pid else None
paths[nid] = "/".join(reversed(parts))
return paths
def _normalize_department(
node: dict[str, Any], path: str
) -> dict[str, Any]:
"""Map one dept-tree node to the ``Department`` column shape.
``parent_id`` is carried verbatim from ``pId`` — the scraper does NOT
synthesize ``NULL`` for root nodes; downstream code treats falsy
``parent_id`` (``"0"`` / ``""`` / ``None``) as "no parent" consistently
with the materialized-path builder. ``path`` is the precomputed
slash-joined ancestor-name chain from :func:`_build_dept_paths`.
"""
return {
"dept_id": node.get("id"),
"parent_id": node.get("pId"),
"name": node.get("name"),
"path": path,
"raw_json": node,
}
async def scrape_departments(
client: YunkeClient,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized department records from ``POST /usercenter/dept/tree``.
Single request (no pagination). The endpoint follows yunke's tree
convention: nodes carry ``id`` / ``pId`` / ``name`` and MAY be delivered
either as a flat list (parents via ``pId``) or as a nested tree (each node
optionally carrying ``children``). Both shapes are flattened to one flat
list of records, each carrying the materialized ``path`` (ancestor names
joined with ``/``).
Defensive envelope: ``data`` may be the node list directly OR an object
wrapping the node list under ``children``; both are unwrapped.
The async-generator signature mirrors the other scrapers so callers can
consume every entity stream with the same ``async for`` pattern, even
though this endpoint always resolves in one round-trip.
"""
resp = await client.post(DEPT_TREE_PATH, json={})
nodes = _extract_dept_node_list(resp)
flat = _flatten_dept_tree(nodes)
paths = _build_dept_paths(flat)
for node in flat:
nid = node.get("id")
path = paths.get(nid, "") if nid is not None else ""
yield _normalize_department(node, path)
# ===========================================================================
# WeChat staff-account scraper (Task 2.1) — POST /wechatweb/wxUser/list
# ===========================================================================
def _parse_wxuser_total(resp: Any) -> int | None:
"""Read the declared row count from the ``wxUser/list`` envelope.
Shape: ``data.total`` (an int). Returns ``None`` if the path is absent so
the paginator falls back to the empty-page stop signal.
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return None
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return None
total = data.get("total")
return int(total) if isinstance(total, (int, float)) and not isinstance(
total, bool
) else None
def _parse_wxuser_rows(resp: Any) -> list[dict[str, Any]]:
"""Extract the account list from the ``wxUser/list`` envelope.
Shape: ``data.data`` (a list of records). Missing / non-list values
collapse to an empty list so the paginator sees "empty page".
"""
try:
body = resp.json()
except (AttributeError, ValueError):
return []
data = body.get("data") if isinstance(body, dict) else None
if not isinstance(data, dict):
return []
rows = data.get("data")
return rows if isinstance(rows, list) else []
def _build_wxuser_body(page_num: int, page_size: int) -> dict[str, Any]:
"""Build the ``wxUser/list`` request body for a given page.
Unlike ``friend_list/es``, the ``wxUser/list`` body carries no
staff/dept filter parameters in the live capture — just the page slice.
"""
return {"pageNum": page_num, "pageSize": page_size}
def _normalize_wxuser_account(record: dict[str, Any]) -> dict[str, Any]:
"""Map one upstream ``wxUser/list`` record to the ``WechatAccount`` shape.
Carries the embedded staff pair (``user_id`` <- ``userId``,
``user_name`` <- ``userName``) so the persistence layer can upsert the
owning :class:`app.db.models.Staff` row alongside the account. Adds
``raw_json`` (the verbatim upstream record). ``None``-safe: missing fields
become ``None`` rather than raising, so partial payloads do not break the
scrape.
"""
return {
"wx_id": record.get("wechatId"),
"nick_name": record.get("nickName"),
"alias": record.get("alias"),
"phone": record.get("phone"),
"user_id": record.get("userId"),
"user_name": record.get("userName"),
"contacts_num": record.get("contactsNum"),
"group_num": record.get("groupNum"),
"send_num": record.get("sendNum"),
"receive_num": record.get("receiveNum"),
"is_on_job": record.get("isOnJob"),
"db_size": record.get("dbSize"),
"online_status": record.get("onlineStatus"),
"last_update": record.get("lastUpdate"),
"version": record.get("version"),
"raw_json": record,
}
async def scrape_wechat_accounts(
client: YunkeClient,
*,
page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
"""Yield normalized WeChat staff-account records from ``wxUser/list``.
Walks ``POST /wechatweb/wxUser/list`` page by page until the upstream
reports no more rows (empty ``data.data`` array — the primary stop signal),
mapping each record to the ``WechatAccount`` column shape plus a verbatim
``raw_json`` copy.
The envelope differs from ``friend_list/es``: there is **no**
``basicResult`` wrapper. The pagination fields sit directly under ``data``
(``data.data`` / ``data.total`` / ``data.pageNum`` / ``data.pageSize``).
Parameters
----------
client:
Authenticated :class:`YunkeClient`. Auth headers / 401 re-login are
handled by the client.
page_size:
Page size to request. Defaults to 50 (matches the live capture's
observed page size for this endpoint).
"""
async for record in paginate_by_page(
client,
WXUSER_LIST_PATH,
page_size=page_size,
parse_total=_parse_wxuser_total,
parse_rows=_parse_wxuser_rows,
build_body=_build_wxuser_body,
):
yield _normalize_wxuser_account(record)