feat: department + wechat account/staff scrapers
M2 organization scrapers for yunke-server: - scrape_departments: POST /usercenter/dept/tree. Single request (no pagination). Defensively unwraps the envelope (data is node list OR an object wrapping nodes under children), flattens BOTH nested and flat shapes, and reconstructs the materialized path by walking the pId chain (ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name. - scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number pagination via the existing paginate_by_page helper. Envelope differs from friend_list (no basicResult wrapper): data.data/data.total. Maps wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/ sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to WechatAccount columns; carries embedded staff pair for the repo layer. Repos: - upsert_department: idempotent upsert into Department (PK dept_id). - upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job coerced to bool). - upsert_wechat_account: now ALSO upserts the embedded Staff row via upsert_staff_from_account (best-effort; skips when account has no user_id). - upsert_staff_from_account: refactored to delegate to upsert_staff and forward phone/sale_phone/is_on_job when present. Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data + empty), test_scrape_wechat_accounts (2-page pagination + body advance + empty first page), test_repos extended (dept idempotency/root parent, direct staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl. departed + orphan). Suite: 97 -> 111 passing. ruff check clean. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
@@ -0,0 +1,107 @@
|
||||
"""Fixture payloads for ``POST /usercenter/dept/tree`` (Task 2.0).
|
||||
|
||||
yunke's dept/tree endpoint follows the same convention as the permissions API:
|
||||
nodes carry ``id`` / ``pId`` / ``name``, and the tree MAY be delivered either
|
||||
as a flat list of nodes or as a nested tree (each node optionally carrying a
|
||||
``children`` array). The scraper must flatten both shapes into one flat list
|
||||
of nodes and reconstruct ``parent_id`` from ``pId``.
|
||||
|
||||
This cassette provides TWO realistic capture-shaped payloads so the scraper
|
||||
tests can prove both shapes flatten to the SAME set of nodes:
|
||||
|
||||
* ``NESTED_TREE_PAYLOAD`` — a root with two children, one of which has its own
|
||||
sub-child (3 levels deep). Nested via ``children``.
|
||||
* ``FLAT_TREE_PAYLOAD`` — the exact same 4 nodes delivered as a flat list
|
||||
(parent links carried by ``pId``).
|
||||
|
||||
Both decode to the same 4 departments with the same ids / parents / names /
|
||||
paths, so the scraper test can swap the envelope and re-assert.
|
||||
|
||||
The 4 nodes:
|
||||
root "Sales" (id="1", pId="0")
|
||||
├── "East" (id="1.1", pId="1")
|
||||
│ └── "TeamA" (id="1.1.1", pId="1.1")
|
||||
└── "North" (id="1.2", pId="1")
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The canonical 4 nodes (single source of truth for both shapes below)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Root has pId="0" — the common yunke convention for "no parent". The scraper
|
||||
# treats any falsy pId (None / "" / "0") OR an unknown pId as "root".
|
||||
ROOT_NODE: dict[str, Any] = {"id": "1", "pId": "0", "name": "Sales"}
|
||||
EAST_NODE: dict[str, Any] = {"id": "1.1", "pId": "1", "name": "East"}
|
||||
TEAMA_NODE: dict[str, Any] = {"id": "1.1.1", "pId": "1.1", "name": "TeamA"}
|
||||
NORTH_NODE: dict[str, Any] = {"id": "1.2", "pId": "1", "name": "North"}
|
||||
|
||||
# All four nodes, in a stable order matching a depth-first traversal of the
|
||||
# nested tree. Tests assert the scraper yields exactly these (order is not
|
||||
# load-bearing — tests compare as sets where appropriate).
|
||||
ALL_NODES: list[dict[str, Any]] = [ROOT_NODE, EAST_NODE, TEAMA_NODE, NORTH_NODE]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Nested envelope — root carries children, recursively
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
NESTED_TREE_NODES: list[dict[str, Any]] = [
|
||||
{
|
||||
**ROOT_NODE,
|
||||
"children": [
|
||||
{
|
||||
**EAST_NODE,
|
||||
"children": [{**TEAMA_NODE}],
|
||||
},
|
||||
{**NORTH_NODE},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Flat envelope — every node at the top level, parents via pId
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
FLAT_TREE_NODES: list[dict[str, Any]] = [
|
||||
{**ROOT_NODE},
|
||||
{**EAST_NODE},
|
||||
{**TEAMA_NODE},
|
||||
{**NORTH_NODE},
|
||||
]
|
||||
|
||||
|
||||
def make_dept_tree_response(
|
||||
nodes: list[dict[str, Any]],
|
||||
*,
|
||||
nested_in_data_object: bool = False,
|
||||
trace_id: str = "trace-dept",
|
||||
) -> dict[str, Any]:
|
||||
"""Build a capture-shaped ``dept/tree`` response envelope.
|
||||
|
||||
The yunke dept/tree response top-level is the standard envelope
|
||||
``{message, code:"10000", success, traceId, data}``. The shape of ``data``
|
||||
itself varies between captures — be defensive:
|
||||
|
||||
* ``data`` is the node list directly (the common case), OR
|
||||
* ``data`` is an object wrapping the node list under ``children`` /
|
||||
``list`` / etc.
|
||||
|
||||
The ``nested_in_data_object`` flag selects the second variant for the
|
||||
"wrapped data object" path so a test can prove the scraper unwraps it.
|
||||
The default (``False``) emits the most common flat-data array shape.
|
||||
"""
|
||||
data: Any = list(nodes)
|
||||
if nested_in_data_object:
|
||||
data = {"children": list(nodes)}
|
||||
return {
|
||||
"message": "调用成功",
|
||||
"code": "10000",
|
||||
"success": True,
|
||||
"traceId": trace_id,
|
||||
"data": data,
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Fixture payloads for ``POST /wechatweb/wxUser/list`` (Task 2.1).
|
||||
|
||||
These records mirror the real field names captured from the live CRM (see the
|
||||
data dictionary §2: ``wechatId, nickName, alias, phone, userId,
|
||||
userName(员工真名), contactsNum, groupNum, sendNum/receiveNum, isOnJob,
|
||||
dbSize, 在线状态, lastUpdate``; total=55).
|
||||
|
||||
The envelope differs from ``wxFriends/friend_list/es``: there is **no**
|
||||
``basicResult`` wrapper. The pagination fields sit directly under ``data``::
|
||||
|
||||
{"code":"10000","data":{"data":[...], "total":.., "pageNum":.., "pageSize":..}}
|
||||
|
||||
The cassette ships two pages so a test can drive the page-number paginator
|
||||
across the empty-page stop signal:
|
||||
|
||||
* page 1 -> 2 of the 4 fixture records, ``total=4``
|
||||
* page 2 -> the other 2 records, ``total=4``
|
||||
* (page 3 -> empty ``data`` array, served inline by the test)
|
||||
|
||||
Records carry every field with a target ``WechatAccount`` column so the test
|
||||
can assert a full mapping end-to-end, plus the embedded ``userId``+``userName``
|
||||
staff pair (the owning employee) so the embedded-staff repo upsert can be
|
||||
exercised against capture-shaped input.
|
||||
"""
|
||||
|
||||
# The module name uses camelCase deliberately to mirror the upstream endpoint
|
||||
# name (``wxUser``); the Task 2.1 spec mandates this exact filename.
|
||||
# ruff: noqa: N999
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Capture-shaped WeChat staff-account records (4 total across 2 pages).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
WXUSER_RECORDS: list[dict[str, Any]] = [
|
||||
{
|
||||
"wechatId": "wxid_staff_alice01",
|
||||
"nickName": "Alice WeChat",
|
||||
"alias": "alice_wx",
|
||||
"phone": "13800000001",
|
||||
"userId": "staff_user_001",
|
||||
"userName": "Alice Wang",
|
||||
"contactsNum": 312,
|
||||
"groupNum": 18,
|
||||
"sendNum": 1204,
|
||||
"receiveNum": 2387,
|
||||
"isOnJob": 1,
|
||||
"dbSize": 482103,
|
||||
"onlineStatus": "online",
|
||||
"headImageUrl": "https://wx.qlogo.cn/avatar_alice.png",
|
||||
"lastUpdate": 1700000000123,
|
||||
"version": "v1.20260731.alice",
|
||||
},
|
||||
{
|
||||
"wechatId": "wxid_staff_bob02",
|
||||
"nickName": "Bob WeChat",
|
||||
"alias": "bob_wx",
|
||||
"phone": "13800000002",
|
||||
"userId": "staff_user_002",
|
||||
"userName": "Bob Chen",
|
||||
"contactsNum": 87,
|
||||
"groupNum": 4,
|
||||
"sendNum": 211,
|
||||
"receiveNum": 402,
|
||||
"isOnJob": 1,
|
||||
"dbSize": 91205,
|
||||
"onlineStatus": "offline",
|
||||
"headImageUrl": "https://wx.qlogo.cn/avatar_bob.png",
|
||||
"lastUpdate": 1700000000456,
|
||||
"version": "v1.20260731.bob",
|
||||
},
|
||||
{
|
||||
"wechatId": "wxid_staff_carol03",
|
||||
"nickName": "Carol We Chat",
|
||||
"alias": "carol_wx",
|
||||
"phone": "13800000003",
|
||||
"userId": "staff_user_003",
|
||||
"userName": "Carol Liu",
|
||||
"contactsNum": 1502,
|
||||
"groupNum": 72,
|
||||
"sendNum": 9034,
|
||||
"receiveNum": 11245,
|
||||
"isOnJob": 0, # departed staff — exercises the falsy isOnJob path
|
||||
"dbSize": 2_100_445,
|
||||
"onlineStatus": "offline",
|
||||
"headImageUrl": "https://wx.qlogo.cn/avatar_carol.png",
|
||||
"lastUpdate": 1700000000789,
|
||||
"version": "v1.20260731.carol",
|
||||
},
|
||||
{
|
||||
# Account with NO embedded staff (userId missing) — the scraper must
|
||||
# still yield it; the embedded-staff repo upsert must skip it cleanly.
|
||||
"wechatId": "wxid_staff_orphan04",
|
||||
"nickName": "Orphan WeChat",
|
||||
"alias": "orphan_wx",
|
||||
"phone": "13800000004",
|
||||
"userId": None,
|
||||
"userName": None,
|
||||
"contactsNum": 5,
|
||||
"groupNum": 0,
|
||||
"sendNum": 1,
|
||||
"receiveNum": 2,
|
||||
"isOnJob": 0,
|
||||
"dbSize": 1024,
|
||||
"onlineStatus": "offline",
|
||||
"headImageUrl": "https://wx.qlogo.cn/avatar_orphan.png",
|
||||
"lastUpdate": 1700000001000,
|
||||
"version": "v1.20260731.orphan",
|
||||
},
|
||||
]
|
||||
|
||||
# Default page size for the fixture (and the scraper). total=4 means the
|
||||
# fixture fills exactly 2 pages of size 2 plus a terminating empty page.
|
||||
TOTAL = len(WXUSER_RECORDS)
|
||||
DEFAULT_PAGE_SIZE = 2
|
||||
|
||||
|
||||
def make_wxuser_list_response(
|
||||
records: list[dict[str, Any]],
|
||||
*,
|
||||
total: int,
|
||||
page_num: int = 1,
|
||||
page_size: int = DEFAULT_PAGE_SIZE,
|
||||
trace_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Build a capture-shaped ``wxUser/list`` response envelope.
|
||||
|
||||
Shape (no ``basicResult`` wrapper)::
|
||||
|
||||
{"message":"调用成功","code":"10000","success":true,"traceId":"...",
|
||||
"data":{"data":[...], "total":.., "pageNum":.., "pageSize":..}}
|
||||
"""
|
||||
return {
|
||||
"message": "调用成功",
|
||||
"code": "10000",
|
||||
"success": True,
|
||||
"traceId": trace_id or f"trace-{page_num}",
|
||||
"data": {
|
||||
"data": list(records),
|
||||
"total": total,
|
||||
"pageNum": page_num,
|
||||
"pageSize": page_size,
|
||||
},
|
||||
}
|
||||
@@ -22,13 +22,15 @@ import sqlalchemy as sa
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from app.db.base import Base
|
||||
from app.db.models import Customer, Message, Staff, WechatAccount
|
||||
from app.db.models import Customer, Department, Message, Staff, WechatAccount
|
||||
from app.storage.repos import (
|
||||
_coerce_bool,
|
||||
_coerce_int,
|
||||
upsert_customer,
|
||||
upsert_customers,
|
||||
upsert_department,
|
||||
upsert_messages,
|
||||
upsert_staff,
|
||||
upsert_staff_from_account,
|
||||
upsert_wechat_account,
|
||||
)
|
||||
@@ -426,3 +428,276 @@ async def test_upsert_staff_from_account_skips_when_no_user_id():
|
||||
assert rows == []
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Department upsert (Task 2.0)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_upsert_department_is_idempotent_and_updates_path():
|
||||
"""Same dept_id twice with different path -> 1 row, path refreshed."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
rec1 = {
|
||||
"dept_id": "1.1",
|
||||
"parent_id": "1",
|
||||
"name": "East",
|
||||
"path": "Sales/East",
|
||||
}
|
||||
rec2 = {
|
||||
"dept_id": "1.1",
|
||||
"parent_id": "1",
|
||||
"name": "East",
|
||||
# Re-scrape reconstructs the same path; verify it is overwritten
|
||||
# rather than producing a duplicate row.
|
||||
"path": "Sales/East",
|
||||
"raw_json": {"id": "1.1", "pId": "1", "name": "East"},
|
||||
}
|
||||
|
||||
await upsert_department(session, rec1)
|
||||
await session.commit()
|
||||
await upsert_department(session, rec2)
|
||||
await session.commit()
|
||||
|
||||
rows = (
|
||||
await session.execute(
|
||||
sa.select(Department)
|
||||
.where(Department.dept_id == "1.1")
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(rows) == 1
|
||||
assert rows[0].dept_id == "1.1"
|
||||
assert rows[0].parent_id == "1"
|
||||
assert rows[0].name == "East"
|
||||
assert rows[0].path == "Sales/East"
|
||||
# raw_json from the second upsert wins (last write).
|
||||
assert rows[0].raw_json == rec2["raw_json"]
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def test_upsert_department_preserves_none_parent_for_root():
|
||||
"""A root dept (parent_id None) is stored as-is; no spurious coercion."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
rec = {
|
||||
"dept_id": "1",
|
||||
"parent_id": None,
|
||||
"name": "Sales",
|
||||
"path": "Sales",
|
||||
}
|
||||
await upsert_department(session, rec)
|
||||
await session.commit()
|
||||
|
||||
fetched = (
|
||||
await session.execute(
|
||||
sa.select(Department).where(Department.dept_id == "1")
|
||||
)
|
||||
).scalar_one()
|
||||
assert fetched.parent_id is None
|
||||
assert fetched.name == "Sales"
|
||||
assert fetched.path == "Sales"
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# upsert_staff — direct staff-shaped dict upsert (Task 2.1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_upsert_staff_is_idempotent_and_updates_name():
|
||||
"""Same user_id twice with different name -> 1 row, name refreshed."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
rec1 = {"user_id": "u_new", "name": "Old Name"}
|
||||
rec2 = {"user_id": "u_new", "name": "New Name"}
|
||||
|
||||
await upsert_staff(session, rec1)
|
||||
await session.commit()
|
||||
await upsert_staff(session, rec2)
|
||||
await session.commit()
|
||||
|
||||
rows = (
|
||||
await session.execute(
|
||||
sa.select(Staff)
|
||||
.where(Staff.user_id == "u_new")
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(rows) == 1
|
||||
assert rows[0].name == "New Name"
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def test_upsert_staff_coerces_is_on_job():
|
||||
"""is_on_job '1' -> True; '0' -> False; None preserved."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
await upsert_staff(
|
||||
session, {"user_id": "u_b1", "is_on_job": "1"}
|
||||
)
|
||||
await upsert_staff(
|
||||
session, {"user_id": "u_b0", "is_on_job": "0"}
|
||||
)
|
||||
await upsert_staff(
|
||||
session, {"user_id": "u_bn", "is_on_job": None}
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
on = (
|
||||
await session.execute(
|
||||
sa.select(Staff).where(Staff.user_id == "u_b1")
|
||||
)
|
||||
).scalar_one()
|
||||
off = (
|
||||
await session.execute(
|
||||
sa.select(Staff).where(Staff.user_id == "u_b0")
|
||||
)
|
||||
).scalar_one()
|
||||
none_ = (
|
||||
await session.execute(
|
||||
sa.select(Staff).where(Staff.user_id == "u_bn")
|
||||
)
|
||||
).scalar_one()
|
||||
assert on.is_on_job is True
|
||||
assert off.is_on_job is False
|
||||
assert none_.is_on_job is None
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# upsert_wechat_account now ALSO upserts the embedded Staff (Task 2.1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_upsert_wechat_account_also_upserts_embedded_staff():
|
||||
"""Account upsert creates BOTH the WechatAccount row and the Staff row."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
rec = {
|
||||
"wx_id": "wxid_dup",
|
||||
"nick_name": "Dup WeChat",
|
||||
"user_id": "u_dup",
|
||||
"user_name": "Dup Real Name",
|
||||
"is_on_job": 1,
|
||||
"phone": "13800000099",
|
||||
}
|
||||
await upsert_wechat_account(session, rec)
|
||||
await session.commit()
|
||||
|
||||
acct = (
|
||||
await session.execute(
|
||||
sa.select(WechatAccount).where(WechatAccount.wx_id == "wxid_dup")
|
||||
)
|
||||
).scalar_one()
|
||||
assert acct.user_id == "u_dup"
|
||||
assert acct.user_name == "Dup Real Name"
|
||||
|
||||
staff = (
|
||||
await session.execute(
|
||||
sa.select(Staff).where(Staff.user_id == "u_dup")
|
||||
)
|
||||
).scalar_one()
|
||||
assert staff.name == "Dup Real Name"
|
||||
assert staff.is_on_job is True
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def test_upsert_wechat_account_reupsert_updates_embedded_staff():
|
||||
"""Re-upserting the account refreshes BOTH account + embedded staff rows."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
rec1 = {
|
||||
"wx_id": "wxid_re",
|
||||
"nick_name": "Old Nick",
|
||||
"user_id": "u_re",
|
||||
"user_name": "Old Name",
|
||||
"is_on_job": 1,
|
||||
}
|
||||
rec2 = {
|
||||
"wx_id": "wxid_re",
|
||||
"nick_name": "New Nick",
|
||||
"user_id": "u_re",
|
||||
"user_name": "New Name",
|
||||
"is_on_job": 0, # staff departed since last scrape
|
||||
}
|
||||
await upsert_wechat_account(session, rec1)
|
||||
await session.commit()
|
||||
await upsert_wechat_account(session, rec2)
|
||||
await session.commit()
|
||||
|
||||
# One account row, refreshed.
|
||||
acct_rows = (
|
||||
await session.execute(
|
||||
sa.select(WechatAccount)
|
||||
.where(WechatAccount.wx_id == "wxid_re")
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(acct_rows) == 1
|
||||
assert acct_rows[0].nick_name == "New Nick"
|
||||
|
||||
# One staff row, refreshed (not duplicated).
|
||||
staff_rows = (
|
||||
await session.execute(
|
||||
sa.select(Staff)
|
||||
.where(Staff.user_id == "u_re")
|
||||
.execution_options(populate_existing=True)
|
||||
)
|
||||
).scalars().all()
|
||||
assert len(staff_rows) == 1
|
||||
assert staff_rows[0].name == "New Name"
|
||||
assert staff_rows[0].is_on_job is False
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def test_upsert_wechat_account_without_user_id_skips_staff():
|
||||
"""Account with no user_id is written; embedded-staff upsert is skipped."""
|
||||
engine = await _make_engine()
|
||||
try:
|
||||
SessionLocal = async_sessionmaker(engine, expire_on_commit=False)
|
||||
async with SessionLocal() as session:
|
||||
rec = {
|
||||
"wx_id": "wxid_no_staff",
|
||||
"nick_name": "Orphan WeChat",
|
||||
# no user_id / user_name
|
||||
}
|
||||
await upsert_wechat_account(session, rec)
|
||||
await session.commit()
|
||||
|
||||
# Account row written.
|
||||
acct = (
|
||||
await session.execute(
|
||||
sa.select(WechatAccount).where(
|
||||
WechatAccount.wx_id == "wxid_no_staff"
|
||||
)
|
||||
)
|
||||
).scalar_one()
|
||||
assert acct.nick_name == "Orphan WeChat"
|
||||
|
||||
# No staff rows at all.
|
||||
staff_rows = (
|
||||
await session.execute(sa.select(Staff))
|
||||
).scalars().all()
|
||||
assert staff_rows == []
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
"""Tests for the department-tree scraper (Task 2.0).
|
||||
|
||||
Drives the real flatten + normalize logic through ``respx`` by wiring a real
|
||||
``YunkeClient`` (primed with a fake session so the login flow is skipped)
|
||||
against a mocked transport.
|
||||
|
||||
The dept/tree endpoint is a SINGLE request (no pagination). The scraper must:
|
||||
|
||||
* decode the envelope defensively (``data`` is the node list, OR an object
|
||||
wrapping the node list under ``children``),
|
||||
* flatten BOTH a nested tree (each node carrying ``children``) AND a flat list
|
||||
of nodes (parents via ``pId``) into the same flat set of records,
|
||||
* map ``id``/``pId``/``name`` to ``dept_id``/``parent_id``/``name``,
|
||||
* build a materialized ``path`` by walking up the parent chain (names joined
|
||||
with ``/``; root nodes have ``path == name``),
|
||||
* preserve the verbatim node under ``raw_json``.
|
||||
|
||||
The two-shape contract (nested vs flat) is asserted by serving the SAME four
|
||||
nodes in both shapes and asserting the scraper yields the SAME result set in
|
||||
both cases.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import respx
|
||||
|
||||
from app.scraper.wechat import scrape_departments
|
||||
from app.session.yunke_client import YunkeClient
|
||||
from tests.cassettes.dept_tree_fixture import (
|
||||
ALL_NODES,
|
||||
EAST_NODE,
|
||||
FLAT_TREE_NODES,
|
||||
NESTED_TREE_NODES,
|
||||
NORTH_NODE,
|
||||
ROOT_NODE,
|
||||
TEAMA_NODE,
|
||||
make_dept_tree_response,
|
||||
)
|
||||
|
||||
BASE = "https://crm2.yunkecn.com"
|
||||
DEPT_TREE_URL = f"{BASE}/usercenter/dept/tree"
|
||||
|
||||
USERID = "a" * 32
|
||||
COMPANY = "8m0atd"
|
||||
|
||||
|
||||
def _primed_client() -> YunkeClient:
|
||||
"""A YunkeClient already in a logged-in state (login flow skipped)."""
|
||||
return YunkeClient(base_url=BASE, userid=USERID, company=COMPANY)
|
||||
|
||||
|
||||
def _expected_path(node: dict) -> str:
|
||||
"""Compute the expected materialized path for a fixture node.
|
||||
|
||||
Root (Sales) -> "Sales"; East -> "Sales/East"; TeamA -> "Sales/East/TeamA";
|
||||
North -> "Sales/North".
|
||||
"""
|
||||
by_id = {n["id"]: n for n in ALL_NODES}
|
||||
parts: list[str] = []
|
||||
cur: dict | None = node
|
||||
while cur is not None:
|
||||
parts.append(cur["name"])
|
||||
pid = cur.get("pId")
|
||||
# Falsy / unknown pId terminates the walk (root reached).
|
||||
cur = by_id.get(pid) if pid else None
|
||||
return "/".join(reversed(parts))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# End-to-end: nested tree flattens correctly with paths reconstructed
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_departments_flattens_nested_tree_with_paths():
|
||||
"""Nested tree -> flat list of 4 depts with correct parent_id + path."""
|
||||
payload = make_dept_tree_response(NESTED_TREE_NODES)
|
||||
route = respx.post(DEPT_TREE_URL).mock(
|
||||
return_value=httpx.Response(200, json=payload)
|
||||
)
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [r async for r in scrape_departments(client)]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
# Single request (no pagination).
|
||||
assert route.call_count == 1
|
||||
|
||||
# All four nodes yielded (order-independent).
|
||||
assert len(results) == 4
|
||||
by_dept_id = {r["dept_id"]: r for r in results}
|
||||
assert set(by_dept_id) == {n["id"] for n in ALL_NODES}
|
||||
|
||||
# Field mapping + parent reconstruction for each node.
|
||||
for node in ALL_NODES:
|
||||
rec = by_dept_id[node["id"]]
|
||||
assert rec["dept_id"] == node["id"]
|
||||
assert rec["name"] == node["name"]
|
||||
# Root node's parent_id is preserved verbatim (pId="0") — the scraper
|
||||
# does not synthesize NULL; it carries pId through. Materialized-path
|
||||
# construction treats falsy/unknown pId as "stop walking".
|
||||
assert rec["parent_id"] == node["pId"]
|
||||
assert rec["path"] == _expected_path(node)
|
||||
# raw_json preserves the original (flattened) node — keys id/pId/name.
|
||||
assert rec["raw_json"]["id"] == node["id"]
|
||||
assert rec["raw_json"]["name"] == node["name"]
|
||||
|
||||
# Spot-check the headline path expectations (defensive against path-helper
|
||||
# bugs masking themselves).
|
||||
assert by_dept_id[ROOT_NODE["id"]]["path"] == "Sales"
|
||||
assert by_dept_id[EAST_NODE["id"]]["path"] == "Sales/East"
|
||||
assert by_dept_id[TEAMA_NODE["id"]]["path"] == "Sales/East/TeamA"
|
||||
assert by_dept_id[NORTH_NODE["id"]]["path"] == "Sales/North"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Flat list of nodes (parents via pId) yields the SAME result as nested
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_departments_flattens_flat_list_to_same_result():
|
||||
"""Flat node list (no children arrays) flattens to the same 4 records."""
|
||||
payload = make_dept_tree_response(FLAT_TREE_NODES)
|
||||
respx.post(DEPT_TREE_URL).mock(return_value=httpx.Response(200, json=payload))
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [r async for r in scrape_departments(client)]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
assert len(results) == 4
|
||||
by_dept_id = {r["dept_id"]: r for r in results}
|
||||
for node in ALL_NODES:
|
||||
rec = by_dept_id[node["id"]]
|
||||
assert rec["parent_id"] == node["pId"]
|
||||
assert rec["path"] == _expected_path(node)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defensive envelope: data is an OBJECT wrapping nodes under `children`
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_departments_unwraps_data_object_children():
|
||||
"""When ``data`` is an object with ``children``, scraper unwraps it."""
|
||||
payload = make_dept_tree_response(
|
||||
FLAT_TREE_NODES, nested_in_data_object=True
|
||||
)
|
||||
respx.post(DEPT_TREE_URL).mock(return_value=httpx.Response(200, json=payload))
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [r async for r in scrape_departments(client)]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
assert len(results) == 4
|
||||
assert {r["dept_id"] for r in results} == {n["id"] for n in ALL_NODES}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Empty tree -> no yields, single call (no infinite loop on missing data)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_departments_empty_tree_yields_nothing():
|
||||
"""Empty data array -> no yields, exactly one call."""
|
||||
payload = make_dept_tree_response([])
|
||||
route = respx.post(DEPT_TREE_URL).mock(
|
||||
return_value=httpx.Response(200, json=payload)
|
||||
)
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [r async for r in scrape_departments(client)]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
assert results == []
|
||||
assert route.call_count == 1
|
||||
@@ -0,0 +1,192 @@
|
||||
"""Tests for the WeChat staff-account scraper (Task 2.1).
|
||||
|
||||
Drives the real page-number pagination + normalization logic through
|
||||
``respx`` by wiring a real ``YunkeClient`` (primed with a fake session so the
|
||||
login flow is skipped) against a mocked transport.
|
||||
|
||||
The fixture ships 4 records across 2 pages of size 2, plus a terminating
|
||||
empty page (the documented primary stop signal for page-number endpoints):
|
||||
|
||||
* page 1 -> 2 records, ``total=4``
|
||||
* page 2 -> 2 records, ``total=4``
|
||||
* page 3 -> empty ``data`` array (no more rows)
|
||||
|
||||
The scraper must yield exactly the 4 records (normalized to ``WechatAccount``
|
||||
snake_case columns + ``raw_json``), make exactly 3 calls, and stop without a
|
||||
4th. The empty-page stop is the primary signal; ``total=4`` is only a
|
||||
defensive over-fetch ceiling that must NOT trigger early (we yield exactly
|
||||
4 == total, so the strict-``>`` ceiling never fires here).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json as _json
|
||||
|
||||
import httpx
|
||||
import respx
|
||||
|
||||
from app.scraper.wechat import scrape_wechat_accounts
|
||||
from app.session.yunke_client import YunkeClient
|
||||
from tests.cassettes.wxUser_fixture import (
|
||||
DEFAULT_PAGE_SIZE,
|
||||
TOTAL,
|
||||
WXUSER_RECORDS,
|
||||
make_wxuser_list_response,
|
||||
)
|
||||
|
||||
BASE = "https://crm2.yunkecn.com"
|
||||
WXUSER_LIST_URL = f"{BASE}/wechatweb/wxUser/list"
|
||||
|
||||
USERID = "a" * 32
|
||||
COMPANY = "8m0atd"
|
||||
|
||||
|
||||
def _primed_client() -> YunkeClient:
|
||||
"""A YunkeClient already in a logged-in state (login flow skipped)."""
|
||||
return YunkeClient(base_url=BASE, userid=USERID, company=COMPANY)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# End-to-end: pagination + field mapping + raw_json
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_wechat_accounts_yields_records_and_paginates():
|
||||
"""Two full pages + a terminating empty page -> 4 records, 3 calls."""
|
||||
page1 = make_wxuser_list_response(
|
||||
WXUSER_RECORDS[:2], total=TOTAL, page_num=1, page_size=DEFAULT_PAGE_SIZE
|
||||
)
|
||||
page2 = make_wxuser_list_response(
|
||||
WXUSER_RECORDS[2:], total=TOTAL, page_num=2, page_size=DEFAULT_PAGE_SIZE
|
||||
)
|
||||
page3 = make_wxuser_list_response(
|
||||
[], total=TOTAL, page_num=3, page_size=DEFAULT_PAGE_SIZE
|
||||
)
|
||||
|
||||
route = respx.post(WXUSER_LIST_URL).mock(
|
||||
side_effect=[
|
||||
httpx.Response(200, json=page1),
|
||||
httpx.Response(200, json=page2),
|
||||
httpx.Response(200, json=page3),
|
||||
]
|
||||
)
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [
|
||||
r
|
||||
async for r in scrape_wechat_accounts(
|
||||
client, page_size=DEFAULT_PAGE_SIZE
|
||||
)
|
||||
]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
# Pagination: exactly 3 calls (page1, page2, empty-page stop).
|
||||
assert route.call_count == 3
|
||||
|
||||
# Exactly the 4 fixture records (not duplicated, not dropped).
|
||||
assert len(results) == 4
|
||||
assert {r["wx_id"] for r in results} == {
|
||||
rec["wechatId"] for rec in WXUSER_RECORDS
|
||||
}
|
||||
|
||||
# Field mapping for each record (compare against the source record).
|
||||
for result, record in zip(results, WXUSER_RECORDS, strict=True):
|
||||
# Identity.
|
||||
assert result["wx_id"] == record["wechatId"]
|
||||
|
||||
# Naming.
|
||||
assert result["nick_name"] == record["nickName"]
|
||||
assert result["alias"] == record["alias"]
|
||||
|
||||
# Phones.
|
||||
assert result["phone"] == record["phone"]
|
||||
|
||||
# Embedded staff fields (the owning employee).
|
||||
assert result["user_id"] == record["userId"]
|
||||
assert result["user_name"] == record["userName"]
|
||||
|
||||
# Counts.
|
||||
assert result["contacts_num"] == record["contactsNum"]
|
||||
assert result["group_num"] == record["groupNum"]
|
||||
assert result["send_num"] == record["sendNum"]
|
||||
assert result["receive_num"] == record["receiveNum"]
|
||||
|
||||
# Status / sizing / timestamps.
|
||||
assert result["is_on_job"] == record["isOnJob"]
|
||||
assert result["db_size"] == record["dbSize"]
|
||||
assert result["online_status"] == record["onlineStatus"]
|
||||
assert result["last_update"] == record["lastUpdate"]
|
||||
|
||||
# Version carried verbatim; raw_json preserves the original record.
|
||||
assert result["version"] == record["version"]
|
||||
assert result["raw_json"] == record
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pagination body: pageNum/pageSize advanced correctly
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_wechat_accounts_sends_pagination_body():
|
||||
"""Outgoing request body carries pageNum/pageSize; advances per page."""
|
||||
page1 = make_wxuser_list_response(
|
||||
WXUSER_RECORDS[:1], total=TOTAL, page_num=1, page_size=1
|
||||
)
|
||||
page2 = make_wxuser_list_response(
|
||||
[], total=TOTAL, page_num=2, page_size=1
|
||||
)
|
||||
route = respx.post(WXUSER_LIST_URL).mock(
|
||||
side_effect=[
|
||||
httpx.Response(200, json=page1),
|
||||
httpx.Response(200, json=page2),
|
||||
]
|
||||
)
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [
|
||||
r async for r in scrape_wechat_accounts(client, page_size=1)
|
||||
]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
# One record on page 1, empty page 2 stops the loop.
|
||||
assert len(results) == 1
|
||||
assert route.call_count == 2
|
||||
|
||||
# First request: pageNum=1.
|
||||
body1 = _json.loads(route.calls[0].request.content.decode())
|
||||
assert body1["pageNum"] == 1
|
||||
assert body1["pageSize"] == 1
|
||||
|
||||
# Second request: pageNum advanced to 2.
|
||||
body2 = _json.loads(route.calls[1].request.content.decode())
|
||||
assert body2["pageNum"] == 2
|
||||
assert body2["pageSize"] == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Empty first page -> no yields, single call
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@respx.mock
|
||||
async def test_scrape_wechat_accounts_stops_on_empty_first_page():
|
||||
"""If the very first page is empty, scraper stops without over-fetching."""
|
||||
empty = make_wxuser_list_response([], total=0, page_num=1)
|
||||
route = respx.post(WXUSER_LIST_URL).mock(
|
||||
return_value=httpx.Response(200, json=empty)
|
||||
)
|
||||
|
||||
client = _primed_client()
|
||||
try:
|
||||
results = [r async for r in scrape_wechat_accounts(client)]
|
||||
finally:
|
||||
await client.aclose()
|
||||
|
||||
assert results == []
|
||||
assert route.call_count == 1
|
||||
Reference in New Issue
Block a user