feat: department + wechat account/staff scrapers

M2 organization scrapers for yunke-server:

- scrape_departments: POST /usercenter/dept/tree. Single request (no
  pagination). Defensively unwraps the envelope (data is node list OR an
  object wrapping nodes under children), flattens BOTH nested and flat
  shapes, and reconstructs the materialized path by walking the pId chain
  (ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.

- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
  pagination via the existing paginate_by_page helper. Envelope differs
  from friend_list (no basicResult wrapper): data.data/data.total. Maps
  wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
  sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
  WechatAccount columns; carries embedded staff pair for the repo layer.

Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
  coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
  upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
  forward phone/sale_phone/is_on_job when present.

Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).

Suite: 97 -> 111 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 13:46:23 +08:00
parent eebff9f786
commit 2b79e7a232
7 changed files with 1267 additions and 22 deletions
+59 -14
View File
@@ -48,12 +48,14 @@ import sqlalchemy as sa
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import DeclarativeBase
from app.db.models import Customer, Message, Staff, WechatAccount
from app.db.models import Customer, Department, Message, Staff, WechatAccount
__all__ = [
"upsert_customer",
"upsert_customers",
"upsert_department",
"upsert_messages",
"upsert_staff",
"upsert_staff_from_account",
"upsert_wechat_account",
]
@@ -303,29 +305,67 @@ async def upsert_messages(session: AsyncSession, recs: list[dict[str, Any]]) ->
return len(recs)
# ---------------------------------------------------------------------------
# Department upsert (Task 2.0)
# ---------------------------------------------------------------------------
async def upsert_department(session: AsyncSession, rec: dict[str, Any]) -> None:
"""Idempotently upsert one :class:`Department` (PK = ``dept_id``).
Same ``dept_id`` twice -> one row, with non-PK fields (``parent_id`` /
``name`` / ``path`` / ``raw_json``) refreshed from the later record.
"""
await _upsert_one(session, Department, rec)
# ---------------------------------------------------------------------------
# WechatAccount + embedded Staff (best-effort)
# ---------------------------------------------------------------------------
async def upsert_wechat_account(session: AsyncSession, rec: dict[str, Any]) -> None:
"""Best-effort idempotent upsert for :class:`WechatAccount` (PK = ``wx_id``).
async def upsert_staff(session: AsyncSession, rec: dict[str, Any]) -> None:
"""Idempotently upsert one :class:`Staff` row from a staff-shaped dict.
The companion :func:`upsert_staff_from_account` covers the embedded staff
row (``user_id`` + ``user_name``); the two are split so callers can pick
either or both. A full staff scraper (departments, phones, etc.) is a
LATER milestone.
PK = ``user_id``. Same ``user_id`` twice -> one row, with non-PK fields
refreshed. Applies the documented coercions (``is_on_job`` -> bool).
This is the primitive staff upsert; the account-scoped helper
:func:`upsert_staff_from_account` (which extracts the staff fields from a
wechat-account record) delegates here, and :func:`upsert_wechat_account`
calls that helper so an account upsert automatically refreshes its
embedded staff row too.
"""
await _upsert_one(session, Staff, rec)
async def upsert_wechat_account(session: AsyncSession, rec: dict[str, Any]) -> None:
"""Idempotently upsert a :class:`WechatAccount` AND its embedded Staff row.
PK = ``wx_id``. The companion :func:`upsert_staff_from_account` covers the
embedded staff row (``user_id`` + ``user_name``); this helper calls it
automatically so an account scrape that lands accounts also lands the
owning staff — callers do not need to upsert staff separately.
The staff upsert is best-effort: an account record with no ``user_id``
(e.g. an orphaned / unassigned wechat number) still writes the account
row; the staff upsert silently no-ops.
"""
await _upsert_one(session, WechatAccount, rec)
await upsert_staff_from_account(session, rec)
async def upsert_staff_from_account(session: AsyncSession, rec: dict[str, Any]) -> None:
"""Best-effort upsert of the Staff row embedded in a wechat-account record.
Maps the account's ``user_id`` + ``user_name`` (plus ``is_on_job`` when
present) onto :class:`Staff` (PK = ``user_id``). Silently no-ops when the
record has no ``user_id`` — there is nothing to key the row on. A
dedicated staff scraper that fills out phone / dept is a LATER milestone.
Maps the account's ``user_id`` + ``user_name`` (plus ``is_on_job`` /
``phone`` / ``sale_phone`` when present) onto :class:`Staff`
(PK = ``user_id``) and delegates to :func:`upsert_staff`. Silently no-ops
when the record has no ``user_id`` — there is nothing to key the row on.
Per Task 2.1, dept linkage for staff is inferred ELSEWHERE (from
customers' ``deptRoute``); staff with no customers have no dept. This
helper therefore does not populate ``dept_id`` — that is a later concern.
"""
user_id = rec.get("user_id")
if not user_id:
@@ -335,6 +375,11 @@ async def upsert_staff_from_account(session: AsyncSession, rec: dict[str, Any])
"name": rec.get("user_name"),
"raw_json": rec,
}
if "is_on_job" in rec:
staff_rec["is_on_job"] = rec["is_on_job"]
await _upsert_one(session, Staff, staff_rec)
# Forward any Staff-relevant fields the account record happens to carry.
# ``phone`` and ``sale_phone`` are usually absent on the wxUser payload
# (they come from the WhatsApp account payload); carry them through when
# present so a future shared caller does not silently drop them.
for key in ("is_on_job", "phone", "sale_phone"):
if key in rec:
staff_rec[key] = rec[key]
await upsert_staff(session, staff_rec)