feat: department + wechat account/staff scrapers
M2 organization scrapers for yunke-server: - scrape_departments: POST /usercenter/dept/tree. Single request (no pagination). Defensively unwraps the envelope (data is node list OR an object wrapping nodes under children), flattens BOTH nested and flat shapes, and reconstructs the materialized path by walking the pId chain (ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name. - scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number pagination via the existing paginate_by_page helper. Envelope differs from friend_list (no basicResult wrapper): data.data/data.total. Maps wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/ sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to WechatAccount columns; carries embedded staff pair for the repo layer. Repos: - upsert_department: idempotent upsert into Department (PK dept_id). - upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job coerced to bool). - upsert_wechat_account: now ALSO upserts the embedded Staff row via upsert_staff_from_account (best-effort; skips when account has no user_id). - upsert_staff_from_account: refactored to delegate to upsert_staff and forward phone/sale_phone/is_on_job when present. Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data + empty), test_scrape_wechat_accounts (2-page pagination + body advance + empty first page), test_repos extended (dept idempotency/root parent, direct staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl. departed + orphan). Suite: 97 -> 111 passing. ruff check clean. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -48,12 +48,14 @@ import sqlalchemy as sa
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import DeclarativeBase
|
||||
|
||||
from app.db.models import Customer, Message, Staff, WechatAccount
|
||||
from app.db.models import Customer, Department, Message, Staff, WechatAccount
|
||||
|
||||
__all__ = [
|
||||
"upsert_customer",
|
||||
"upsert_customers",
|
||||
"upsert_department",
|
||||
"upsert_messages",
|
||||
"upsert_staff",
|
||||
"upsert_staff_from_account",
|
||||
"upsert_wechat_account",
|
||||
]
|
||||
@@ -303,29 +305,67 @@ async def upsert_messages(session: AsyncSession, recs: list[dict[str, Any]]) ->
|
||||
return len(recs)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Department upsert (Task 2.0)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def upsert_department(session: AsyncSession, rec: dict[str, Any]) -> None:
|
||||
"""Idempotently upsert one :class:`Department` (PK = ``dept_id``).
|
||||
|
||||
Same ``dept_id`` twice -> one row, with non-PK fields (``parent_id`` /
|
||||
``name`` / ``path`` / ``raw_json``) refreshed from the later record.
|
||||
"""
|
||||
await _upsert_one(session, Department, rec)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WechatAccount + embedded Staff (best-effort)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def upsert_wechat_account(session: AsyncSession, rec: dict[str, Any]) -> None:
|
||||
"""Best-effort idempotent upsert for :class:`WechatAccount` (PK = ``wx_id``).
|
||||
async def upsert_staff(session: AsyncSession, rec: dict[str, Any]) -> None:
|
||||
"""Idempotently upsert one :class:`Staff` row from a staff-shaped dict.
|
||||
|
||||
The companion :func:`upsert_staff_from_account` covers the embedded staff
|
||||
row (``user_id`` + ``user_name``); the two are split so callers can pick
|
||||
either or both. A full staff scraper (departments, phones, etc.) is a
|
||||
LATER milestone.
|
||||
PK = ``user_id``. Same ``user_id`` twice -> one row, with non-PK fields
|
||||
refreshed. Applies the documented coercions (``is_on_job`` -> bool).
|
||||
|
||||
This is the primitive staff upsert; the account-scoped helper
|
||||
:func:`upsert_staff_from_account` (which extracts the staff fields from a
|
||||
wechat-account record) delegates here, and :func:`upsert_wechat_account`
|
||||
calls that helper so an account upsert automatically refreshes its
|
||||
embedded staff row too.
|
||||
"""
|
||||
await _upsert_one(session, Staff, rec)
|
||||
|
||||
|
||||
async def upsert_wechat_account(session: AsyncSession, rec: dict[str, Any]) -> None:
|
||||
"""Idempotently upsert a :class:`WechatAccount` AND its embedded Staff row.
|
||||
|
||||
PK = ``wx_id``. The companion :func:`upsert_staff_from_account` covers the
|
||||
embedded staff row (``user_id`` + ``user_name``); this helper calls it
|
||||
automatically so an account scrape that lands accounts also lands the
|
||||
owning staff — callers do not need to upsert staff separately.
|
||||
|
||||
The staff upsert is best-effort: an account record with no ``user_id``
|
||||
(e.g. an orphaned / unassigned wechat number) still writes the account
|
||||
row; the staff upsert silently no-ops.
|
||||
"""
|
||||
await _upsert_one(session, WechatAccount, rec)
|
||||
await upsert_staff_from_account(session, rec)
|
||||
|
||||
|
||||
async def upsert_staff_from_account(session: AsyncSession, rec: dict[str, Any]) -> None:
|
||||
"""Best-effort upsert of the Staff row embedded in a wechat-account record.
|
||||
|
||||
Maps the account's ``user_id`` + ``user_name`` (plus ``is_on_job`` when
|
||||
present) onto :class:`Staff` (PK = ``user_id``). Silently no-ops when the
|
||||
record has no ``user_id`` — there is nothing to key the row on. A
|
||||
dedicated staff scraper that fills out phone / dept is a LATER milestone.
|
||||
Maps the account's ``user_id`` + ``user_name`` (plus ``is_on_job`` /
|
||||
``phone`` / ``sale_phone`` when present) onto :class:`Staff`
|
||||
(PK = ``user_id``) and delegates to :func:`upsert_staff`. Silently no-ops
|
||||
when the record has no ``user_id`` — there is nothing to key the row on.
|
||||
|
||||
Per Task 2.1, dept linkage for staff is inferred ELSEWHERE (from
|
||||
customers' ``deptRoute``); staff with no customers have no dept. This
|
||||
helper therefore does not populate ``dept_id`` — that is a later concern.
|
||||
"""
|
||||
user_id = rec.get("user_id")
|
||||
if not user_id:
|
||||
@@ -335,6 +375,11 @@ async def upsert_staff_from_account(session: AsyncSession, rec: dict[str, Any])
|
||||
"name": rec.get("user_name"),
|
||||
"raw_json": rec,
|
||||
}
|
||||
if "is_on_job" in rec:
|
||||
staff_rec["is_on_job"] = rec["is_on_job"]
|
||||
await _upsert_one(session, Staff, staff_rec)
|
||||
# Forward any Staff-relevant fields the account record happens to carry.
|
||||
# ``phone`` and ``sale_phone`` are usually absent on the wxUser payload
|
||||
# (they come from the WhatsApp account payload); carry them through when
|
||||
# present so a future shared caller does not silently drop them.
|
||||
for key in ("is_on_job", "phone", "sale_phone"):
|
||||
if key in rec:
|
||||
staff_rec[key] = rec[key]
|
||||
await upsert_staff(session, staff_rec)
|
||||
|
||||
Reference in New Issue
Block a user