Commit Graph

63 Commits

Author SHA1 Message Date
浅念 c00522fc07 feat(staff): 微信员工沟通数 3 数字 → 明细 modal(补 Plan1 误判漏做的操作)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 17:37:08 +08:00
浅念 1368e3a471 feat(modal): YkStaffDetailModal Col 支持可选 onClick(单元格可点)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 17:32:05 +08:00
浅念 ebac26822b feat(api): add downloadBlob helper for blob attachment download
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 17:27:16 +08:00
浅念 d85bf64dde feat(api): GET /api/export/chat-html — 导出 1:1/群聊天为 HTML
支持 DM(talker_id) 与 群(room_id),可选 start/end 时间范围;
is_send 决定气泡左右。对齐云客微信客户/群列表「导出HTML聊天记录」。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 17:19:46 +08:00
浅念 bfdb5852bb feat(groups): align GroupList to 云客 — per-channel tabs, view-chat action
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 11:14:00 +08:00
浅念 de08bd09c7 docs: Plan 3 — GroupList 对齐云客(采集 + TDD plan)
phase0 采集云客「沟通」板块群列表(微信 wechatdetail/2 + WA friendlist 群列表 tab),
写 Plan 3(分 channel tab 各自列 + viewChat),经 plan-reviewer 评审 Approved。
已知 gap:smoke.db 0 群(空表验证)+ WA 群抓取缺失(字段推断)。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 11:11:37 +08:00
浅念 5d4a0c5901 feat(customers): align CustomerList to 云客 — per-channel tabs, 7 cols each, view-chat action
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 10:28:39 +08:00
浅念 3507de55fa feat(customerFields): add fromTypeLabel helper
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 10:13:40 +08:00
浅念 c31d79f1e2 docs: Plan 2 — CustomerList 对齐云客(采集 + TDD plan)
phase0 采集云客「沟通」板块客户列表(wechat wechatdetail/1 + waBusiness/friendlist),
dump raw_json 字段覆盖,产出列+交互+字段映射;写 Plan 2(分 channel tab 各 7 列 +
查看聊天记录→ChatTimeline),经 plan-reviewer 评审 Approved。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 10:12:36 +08:00
浅念 20b01ab849 fix(staff): review fixes — CSS ellipsis, owner_account_id scoping, viewChat guard
1. .yk-staffinfo__nick display:block (ellipsis now applies to WA inline span). 2. openAdded/openTalked modal use owner_account_id (always present) instead of user_id! (null would leak whole customer table). 3. viewChat guards null talker+room + awaits router.push. 4. WA 添加人数 title → 客户列表 (honest: WA has no add_time, shows all).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 19:25:24 +08:00
浅念 5bf43fc67b feat(staff): wechat stat-number/nickname → navigation
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 19:15:05 +08:00
浅念 16149fd499 feat(staff): WhatsApp nickname → chat timeline
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 19:09:24 +08:00
浅念 dfdd0595d2 feat(staff): action buttons → filtered customer/group list
Adds groups.user_id filter; viewCustomers/viewGroups push owner_account_id query; CustomerList/GroupList read it.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 19:06:27 +08:00
浅念 02a709ba93 fix(staff): format time columns in detail modal
YkStaffDetailModal Col gains optional formatter; baseCustomerCols time cols render via formatTime instead of raw epoch.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:58:06 +08:00
浅念 55578cf299 feat(staff): WhatsApp stat-number click → detail modal
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:56:27 +08:00
浅念 8e79f80925 feat(ui): YkStaffDetailModal shared detail-modal component
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:51:35 +08:00
浅念 635e1145b4 feat(api): customer query params user_id/add_time_since/last_chat_since
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:50:14 +08:00
浅念 fef176bd90 feat(api): customer filters user_id / add_time_since / last_chat_since
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:48:59 +08:00
浅念 8d7774b9f5 feat(scripts): backfill Customer.add_time from raw_json
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:45:55 +08:00
浅念 8494d564ef feat(scraper): map customer add_time from raw_json
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:44:04 +08:00
浅念 b5733f15d2 feat(db): add Customer.add_time column + migration
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:39:17 +08:00
浅念 9499110427 docs(plan): record wechat staff-list probe + adjust Task 8/9
WeChat interactions are navigation (not modal): 好友数/群数→customer/group list, 昵称→chat. Task 8 redone as navigation; Task 9 uses owner_account_id + row.channel.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:35:18 +08:00
浅念 1d815d84b9 docs: UI alignment spec + StaffList interaction plan
Spec: align all list/detail pages to 云客 (columns + interactions); covers interaction taxonomy, backend extensions (customer add_time/filters, /api/sessions), phase-0 capture. Plan 1: StaffList interaction closure + customer backend + YkStaffDetailModal.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:30:12 +08:00
浅念 5336431025 feat(staff): align StaffList columns to 云客 backend
WeChat tab: 6 compound columns mirroring wechatlist (员工信息/数据更新·最新消息时间/好友数/消息数/沟通数/及时回复). WhatsApp tab: 10 columns incl 操作 mirroring waBusiness/userlist. Fixes string-time bug (lastUpdate/uploadTime parsed as number showed --; cellTimeText renders strings).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:30:12 +08:00
浅念 62dc663294 feat: employee page with channel tabs (WeChat/WhatsApp account lists)
Replace the sparse staff-record table on /staff with a channel-tabbed
employee account view mirroring the 云客 backend's 员工微信列表 +
WhatsApp 员工列表. Each tab loads /api/accounts?channel=<channel> with
its own pagination state; rows open the existing account all-fields
drawer (reused from AccountList via accountFields.ts).

WeChat tab columns: 头像/微信昵称/微信号/手机号/员工姓名/好友数/群数/
发送/接收/数据库大小/在职/在线状态/最后更新. WhatsApp tab columns:
头像(el-avatar initials)/昵称/员工姓名/手机/销售手机/好友数/群数/
联系人数/新增好友/在职/最后更新. Reads raw_json via the structured-first
readField helper; tolerates boolean and 1/0 forms of isOnJob/onlineStatus.

Also fix the avatar field key in accountFields.ts (headImageUrl, with
headImgUrl as a legacy alias) so the WeChat drawer avatar renders — the
upstream wxUser record uses headImageUrl.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 14:13:21 +08:00
浅念 df29dd027a feat: scheduler + CLI + docker-compose + README
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.

Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
  message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
  (departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
  all_wechat/all).
- Conversations for the message run are discovered from the local customer
  mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
  so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).

Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
  — long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
  mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.

Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
  image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
  reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
  SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
  backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.

Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).

Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 12:11:29 +08:00
浅念 c8311bb496 fix: sanitize lone surrogate chars in stored data + backfill script
Upstream API records may contain lone surrogate chars (emoji fragments)
that break SQLite/MySQL utf-8 encoding. Now sanitized to U+FFFD before
storage. Includes backfill_customers.py for full customer backfill.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 11:57:12 +08:00
浅念 a92fa2927c feat: full-text search (messages + customers)
Backend (yunke-server):
- GET /api/search?q=&type=messages|customers|all&channel=&start=&end=&page=&page_size=
  LIKE '%q%' scan over message.text + customer friend_nickname/real_name/
  phone/wechat_phone. type=all returns both sections; each is its own Page[T].
- type=messages orders by timestamp desc; type=customers orders by
  last_chat_time desc (nulls last).
- q required: missing/empty/whitespace -> HTTP 400.
- 14 new tests in tests/test_api_search.py (TDD red -> green).

Frontend (yunkes):
- New /search route + 搜索 nav item with Search icon.
- SearchView.vue: search bar, two stacked sections (消息结果 / 客户结果),
  counts + independent pagination per section, keyword highlighting (escaped),
  click message -> /chat/:channel/:ownerId/:talkerId,
  click customer -> all-fields drawer (CustomerList pattern reuse).
- search() wrapper in src/api/index.ts with SearchQuery/SearchPayload types.
- Empty state: "输入关键词搜索消息和客户".

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 11:46:21 +08:00
浅念 8fde1d8597 fix: render chat media by URL (images/video/voice) instead of unknown-type placeholder
MessageBubble.vue was keys off the (incomplete) type code, so many messages
with real OSS media URLs (https://wx.oss.yunkecn.com/.../*.webp|*.mp4) showed
as '[未知消息]'. Rendering is now URL-driven: pick the best http URL from
file_url > file_hd_url > file_th_url, then render by extension
(image/video/audio/file). Type code is only used as a last-resort label.

- MessageBubble.vue: URL-driven cascade (URL by ext → duration →
  transcription → text → structured-type placeholder → localized tag).
  Image/video/audio/file children @click.stop so the all-fields drawer
  still opens on bubble click but not on media interactions.
- format.ts: wechatTypeLabel(type_raw) maps known codes (3→图片, 34→语音,
  43→图片/表情, 49→文件/名片, 42→名片, 9→表情, 20→红包/转账).
- ChatTimeline.vue drawer type tag uses wechatTypeLabel for WeChat msgs.
- style.css: compact .yk-msg__audio player for voice-with-URL messages.

Verified against smoke.db: 25 type-3 images + 3 type-43 videos that
previously showed '[未知消息]' now render as images/video. Build clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 10:23:01 +08:00
浅念 0d466c4e4a feat: full-field display for group/call/sms (raw_json + drawers)
Mirrors the Customer/Message/Staff/Accounts pattern: backend exposes the
verbatim upstream record on the response schema; the frontend list shows
source-matching columns and a row-click opens an all-fields el-drawer with
Chinese-labeled camelCase keys (unknown keys roll up under 其他).

Backend (yunke-server/app/api/schemas.py)
- GroupOut / PhoneCallOut / SmsOut: add raw_json: dict[str, Any] | None
  (Pydantic v2 from_attributes picks up the existing ORM column).
- Tests: extend tests/test_api_{groups,calls,sms}.py with a per-schema
  raw_json round-trip assertion (phone_call + voice_call payload shapes
  both covered for calls).

Frontend (yunkes)
- src/utils/{groupFields,callFields,smsFields}.ts: 中文 label map +
  grouped sections + readField (structured-column preference) +
  toDisplayField (image / time / audio / bool / status / direction).
- src/api/types.ts: add raw_json to Group / PhoneCall / Sms.
- src/views/GroupList.vue: richer columns (群名/群ID/成员数/群主/归属微信/
  消息数/最近聊天/渠道) + all-fields drawer (chatRoomId/chatRoomName,
  memberCount/memberIds, headImageUrl, wechatId/wechatNickName,
  userName/userPhone/userId, messageCount/ownMessageCount,
  lossMembers/addMembers, lastChatTime/updateTime).
- src/views/CallList.vue: keeps existing columns + adds all-fields drawer
  (id/startCallTime, callDuration/callSeconds/ringTime, callStatus/
  incomingCall, userId/userIdName/departmentId, callNumber/simPhone/
  simNumber/imei, recordFile/isUpload/downloadCount/aiAsr/qcStatus/
  planCustomerName/version + voice-call extras ossFileName/duration/
  startTime/endTime/remark). Drawer plays recording inline via audio
  render kind; existing play-record dialog retained.
- src/views/SmsList.vue: keeps existing columns + adds all-fields drawer
  (id/customerPhone, sms/smsSub, sender, direction, userId/
  senderDepartment, receiveName, createTime, simPhone/simNumber, source)
  with a full-body preview block at the top.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 10:12:49 +08:00
浅念 fd86adfb63 chore: seed chat messages for listed customers (seed_chats)
Iterates smoke.db customers, scrapes each conversation's recent messages
so timelines aren't empty. Bounded per-customer.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 09:54:40 +08:00
浅念 4bfaed4693 fix: parse WhatsApp lastUpdate datetime string to ms epoch
WA list endpoints return wall-clock strings ('2026-08-01 16:39') for
lastUpdate; storing raw broke CustomerOut.last_chat_time (int) validation
-> 500 on /api/customers?channel=whatsapp. Now parsed via parse_datetime_to_ms.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:58:00 +08:00
浅念 a57ed57616 fix: whatsapp businessFriendList needs page+departmentId+type
Live validation: businessFriendList uses 'page' (not pageNum) and REQUIRES
departmentId (root) + type or returns empty. Same class of bug as wxUser/list.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:53:26 +08:00
浅念 6ac47a1203 feat: full-field display for message/staff/accounts (raw_json + drawers)
Replicates the Customer all-fields pattern across the remaining three
entities. Backend now exposes `raw_json` on MessageOut / StaffOut /
AccountOut so the frontend can render every 云客 source field without a
second endpoint. Each entity gains a Chinese-label field utility + an
el-drawer surfacing all fields (structured-first, raw_json fallback,
unknown keys under 其他).

Backend:
- schemas.py: add `raw_json: dict | None` to MessageOut / StaffOut /
  AccountOut. MessageOut/StaffOut pick the column up via from_attributes;
  AccountOut is built explicitly in the router, which now threads
  `raw_json` through from both WechatAccount and WhatsappAccount rows.
- tests: new raw_json serialization tests for messages, staff, accounts
  (assert populated dict round-trips verbatim + NULL serializes as null).

Frontend (yunkes/):
- New utils: messageFields.ts, staffFields.ts, accountFields.ts — each
  with a 中文 label map, ordered groups, structured-column preference,
  and an 其他 fallback for unknown raw_json keys.
- Message: clicking a chat bubble opens an all-fields drawer (referMsg
  renders as pretty-printed JSON). Image preview + file download links
  stopPropagation so their own interactions still work.
- Staff: row click → drawer; columns relabeled (员工ID, 销售手机).
- Accounts: new AccountList.vue at /accounts with a 账号 nav item,
  unified WeChat/WhatsApp table (渠道/账号ID/昵称/微信号/归属员工/好友数/
  群数/收发/在职/最后更新), channel filter, pagination, row-click drawer.
- api: add Account type + getAccounts(); add raw_json to Message/Staff.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:47:50 +08:00
浅念 e5af1a2bc2 feat: customer full-field display (raw_json + cloud-matching list + all-fields drawer)
Backend:
- CustomerOut now exposes raw_json (verbatim upstream friend record) so the
  frontend can render every 云客 field without per-field structured columns.
- The verbatim blob round-trips via from_attributes; no model change needed.

Frontend:
- CustomerList table mirrors the 云客 客户列表 column set (头像/微信昵称/微信号/
  备注/真实姓名/手机/性别/地区/描述/销售员工/部门/来源/添加方式/添加时间/最近
  聊天/消息数/渠道). A readField helper reads the structured column first and
  falls back to raw_json.<camelCase>.
- Row click opens an all-fields drawer (right side) rendering EVERY field of
  the record with Chinese labels, grouped (基本信息/联系方式/归属员工与部门/时间
  与来源/消息统计/好友状态/其他). Timestamps format to local datetime, flags
  render 是/否, image URLs render as el-image thumbnails, empty values show —.
- New src/utils/customerFields.ts holds the camelCase→中文 label map, the
  structured-column alias map, group definitions, and value-type formatters;
  unknown raw_json keys auto-flow into an 其他 section so new upstream fields
  surface without code changes.
- Drawer 查看聊天记录 action navigates to /chat/:channel/:ownerId/:talkerId
  (owner_account_id + friend_id + channel), preserving the existing path.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:24:38 +08:00
浅念 0df53c6b1a fix: wechat account scrape (form-data+dates) + frontend staff page
Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
  isOnJob / departmentId / startdate / enddate + empty userId/salesname/
  weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
  plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
  Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.

Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:11:17 +08:00
浅念 4dcfc66741 fix: drop source header (breaks msg/chats) + context-aware talker_id
Live validation found two real bugs:
- YunkeClient sent 'source: yunkepc' on every request; msg/chats binds
  source as Integer -> 500. Removed (cookie auth suffices, per Phase 0).
- WeChat message talker_id used the masked 'talker' field; the timeline
  joins on the full peer id. Now: 1:1 -> oriTalker (full), group -> talker
  (member). ori_talker keeps the raw oriTalker (peer/chatroom).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:52:48 +08:00
浅念 b822e87d4d feat: remaining frontend views (groups/calls/sms/sync) + messageout media fields
Backend (Part A):
- MessageOut: add file_th_url, file_hd_url, duration, transcription_text, md5
  (room_id + sender_head_url already existed) so the chat timeline can render
  image thumbnails, voice duration/transcription, and avatars from a single
  response.
- PhoneCallOut: add record_file so the Calls view can offer recording playback.
- SmsOut: add content (full body) so the SMS detail dialog shows the whole text.
- Tests: red->green for each new field (media fields default None + populated,
  record_file round-trip, content round-trip). Suite 267 -> 271.

Frontend (Part B):
- api/index.ts: wrappers getGroups/getGroupMembers/getCalls/getSms/getSyncStatus
  + Group/GroupMember/PhoneCall/Sms/SyncState/SyncRun/SyncStatus types in
  types.ts (mirror the *Out pydantic schemas).
- views/GroupList.vue (/groups): table + channel filter + pagination; row click
  opens the group timeline.
- views/CallList.vue (/calls): table + call-type filter (phone_call/voice_call/
  all); "播放录音" action opens an el-dialog with <audio controls>.
- views/SmsList.vue (/sms): table; row click opens a detail dialog with the full
  content.
- views/SyncStatus.vue (/sync): two tables (states + recent_runs) with a manual
  refresh; run status color-coded.
- ChatTimeline.vue: reused for both 1:1 DM (talkerId) and group (roomId) by
  branching the query; group title + per-member sender already handled by
  MessageBubble.
- router: add /groups /calls /sms /sync + /group/:channel/:ownerId/:roomId.
- layouts/MainLayout.vue: replace the single 客户列表 button with an el-menu
  (客户 / 群 / 通话 / 短信 / 同步); org tree kept on the left.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:38:47 +08:00
浅念 6c4cd92ae6 feat: vue3 frontend scaffold + org tree + customer list + chat timeline
M5b viewing frontend for the yunke data-mirror, under yunkes/.

- Scaffold: Vite + Vue 3 + TS, with element-plus, @element-plus/icons-vue,
  pinia, vue-router@4, axios. Dev server on :5173 with /api and /api/media
  proxied to the FastAPI backend on :8000.
- API client (src/api/): axios instance (baseURL '/api') whose response
  interceptor unwraps the axios envelope; typed wrappers (getCustomers,
  getMessages, getDepartments, getStaff) returning Page<T> / arrays; types
  mirroring the backend *Out schemas (Message typed as a permissive superset
  so file_th_url/duration/transcription_text degrade gracefully today and
  light up once MessageOut exposes them).
- Layout: el-header top bar (title + channel switch) + el-aside department
  tree (built from the flat parent_id list via useOrgStore, with search
  filter) + el-main router-view.
- CustomerList (/customers): el-table with channel filter, department filter
  driven by the tree selection (route query), pagination, and row-click into
  the chat timeline (channel + owner_account_id + talker_id).
- ChatTimeline (/chat/:channel/:ownerId/:talkerId): asc-ordered message
  bubbles keyed by is_send (direction) and type (text/image/video/voice/file/
  divider/system/unknown); lands on the most-recent page and prepends older
  pages via load-more with scroll anchoring.
- Home '/' redirects to '/customers'.

npm run build (vue-tsc -b + vite build) is clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:27:22 +08:00
浅念 ff7e9cb1d9 feat: backend read api for all entities (messages/groups/calls/sms/org/sync)
Adds the read-only API surface the upcoming Vue frontend needs to browse all
mirrored entities. Follows the existing customer-router pattern: Annotated
deps, Page[T] envelope, select+where+order_by, count() for total.

Endpoints (all GET):
* /api/departments          -> list[DepartmentOut] (path-ordered, plain list)
* /api/staff                -> Page[StaffOut] (department_id, is_on_job filters)
* /api/accounts             -> Page[AccountOut] (channel filter; unified
                             wechat+whatsapp projection built in memory)
* /api/messages             -> Page[MessageOut] (channel/owner/talker/room/
                             [start,end] filters; ordered timestamp asc)
* /api/groups               -> Page[GroupOut] (channel/owner filters)
* /api/groups/{id}/members  -> list[GroupMemberOut] (404 on unknown room)
* /api/calls                -> Page[PhoneCallOut] (channel/user/time window;
                             ordered start_call_time desc)
* /api/sms                  -> Page[SmsOut] (user/time window; create_time desc)
* /api/sync/status          -> {states, recent_runs} monitor payload

Schemas (Pydantic v2, from_attributes=True): MessageOut, GroupOut,
GroupMemberOut, PhoneCallOut, SmsOut, DepartmentOut, StaffOut, AccountOut
(unified across two tables), SyncStateOut, SyncRunOut, SyncStatusOut.
Only human-relevant fields per entity are exposed.

Tests: per-router TDD via httpx ASGITransport against an in-memory SQLite
override of get_session. A shared make_client factory in conftest.py builds
the seeded client, deduping the ~40-line fixture across the six new files.
41 new tests cover shape + every documented filter + pagination + ordering
(nulls-last verified on all desc/asc time columns).

Suite: 226 -> 267 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:15:40 +08:00
浅念 44d64dd237 feat: phone-call + sms scrapers
M4 (/pc/ CRM module): mirror phone-call + SMS endpoints.

Scraper (app/scraper/call.py):
- scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call')
  Envelope data.data / data.totalCount / data.pageCount; page-number pagination.
  incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike
  voice-call isSend); callSeconds preferred over legacy callDuration string;
  startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch.
- scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms')
  Envelope data.data / data.dataCount (NOT totalCount), no pageCount;
  relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch;
  empty smsSub normalized to None content_preview.

Shared helper: extracted parse_datetime_to_ms(value, formats) to
app/scraper/parsers.py and refactored _parse_voice_start_time to delegate
(DRY across all three call/SMS datetime sources).

Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id /
sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion,
matching start_call_time's existing treatment.

Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert
+ SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types
'phone_call' and 'sms'.

Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures
exercising pagination + full field mapping), repo idempotency + coercion
tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers.
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 15:59:25 +08:00
浅念 8a22fd9ff8 feat: whatsapp account/friend/message scrapers
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.

- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
  scrape_whatsapp_friends (page-number pagination over the
  data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
  (backward-cursor pagination, JSON body {uid,friendUid,start,end} —
  NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
  maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
  1->image, -2->divider, else unknown; never raises) and a shared
  coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
  scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
  also upserts the embedded Staff(user_id, name) via the existing
  upsert_staff_from_account helper; WhatsappAccount imported.

Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.

Suite: 182 -> 211 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 15:48:03 +08:00
浅念 2508b5aec1 fix: live-validation fixes (list-coercion, .env session loading, smoke summary)
Cleanup after successful live validation (10 real customers scraped+stored):

- app/config.py: add yk_userid / yk_company / yk_cookie Settings fields so
  existing-session creds can be supplied via .env (bypasses programmatic
  login for validation with a live browser session).
- app/storage/repos.py: coerce list values to ';'-joined strings in
  _build_column_values. Fixes a real bug: the live friends-list API returns
  Customer.deptRoute as a LIST, but dept_route is a scalar String(255)
  column. raw_json still preserves the original list (no data loss).
- scripts/smoke_wechat.py:
  * Revert build_client_from_env to read os.environ ONLY (no get_settings
    fallbacks), so the monkeypatch.setenv unit tests stay hermetic and the
    real .env stops leaking in. 3 previously-red tests green again.
  * Add _push_settings_into_environ() (Settings -> os.environ, real env
    wins, only fills gaps) and call it as the first line of main(); so the
    smoke run still honors .env without the pure builder depending on
    Settings. .env -> Settings -> os.environ (once) -> build_client_from_env.
  * Keep the await _print_summary fix from validation.
- tests/test_repos.py: regression test for list dept_route coercion
  (stored joined 'A;B', raw_json keeps the list).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 15:23:43 +08:00
浅念 46e3880c0b feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):

- app/config.py: add 6 tunable settings with conservative defaults
  (request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
  risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
  method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
  for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
  headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
  headers (per-request overrides win); awaits the shared jittered limiter
  before sending; enforces a per-run request cap (RequestCapReached); and
  backs off exponentially on risk signals (HTTP 429 or 200 body carrying
  频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
  conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
  override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
  raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.

Suite: 171 -> 181 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:51:46 +08:00
浅念 83b1aeb97c feat: simple shared-token api auth
Adds a shared-token auth dependency for the viewing API (Task 2.9). Trust
model: when API_TOKEN is empty (the default) auth is disabled — we trust the
network boundary (local/intranet tool). When set, every request must present
the token via `Authorization: Bearer <token>` or `?token=<token>`, else 401.
`/api/health` is exempt so liveness probes work without the token.

- app/config.py: add `api_token: str = ""` to Settings.
- app/api/auth.py: `require_auth` dependency (secrets.compare_digest, reads
  get_settings() per-request so token rotation via cache_clear() works).
- app/api/main.py: apply the dependency globally; document the trust model.
- .env.example: document API_TOKEN.
- tests/test_api_auth.py: 8 tests covering disabled/correct/missing/wrong/
  malformed-header/query-param/health-exempt paths.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:30:28 +08:00
浅念 9cbd796975 feat: incremental sync orchestrator + watermarks
Task 2.7 — the coordination layer tying scrapers + repos together with
resumable, incremental sync.

app/sync/watermarks.py — SyncState CRUD keyed on (entity_type, channel,
scope_id): get_watermark / get_last_timestamp / get_last_version /
set_watermark (upsert). A partial advance (timestamp-only) does not clobber
the other cursor with NULL.

app/sync/orchestrator.py — the sync brain:
* sync_wechat_messages: time-window incremental. Reads last_timestamp for
  (message, wechat, "<wechat_id>:<friend_id>"), queries [last_ts, now] via
  scrape_wechat_messages, upserts each, best-effort downloads media, advances
  the watermark to max(timestamp). A re-run with no new messages queries an
  empty window and fetches nothing — the watermark advanced past.
* sync_{departments,wechat_accounts,wechat_friends,wechat_groups,
  wechat_voice_calls}: full re-scrape + idempotent upsert (these list APIs
  have no since-param; 'incremental' = re-scrape all, rely on PK idempotency).
  Watermark records last_synced_at only.
* sync_all_wechat: coordinator running departments -> accounts -> friends ->
  groups -> voice_calls, then per-conversation messages for an explicit
  (wechat_id, friend_id) list. Auto-discovery deferred.

Each run logs a SyncRun row (started -> success/failed, counts, error).
Failure policy: a scraper that raises is caught, recorded as failed, and a
failed SyncResult is returned — the orchestrator does not re-raise, so one
broken entity does not abort the batch. Media-download failures are
best-effort. The orchestrator never commits; callers own the transaction.

Tests (TDD red -> green; in-memory SQLite; scrapers/repos monkeypatched):
- watermarks: set/get, upsert (same key twice -> 1 row), partial-advance
  preserves the other cursor, defaults when absent.
- orchestrator messages: first run upserts 2 + advances watermark to T2;
  second run scrape called with start==T2 and upserts 0 (incremental proof);
  media failure does not abort; SyncRun success recorded.
- orchestrator friends: 3 records upserted, SyncRun success, watermark
  last_synced_at set.
- failure policy: scraper raises -> SyncRun failed + failed result, no raise
  (both friends and message paths).
- sync_all_wechat: 5 entities + 2 message runs in order, all success.

Suite: 149 -> 163 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:23:45 +08:00
浅念 7b49f94dcb feat: media downloader + serve api
M2 Task 2.6 (media downloader) + Task 2.8 (media serve API).

Downloader (app/scraper/media.py):
- download_url(client, url): fetches bytes via the YunkeClient's underlying
  httpx client (carries session cookies; respx-mockable; follows redirects).
- download_message_media(client, msg, store, session): URL-driven fetcher that
  picks the best http(s) URL (file_url -> file_hd_url -> file_th_url), dedups
  by md5 (skips the network when a Media row with local file already exists),
  falls back to sha256(bytes) when no md5, stores via ObjectStore.put_by_hash,
  and upserts a Media row. Skips type-10 system messages (non-media per
  phase0-findings §2) and type-3 opaque-id / text (no http url). Resilient:
  download errors are logged and return None (no raise).
- media_type_from_message / guess_mime_type helpers.

Repo (app/storage/repos.py): upsert_media — idempotent Core upsert into Media by
media_hash. Volatile download artifacts (local_path/size_bytes/mime_type) use
COALESCE(incoming, existing) on conflict so a re-scrape that did not re-download
does not clobber previously stored values. Exported from app.storage.

Serve API (app/api/routers/media.py): GET /api/media/{media_hash} streams the
local file via FileResponse (handles HTTP Range / 206). 404 when the row is
absent, has no local file, or the file is missing on disk. Content-type from the
row's mime_type, falling back to ext-derived guess.

Tests (+19, suite 130 -> 149, all green; ruff clean):
- test_media_downloader.py: image/video download + upsert; md5 dedup skips
  second network hit; sha256 fallback when no md5; type-3/text/system -> None;
  download failure -> None (no raise); upsert_media idempotency +
  null-local_path preservation + 3x-same-key-one-row.
- test_api_media.py: 200 + content-type + bytes; 404 unknown hash; 404 missing
  file on disk; Range -> 206 partial; ext-derived content-type when mime NULL.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:13:50 +08:00
浅念 41b9ec2571 feat: wechat voice-call scraper + group message coverage
- scrape_wechat_voice_calls: wraps POST /wechatweb/callRecord/queryWeChatVoiceList
  with page-number pagination (reuses paginate_by_page); normalizes records to
  the PhoneCall column shape with channel='voice_call'. Field mapping per
  phase0-findings §4: id->call_id, duration(str s)->call_duration(int),
  startTime(YYYY-MM-DD HH:MM:SS)->start_call_time(ms epoch), callStatus->
  call_status, talker->call_number, talkerNickName->plan_customer_name,
  ossFileName->record_file (may be null). isSend (0=in/1=out) is inverted to
  incoming_call (1=in/0=out) so the column's meaning matches the phone_call
  channel.
- upsert_voice_call: idempotent Core upsert into PhoneCall by call_id
  (sqlite/mysql dialect-aware). Adds call_duration/start_call_time to the
  _INT_FIELDS coercion net (scraper already emits ints; defensive for
  hand-built records).
- Group-message test (Task 2.3): locks in that scrape_wechat_messages already
  handles chatroom conversations (friendId=@chatroom, roomid set, talker=member
  wxid) — yields room_id populated and talker_id = the sending member. No
  scraper change required.

Suite: 123 -> 130 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:03:47 +08:00
浅念 2a74d3888d feat: wechat group+member and talker scrapers
Add scrape_wechat_groups (POST /wechatweb/chatroom/getChatroomList) and
scrape_wechat_talkers (POST /wechatweb/wxFriends/talkerList), both reusing
the page-number paginator. Groups parse the semicolon-joined memberIds into
a member_ids_list so upsert_chat_group can mirror GroupMember rows
(composite PK channel/room_id/member_id) idempotently alongside the
ChatGroup row. Talkers are a sync-planning input (member_count>0 marks a
group conversation); no dedicated table.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 13:54:16 +08:00
浅念 2b79e7a232 feat: department + wechat account/staff scrapers
M2 organization scrapers for yunke-server:

- scrape_departments: POST /usercenter/dept/tree. Single request (no
  pagination). Defensively unwraps the envelope (data is node list OR an
  object wrapping nodes under children), flattens BOTH nested and flat
  shapes, and reconstructs the materialized path by walking the pId chain
  (ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.

- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
  pagination via the existing paginate_by_page helper. Envelope differs
  from friend_list (no basicResult wrapper): data.data/data.total. Maps
  wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
  sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
  WechatAccount columns; carries embedded staff pair for the repo layer.

Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
  coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
  upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
  forward phone/sale_phone/is_on_job when present.

Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).

Suite: 97 -> 111 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 13:46:23 +08:00