Commit Graph

58 Commits

Author SHA1 Message Date
浅念 2d1943a74e fix(api): 客户/群排序改回last_chat_time DESC(云客按最近沟通优先)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-15 14:18:42 +08:00
浅念 854da6e566 fix(api): 客户/群列表排序改为friend_id/room_id ASC(对齐云客API自然顺序)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-15 14:14:24 +08:00
浅念 2154d050e5 fix(media): image_legacy渲染+下载覆盖+WA raw_json URL回退
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-13 14:02:42 +08:00
浅念 c261b2bc9a fix(security): CORS限制+SSRF防护+PATCH乐观锁+accounts安全cap(4 important bugs)
- main.py + config.py: CORS origins read from Settings.cors_origins (comma-separated).
  Empty falls back to dev origin localhost:5173; never wildcard. Methods locked
  to GET/PATCH/POST; allow_credentials=False.
- scraper/media.py: download_url now passes follow_redirects=False to httpx.
  OSS URLs return 200 directly; a 3xx is treated as a download failure (caught
  by existing except clauses) instead of opening an SSRF vector to internal hosts.
- routers/voice_clips.py: PATCH /voice-clips/{id}/match accepts an optional
  If-Match header carrying the client's last-seen updated_at. On mismatch the
  row was modified by another request and the server returns 409. Without the
  header the behaviour is unchanged (backward compatible). Response now echoes
  the post-update updated_at so the client can fold it into the next request.
- routers/organization.py: /api/accounts now UNION ALLs the wechat_account and
  whatsapp_account projections at the SQL level and pushes COUNT + pagination
  (offset/limit) down to the DB. Stable ordering on (channel, account_id).
  Removes the in-memory materialization that broke at scale.

Tests: 395 passed. Added optimistic-lock coverage (If-Match accepted / stale 409 /
absent header backward-compat). Updated existing PATCH tests for the new
updated_at response field.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-12 11:41:30 +08:00
浅念 c7d54a135e fix(tests): 更新风控参数断言匹配新默认值(2.0/2.5/300/5-120/6)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-12 10:41:46 +08:00
浅念 caef32891e fix(security): 路径穿越防护+导出行数上限+并发同步锁(3 critical/important)
- media.py: GET /api/media/{hash} 现强制 file 必须位于 Settings.media_dir
  沙箱内(resolve + relative_to),杜绝 local_path 越权读取任意文件。
- export.py: /api/export/chat-html 加 EXPORT_MAX_ROWS=5000 硬上限(.limit()),
  超出时仅渲染前 5000 条并插入 .export-warning 横幅,防止 OOM。
- scheduler.py: _safe_run 增加 SyncRun advisory lock(60min TTL),发现同
  entity 存在 status='running' 且在 TTL 内的记录即返回 skipped SyncResult;
  同时为本轮 stake 一行 wrapping SyncRun,让并发 CLI/scheduler/手动触发
  能相互避让。崩溃场景由 TTL 自动恢复。
- tests: 新增 path-traversal 拒绝/export 截断警告/advisory lock skip+TTL
  恢复 5 个测试;test_api_media 的 fixture 改为 MEDIA_DIR=tmp_path 以适配
  新沙箱语义。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-12 10:37:13 +08:00
浅念 5ce33232c1 feat(sync): 聊天图片本地化(message media_hash+download_message_images+local-first rendering)
- Message 模型新增 media_hash 列 (sha256 FK -> Media.media_hash)
- Alembic 迁移 e5f6a7b8c9d0_message_media_hash (batch_alter_table, SQLite/MySQL 通用)
- app/scraper/media.py:download_message_images 批量下载器,镜像 download_call_recordings
  - 候选: media_hash IS NULL AND channel IN ('wechat','whatsapp') AND type IN ('image','video','file') AND 任一 file_*_url LIKE 'http%'
  - image 类型优先 file_th_url (缩略图,小快省),video/file 优先 file_url
  - per-row try/except + per-run tried set,失败不挂循环
- sync_message_images orchestrator 入口 + ENTITY_MESSAGE_IMAGES 常量 (提交自身 SyncRun,镜像 call_recordings)
- scheduler: _run_message_images + ENTITY_FUNCS['message_images'] + 加入 daily all job (尾部)
- MessageOut schema 暴露 media_hash 字段
- MessageBubble.vue local-first: media_hash 优先 /api/media/{hash},否则回退 OSS file_url/file_th_url/file_hd_url
- types.ts: Message 接口添加 media_hash 字段
- 新增 test_message_images_downloader.py (6 tests, 镜像 test_recording_downloader)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-11 18:24:11 +08:00
浅念 af7f8d71c8 feat(export): 导出 HTML 媒体内嵌(图片/语音/视频/附件, OSS URL 直接嵌入)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-11 17:22:11 +08:00
浅念 0e1cb87156 feat(scraper): WA 群 scraper(businessFriendList+type=1,待风控重置后实测)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-11 16:04:03 +08:00
浅念 9da272b992 feat(calls): voice_clip 手动匹配(voice_clip↔customer 本地关联写操作)
- VoiceClip: 新增 matched_friend_id 列 (String(128), nullable) + alembic 迁移 b3c4d5e6f7a8
- PATCH /api/voice-clips/{id}/match: friend_id 关联 / null 取消 (本地写, 不回写云客)
- GET /api/voice-clips: 批量解析 matched_friend_name (remark>nickname>real>id)
- VoiceClipModal: 已匹配客户列 + 匹配/取消匹配按钮 + 嵌套客户选择 picker (默认按员工列客户, 名搜索走 /search)
- 测试: 17 passed (新增 7 个 match 用例覆盖 set/clear/name 解析/404)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-11 10:43:34 +08:00
浅念 d9060b6920 feat(api): 长语音 voice_clip 后端(model+migration+scraper+sync+endpoint)
- VoiceClip model (BigInteger PK voice_clip_id, structured cols + raw_json)
- Alembic migration a1b2c3d4e5f6 (chains from 8c4f1d2a9e6b)
- scrape_voice_clips(client, user_id, *, channel, page_size) per-employee scraper
  over POST /wechatweb/call/list {pageNum, pageSize, userId}; channel stamped
  locally (wechat/whatsapp), startTime/endTime ms-epoch string -> int
- upsert_voice_clip repo + start_time/end_time/mp3_duration int coercion
- sync_voice_clips orchestrator: iterates distinct user_id across
  WechatAccount + WhatsappAccount, stamps channel per account table
- scheduler ENTITY_FUNCS['voice_clips'] + added to daily 'all' job
- GET /api/voice-clips (user_id exact beats user_real_name fuzzy; channel;
  company_code; start/end window on end_time; ordered end_time desc)
- VoiceClipOut schema + router registered in main.py
- tests: scraper (5) + API (11) = 16 new; full suite 370 green

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-10 18:12:28 +08:00
浅念 d220fa1a42 fix(scraper): 群抓取改用 friend_list/es+friendTypeYunke=2(替换 getChatroomList)
getChatroomList 在云客真机返回 0 行(已 Playwright 2026-08-10 验证)。
群与好友共用 /wechatweb/wxFriends/friend_list/es,仅以 friendTypeYunke 区分
(1=好友 / 2=群),响应信封一致(basicResult),复用 _parse_total/_parse_rows。

- 删除 CHATROOM_LIST_PATH / _parse_chatroom_total / _parse_chatroom_rows
  / _build_chatroom_body / _parse_member_ids(改写后已无引用)
- 新增 _build_group_body(pageNum/pageSize/friendTypeYunke="2")
- 重写 _normalize_chat_group 映射 friend_list/es 字段
  (friendWxId->room_id, friendWxNickname->room_name, friendNum->member_count,
   wxId/ownerName/userId, phone||wechatPhone, messageCount/sendNum/lastChatTime)
- friend_list/es 不返回成员明细,member_ids/member_ids_list 固定 None/[]
  (群列表可用;成员镜像延后)
- 删除 tests/cassettes/chatroom_fixture.py,改复用 wxFriends_fixture 信封
- 重写 test_scrape_wechat_groups:断言新端点 URL + friendTypeYunke="2" body
  + 新字段映射;member_ids 边界用例改为"始终 None/[]"

scrape_wechat_groups 签名不变(client + page_size),orchestrator/scheduler 未改。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-10 15:33:59 +08:00
浅念 a418830a90 feat(api): /api/sessions 账号会话列表(好友+群,今日消息数+最近联系时间,全本地聚合)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-08 14:58:23 +08:00
浅念 628659b849 refactor(api): staff-chat-stats 确定性排序 + last_is_send=None 分支测试 + 类型标注
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 18:37:26 +08:00
浅念 d2b2c0f5b3 feat(api): staff-chat-stats 聚合端点(微信员工 5 数字 + 客户明细,全本地 message)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 18:26:25 +08:00
浅念 1578db3467 feat(api): PhoneCallOut 暴露 record_media_hash
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 16:52:41 +08:00
浅念 b59e3cd9fe chore(sync): Task 3 review fixes——删未用 import + module docstring 诚实化 + repr(exc) 一致 + 批处理 guard
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 16:51:10 +08:00
浅念 38071e40ba feat(sync): sync_call_recordings + ENTITY_FUNCS['call_recordings'] 独立下载 pass
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 16:40:06 +08:00
浅念 1e399ff8c0 feat(scraper): download_call_recordings 下载通话录音到本地 Media + 回写 record_media_hash
复用 download_url/put_by_hash/upsert_media;幂等+resilient(tried 集合防死循环);media_type='audio'。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 16:30:29 +08:00
浅念 21c555a7e4 test: orchestrator voice sync skip/passthrough 路径覆盖
照 sync_wechat_accounts 的现有测试模式给 WA + 微信 voice 各加 2 个测试:
skip-when-no-root-department(断言 status=skipped + scraper 未调用 + 无 SyncRun)
+ passes-resolved-root-to-scraper(断言 resolved department_id 透传到 scraper)。
顺手补 SyncResult.status 的 'skipped' 第三态文档。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 09:57:34 +08:00
浅念 79b89bc919 fix(scraper): voice body 加 departmentId/isRoom/时间窗 + sync 解析 root department_id
实测 queryWeChatVoiceList 必须带 callSource+departmentId+isRoom+startTime+endTime,
否则 200 但 data 空(0 数据)。WA callSource=4 / 微信 callSource=0,宽窗抓全量。
sync_*_voice_calls 照 sync_wechat_accounts 模式解析 _find_root_department_id。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 09:49:31 +08:00
浅念 18620870c6 feat(sync): sync_wa_voice_calls + scheduler ENTITY_FUNCS wa_voice_calls
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 09:36:14 +08:00
浅念 169a64ee29 feat(scraper): WA 语音通话抓取 scrape_wa_voice_calls(callSource=4)
复用云客统一语音接口 queryWeChatVoiceList + callSource=4;channel=wa_voice_call
区分;WA 特有字段落 raw_json;复用 PhoneCall 表 + upsert_voice_call。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 09:26:16 +08:00
浅念 a66c3be00a chore: final Minor polish — export filename/test断言, downloadBlob 类型/清理, plan docs
address review Minors: drop redundant filename replace, assert 暂无消息,
tighten downloadBlob return type + try/finally, fix plan group time-picker wording.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 18:06:17 +08:00
浅念 d85bf64dde feat(api): GET /api/export/chat-html — 导出 1:1/群聊天为 HTML
支持 DM(talker_id) 与 群(room_id),可选 start/end 时间范围;
is_send 决定气泡左右。对齐云客微信客户/群列表「导出HTML聊天记录」。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-06 17:19:46 +08:00
浅念 dfdd0595d2 feat(staff): action buttons → filtered customer/group list
Adds groups.user_id filter; viewCustomers/viewGroups push owner_account_id query; CustomerList/GroupList read it.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 19:06:27 +08:00
浅念 fef176bd90 feat(api): customer filters user_id / add_time_since / last_chat_since
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:48:59 +08:00
浅念 8494d564ef feat(scraper): map customer add_time from raw_json
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 18:44:04 +08:00
浅念 df29dd027a feat: scheduler + CLI + docker-compose + README
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.

Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
  message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
  (departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
  all_wechat/all).
- Conversations for the message run are discovered from the local customer
  mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
  so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).

Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
  — long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
  mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.

Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
  image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
  reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
  SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
  backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.

Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).

Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 12:11:29 +08:00
浅念 a92fa2927c feat: full-text search (messages + customers)
Backend (yunke-server):
- GET /api/search?q=&type=messages|customers|all&channel=&start=&end=&page=&page_size=
  LIKE '%q%' scan over message.text + customer friend_nickname/real_name/
  phone/wechat_phone. type=all returns both sections; each is its own Page[T].
- type=messages orders by timestamp desc; type=customers orders by
  last_chat_time desc (nulls last).
- q required: missing/empty/whitespace -> HTTP 400.
- 14 new tests in tests/test_api_search.py (TDD red -> green).

Frontend (yunkes):
- New /search route + 搜索 nav item with Search icon.
- SearchView.vue: search bar, two stacked sections (消息结果 / 客户结果),
  counts + independent pagination per section, keyword highlighting (escaped),
  click message -> /chat/:channel/:ownerId/:talkerId,
  click customer -> all-fields drawer (CustomerList pattern reuse).
- search() wrapper in src/api/index.ts with SearchQuery/SearchPayload types.
- Empty state: "输入关键词搜索消息和客户".

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 11:46:21 +08:00
浅念 0d466c4e4a feat: full-field display for group/call/sms (raw_json + drawers)
Mirrors the Customer/Message/Staff/Accounts pattern: backend exposes the
verbatim upstream record on the response schema; the frontend list shows
source-matching columns and a row-click opens an all-fields el-drawer with
Chinese-labeled camelCase keys (unknown keys roll up under 其他).

Backend (yunke-server/app/api/schemas.py)
- GroupOut / PhoneCallOut / SmsOut: add raw_json: dict[str, Any] | None
  (Pydantic v2 from_attributes picks up the existing ORM column).
- Tests: extend tests/test_api_{groups,calls,sms}.py with a per-schema
  raw_json round-trip assertion (phone_call + voice_call payload shapes
  both covered for calls).

Frontend (yunkes)
- src/utils/{groupFields,callFields,smsFields}.ts: 中文 label map +
  grouped sections + readField (structured-column preference) +
  toDisplayField (image / time / audio / bool / status / direction).
- src/api/types.ts: add raw_json to Group / PhoneCall / Sms.
- src/views/GroupList.vue: richer columns (群名/群ID/成员数/群主/归属微信/
  消息数/最近聊天/渠道) + all-fields drawer (chatRoomId/chatRoomName,
  memberCount/memberIds, headImageUrl, wechatId/wechatNickName,
  userName/userPhone/userId, messageCount/ownMessageCount,
  lossMembers/addMembers, lastChatTime/updateTime).
- src/views/CallList.vue: keeps existing columns + adds all-fields drawer
  (id/startCallTime, callDuration/callSeconds/ringTime, callStatus/
  incomingCall, userId/userIdName/departmentId, callNumber/simPhone/
  simNumber/imei, recordFile/isUpload/downloadCount/aiAsr/qcStatus/
  planCustomerName/version + voice-call extras ossFileName/duration/
  startTime/endTime/remark). Drawer plays recording inline via audio
  render kind; existing play-record dialog retained.
- src/views/SmsList.vue: keeps existing columns + adds all-fields drawer
  (id/customerPhone, sms/smsSub, sender, direction, userId/
  senderDepartment, receiveName, createTime, simPhone/simNumber, source)
  with a full-body preview block at the top.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 10:12:49 +08:00
浅念 a57ed57616 fix: whatsapp businessFriendList needs page+departmentId+type
Live validation: businessFriendList uses 'page' (not pageNum) and REQUIRES
departmentId (root) + type or returns empty. Same class of bug as wxUser/list.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:53:26 +08:00
浅念 6ac47a1203 feat: full-field display for message/staff/accounts (raw_json + drawers)
Replicates the Customer all-fields pattern across the remaining three
entities. Backend now exposes `raw_json` on MessageOut / StaffOut /
AccountOut so the frontend can render every 云客 source field without a
second endpoint. Each entity gains a Chinese-label field utility + an
el-drawer surfacing all fields (structured-first, raw_json fallback,
unknown keys under 其他).

Backend:
- schemas.py: add `raw_json: dict | None` to MessageOut / StaffOut /
  AccountOut. MessageOut/StaffOut pick the column up via from_attributes;
  AccountOut is built explicitly in the router, which now threads
  `raw_json` through from both WechatAccount and WhatsappAccount rows.
- tests: new raw_json serialization tests for messages, staff, accounts
  (assert populated dict round-trips verbatim + NULL serializes as null).

Frontend (yunkes/):
- New utils: messageFields.ts, staffFields.ts, accountFields.ts — each
  with a 中文 label map, ordered groups, structured-column preference,
  and an 其他 fallback for unknown raw_json keys.
- Message: clicking a chat bubble opens an all-fields drawer (referMsg
  renders as pretty-printed JSON). Image preview + file download links
  stopPropagation so their own interactions still work.
- Staff: row click → drawer; columns relabeled (员工ID, 销售手机).
- Accounts: new AccountList.vue at /accounts with a 账号 nav item,
  unified WeChat/WhatsApp table (渠道/账号ID/昵称/微信号/归属员工/好友数/
  群数/收发/在职/最后更新), channel filter, pagination, row-click drawer.
- api: add Account type + getAccounts(); add raw_json to Message/Staff.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:47:50 +08:00
浅念 e5af1a2bc2 feat: customer full-field display (raw_json + cloud-matching list + all-fields drawer)
Backend:
- CustomerOut now exposes raw_json (verbatim upstream friend record) so the
  frontend can render every 云客 field without per-field structured columns.
- The verbatim blob round-trips via from_attributes; no model change needed.

Frontend:
- CustomerList table mirrors the 云客 客户列表 column set (头像/微信昵称/微信号/
  备注/真实姓名/手机/性别/地区/描述/销售员工/部门/来源/添加方式/添加时间/最近
  聊天/消息数/渠道). A readField helper reads the structured column first and
  falls back to raw_json.<camelCase>.
- Row click opens an all-fields drawer (right side) rendering EVERY field of
  the record with Chinese labels, grouped (基本信息/联系方式/归属员工与部门/时间
  与来源/消息统计/好友状态/其他). Timestamps format to local datetime, flags
  render 是/否, image URLs render as el-image thumbnails, empty values show —.
- New src/utils/customerFields.ts holds the camelCase→中文 label map, the
  structured-column alias map, group definitions, and value-type formatters;
  unknown raw_json keys auto-flow into an 其他 section so new upstream fields
  surface without code changes.
- Drawer 查看聊天记录 action navigates to /chat/:channel/:ownerId/:talkerId
  (owner_account_id + friend_id + channel), preserving the existing path.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:24:38 +08:00
浅念 0df53c6b1a fix: wechat account scrape (form-data+dates) + frontend staff page
Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
  isOnJob / departmentId / startdate / enddate + empty userId/salesname/
  weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
  plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
  Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.

Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:11:17 +08:00
浅念 4dcfc66741 fix: drop source header (breaks msg/chats) + context-aware talker_id
Live validation found two real bugs:
- YunkeClient sent 'source: yunkepc' on every request; msg/chats binds
  source as Integer -> 500. Removed (cookie auth suffices, per Phase 0).
- WeChat message talker_id used the masked 'talker' field; the timeline
  joins on the full peer id. Now: 1:1 -> oriTalker (full), group -> talker
  (member). ori_talker keeps the raw oriTalker (peer/chatroom).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:52:48 +08:00
浅念 b822e87d4d feat: remaining frontend views (groups/calls/sms/sync) + messageout media fields
Backend (Part A):
- MessageOut: add file_th_url, file_hd_url, duration, transcription_text, md5
  (room_id + sender_head_url already existed) so the chat timeline can render
  image thumbnails, voice duration/transcription, and avatars from a single
  response.
- PhoneCallOut: add record_file so the Calls view can offer recording playback.
- SmsOut: add content (full body) so the SMS detail dialog shows the whole text.
- Tests: red->green for each new field (media fields default None + populated,
  record_file round-trip, content round-trip). Suite 267 -> 271.

Frontend (Part B):
- api/index.ts: wrappers getGroups/getGroupMembers/getCalls/getSms/getSyncStatus
  + Group/GroupMember/PhoneCall/Sms/SyncState/SyncRun/SyncStatus types in
  types.ts (mirror the *Out pydantic schemas).
- views/GroupList.vue (/groups): table + channel filter + pagination; row click
  opens the group timeline.
- views/CallList.vue (/calls): table + call-type filter (phone_call/voice_call/
  all); "播放录音" action opens an el-dialog with <audio controls>.
- views/SmsList.vue (/sms): table; row click opens a detail dialog with the full
  content.
- views/SyncStatus.vue (/sync): two tables (states + recent_runs) with a manual
  refresh; run status color-coded.
- ChatTimeline.vue: reused for both 1:1 DM (talkerId) and group (roomId) by
  branching the query; group title + per-member sender already handled by
  MessageBubble.
- router: add /groups /calls /sms /sync + /group/:channel/:ownerId/:roomId.
- layouts/MainLayout.vue: replace the single 客户列表 button with an el-menu
  (客户 / 群 / 通话 / 短信 / 同步); org tree kept on the left.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:38:47 +08:00
浅念 ff7e9cb1d9 feat: backend read api for all entities (messages/groups/calls/sms/org/sync)
Adds the read-only API surface the upcoming Vue frontend needs to browse all
mirrored entities. Follows the existing customer-router pattern: Annotated
deps, Page[T] envelope, select+where+order_by, count() for total.

Endpoints (all GET):
* /api/departments          -> list[DepartmentOut] (path-ordered, plain list)
* /api/staff                -> Page[StaffOut] (department_id, is_on_job filters)
* /api/accounts             -> Page[AccountOut] (channel filter; unified
                             wechat+whatsapp projection built in memory)
* /api/messages             -> Page[MessageOut] (channel/owner/talker/room/
                             [start,end] filters; ordered timestamp asc)
* /api/groups               -> Page[GroupOut] (channel/owner filters)
* /api/groups/{id}/members  -> list[GroupMemberOut] (404 on unknown room)
* /api/calls                -> Page[PhoneCallOut] (channel/user/time window;
                             ordered start_call_time desc)
* /api/sms                  -> Page[SmsOut] (user/time window; create_time desc)
* /api/sync/status          -> {states, recent_runs} monitor payload

Schemas (Pydantic v2, from_attributes=True): MessageOut, GroupOut,
GroupMemberOut, PhoneCallOut, SmsOut, DepartmentOut, StaffOut, AccountOut
(unified across two tables), SyncStateOut, SyncRunOut, SyncStatusOut.
Only human-relevant fields per entity are exposed.

Tests: per-router TDD via httpx ASGITransport against an in-memory SQLite
override of get_session. A shared make_client factory in conftest.py builds
the seeded client, deduping the ~40-line fixture across the six new files.
41 new tests cover shape + every documented filter + pagination + ordering
(nulls-last verified on all desc/asc time columns).

Suite: 226 -> 267 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:15:40 +08:00
浅念 44d64dd237 feat: phone-call + sms scrapers
M4 (/pc/ CRM module): mirror phone-call + SMS endpoints.

Scraper (app/scraper/call.py):
- scrape_phone_calls: POST /pc/callLog/list -> PhoneCall (channel='phone_call')
  Envelope data.data / data.totalCount / data.pageCount; page-number pagination.
  incomingCall (1=incoming/0=outgoing) stored DIRECTLY (no inversion, unlike
  voice-call isSend); callSeconds preferred over legacy callDuration string;
  startCallTime (YYYY-MM-DD HH:MM:SS) -> ms epoch.
- scrape_sms: POST /pc/customer/smsLogsList -> Sms (channel='sms')
  Envelope data.data / data.dataCount (NOT totalCount), no pageCount;
  relies on empty-page stop signal. createTime (YYYY-MM-DD HH:MM) -> ms epoch;
  empty smsSub normalized to None content_preview.

Shared helper: extracted parse_datetime_to_ms(value, formats) to
app/scraper/parsers.py and refactored _parse_voice_start_time to delegate
(DRY across all three call/SMS datetime sources).

Repos: upsert_phone_call + upsert_sms (idempotent Core upsert by call_id /
sms_id). Added create_time to _INT_FIELDS for defensive numeric-str coercion,
matching start_call_time's existing treatment.

Orchestrator: sync_phone_calls + sync_sms (full re-scrape + idempotent upsert
+ SyncRun log + watermark), mirroring sync_wechat_voice_calls. Entity types
'phone_call' and 'sms'.

Tests (+15, suite 211 -> 226): respx-mocked scraper tests (2-page fixtures
exercising pagination + full field mapping), repo idempotency + coercion
tests, orchestrator SyncRun/watermark tests with monkeypatched scrapers.
ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 15:59:25 +08:00
浅念 8a22fd9ff8 feat: whatsapp account/friend/message scrapers
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.

- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
  scrape_whatsapp_friends (page-number pagination over the
  data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
  (backward-cursor pagination, JSON body {uid,friendUid,start,end} —
  NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
  maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
  1->image, -2->divider, else unknown; never raises) and a shared
  coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
  scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
  also upserts the embedded Staff(user_id, name) via the existing
  upsert_staff_from_account helper; WhatsappAccount imported.

Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.

Suite: 182 -> 211 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 15:48:03 +08:00
浅念 2508b5aec1 fix: live-validation fixes (list-coercion, .env session loading, smoke summary)
Cleanup after successful live validation (10 real customers scraped+stored):

- app/config.py: add yk_userid / yk_company / yk_cookie Settings fields so
  existing-session creds can be supplied via .env (bypasses programmatic
  login for validation with a live browser session).
- app/storage/repos.py: coerce list values to ';'-joined strings in
  _build_column_values. Fixes a real bug: the live friends-list API returns
  Customer.deptRoute as a LIST, but dept_route is a scalar String(255)
  column. raw_json still preserves the original list (no data loss).
- scripts/smoke_wechat.py:
  * Revert build_client_from_env to read os.environ ONLY (no get_settings
    fallbacks), so the monkeypatch.setenv unit tests stay hermetic and the
    real .env stops leaking in. 3 previously-red tests green again.
  * Add _push_settings_into_environ() (Settings -> os.environ, real env
    wins, only fills gaps) and call it as the first line of main(); so the
    smoke run still honors .env without the pure builder depending on
    Settings. .env -> Settings -> os.environ (once) -> build_client_from_env.
  * Keep the await _print_summary fix from validation.
- tests/test_repos.py: regression test for list dept_route coercion
  (stored joined 'A;B', raw_json keeps the list).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 15:23:43 +08:00
浅念 46e3880c0b feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):

- app/config.py: add 6 tunable settings with conservative defaults
  (request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
  risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
  method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
  for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
  headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
  headers (per-request overrides win); awaits the shared jittered limiter
  before sending; enforces a per-run request cap (RequestCapReached); and
  backs off exponentially on risk signals (HTTP 429 or 200 body carrying
  频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
  conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
  override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
  raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.

Suite: 171 -> 181 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:51:46 +08:00
浅念 83b1aeb97c feat: simple shared-token api auth
Adds a shared-token auth dependency for the viewing API (Task 2.9). Trust
model: when API_TOKEN is empty (the default) auth is disabled — we trust the
network boundary (local/intranet tool). When set, every request must present
the token via `Authorization: Bearer <token>` or `?token=<token>`, else 401.
`/api/health` is exempt so liveness probes work without the token.

- app/config.py: add `api_token: str = ""` to Settings.
- app/api/auth.py: `require_auth` dependency (secrets.compare_digest, reads
  get_settings() per-request so token rotation via cache_clear() works).
- app/api/main.py: apply the dependency globally; document the trust model.
- .env.example: document API_TOKEN.
- tests/test_api_auth.py: 8 tests covering disabled/correct/missing/wrong/
  malformed-header/query-param/health-exempt paths.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:30:28 +08:00
浅念 9cbd796975 feat: incremental sync orchestrator + watermarks
Task 2.7 — the coordination layer tying scrapers + repos together with
resumable, incremental sync.

app/sync/watermarks.py — SyncState CRUD keyed on (entity_type, channel,
scope_id): get_watermark / get_last_timestamp / get_last_version /
set_watermark (upsert). A partial advance (timestamp-only) does not clobber
the other cursor with NULL.

app/sync/orchestrator.py — the sync brain:
* sync_wechat_messages: time-window incremental. Reads last_timestamp for
  (message, wechat, "<wechat_id>:<friend_id>"), queries [last_ts, now] via
  scrape_wechat_messages, upserts each, best-effort downloads media, advances
  the watermark to max(timestamp). A re-run with no new messages queries an
  empty window and fetches nothing — the watermark advanced past.
* sync_{departments,wechat_accounts,wechat_friends,wechat_groups,
  wechat_voice_calls}: full re-scrape + idempotent upsert (these list APIs
  have no since-param; 'incremental' = re-scrape all, rely on PK idempotency).
  Watermark records last_synced_at only.
* sync_all_wechat: coordinator running departments -> accounts -> friends ->
  groups -> voice_calls, then per-conversation messages for an explicit
  (wechat_id, friend_id) list. Auto-discovery deferred.

Each run logs a SyncRun row (started -> success/failed, counts, error).
Failure policy: a scraper that raises is caught, recorded as failed, and a
failed SyncResult is returned — the orchestrator does not re-raise, so one
broken entity does not abort the batch. Media-download failures are
best-effort. The orchestrator never commits; callers own the transaction.

Tests (TDD red -> green; in-memory SQLite; scrapers/repos monkeypatched):
- watermarks: set/get, upsert (same key twice -> 1 row), partial-advance
  preserves the other cursor, defaults when absent.
- orchestrator messages: first run upserts 2 + advances watermark to T2;
  second run scrape called with start==T2 and upserts 0 (incremental proof);
  media failure does not abort; SyncRun success recorded.
- orchestrator friends: 3 records upserted, SyncRun success, watermark
  last_synced_at set.
- failure policy: scraper raises -> SyncRun failed + failed result, no raise
  (both friends and message paths).
- sync_all_wechat: 5 entities + 2 message runs in order, all success.

Suite: 149 -> 163 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:23:45 +08:00
浅念 7b49f94dcb feat: media downloader + serve api
M2 Task 2.6 (media downloader) + Task 2.8 (media serve API).

Downloader (app/scraper/media.py):
- download_url(client, url): fetches bytes via the YunkeClient's underlying
  httpx client (carries session cookies; respx-mockable; follows redirects).
- download_message_media(client, msg, store, session): URL-driven fetcher that
  picks the best http(s) URL (file_url -> file_hd_url -> file_th_url), dedups
  by md5 (skips the network when a Media row with local file already exists),
  falls back to sha256(bytes) when no md5, stores via ObjectStore.put_by_hash,
  and upserts a Media row. Skips type-10 system messages (non-media per
  phase0-findings §2) and type-3 opaque-id / text (no http url). Resilient:
  download errors are logged and return None (no raise).
- media_type_from_message / guess_mime_type helpers.

Repo (app/storage/repos.py): upsert_media — idempotent Core upsert into Media by
media_hash. Volatile download artifacts (local_path/size_bytes/mime_type) use
COALESCE(incoming, existing) on conflict so a re-scrape that did not re-download
does not clobber previously stored values. Exported from app.storage.

Serve API (app/api/routers/media.py): GET /api/media/{media_hash} streams the
local file via FileResponse (handles HTTP Range / 206). 404 when the row is
absent, has no local file, or the file is missing on disk. Content-type from the
row's mime_type, falling back to ext-derived guess.

Tests (+19, suite 130 -> 149, all green; ruff clean):
- test_media_downloader.py: image/video download + upsert; md5 dedup skips
  second network hit; sha256 fallback when no md5; type-3/text/system -> None;
  download failure -> None (no raise); upsert_media idempotency +
  null-local_path preservation + 3x-same-key-one-row.
- test_api_media.py: 200 + content-type + bytes; 404 unknown hash; 404 missing
  file on disk; Range -> 206 partial; ext-derived content-type when mime NULL.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:13:50 +08:00
浅念 41b9ec2571 feat: wechat voice-call scraper + group message coverage
- scrape_wechat_voice_calls: wraps POST /wechatweb/callRecord/queryWeChatVoiceList
  with page-number pagination (reuses paginate_by_page); normalizes records to
  the PhoneCall column shape with channel='voice_call'. Field mapping per
  phase0-findings §4: id->call_id, duration(str s)->call_duration(int),
  startTime(YYYY-MM-DD HH:MM:SS)->start_call_time(ms epoch), callStatus->
  call_status, talker->call_number, talkerNickName->plan_customer_name,
  ossFileName->record_file (may be null). isSend (0=in/1=out) is inverted to
  incoming_call (1=in/0=out) so the column's meaning matches the phone_call
  channel.
- upsert_voice_call: idempotent Core upsert into PhoneCall by call_id
  (sqlite/mysql dialect-aware). Adds call_duration/start_call_time to the
  _INT_FIELDS coercion net (scraper already emits ints; defensive for
  hand-built records).
- Group-message test (Task 2.3): locks in that scrape_wechat_messages already
  handles chatroom conversations (friendId=@chatroom, roomid set, talker=member
  wxid) — yields room_id populated and talker_id = the sending member. No
  scraper change required.

Suite: 123 -> 130 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 14:03:47 +08:00
浅念 2a74d3888d feat: wechat group+member and talker scrapers
Add scrape_wechat_groups (POST /wechatweb/chatroom/getChatroomList) and
scrape_wechat_talkers (POST /wechatweb/wxFriends/talkerList), both reusing
the page-number paginator. Groups parse the semicolon-joined memberIds into
a member_ids_list so upsert_chat_group can mirror GroupMember rows
(composite PK channel/room_id/member_id) idempotently alongside the
ChatGroup row. Talkers are a sync-planning input (member_count>0 marks a
group conversation); no dedicated table.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 13:54:16 +08:00
浅念 2b79e7a232 feat: department + wechat account/staff scrapers
M2 organization scrapers for yunke-server:

- scrape_departments: POST /usercenter/dept/tree. Single request (no
  pagination). Defensively unwraps the envelope (data is node list OR an
  object wrapping nodes under children), flattens BOTH nested and flat
  shapes, and reconstructs the materialized path by walking the pId chain
  (ancestor names joined with /). Maps id/pId/name -> dept_id/parent_id/name.

- scrape_wechat_accounts: POST /wechatweb/wxUser/list. Page-number
  pagination via the existing paginate_by_page helper. Envelope differs
  from friend_list (no basicResult wrapper): data.data/data.total. Maps
  wechatId/nickName/alias/phone/userId/userName/contactsNum/groupNum/
  sendNum/receiveNum/isOnJob/dbSize/onlineStatus/lastUpdate/version to
  WechatAccount columns; carries embedded staff pair for the repo layer.

Repos:
- upsert_department: idempotent upsert into Department (PK dept_id).
- upsert_staff: primitive idempotent staff upsert (PK user_id, is_on_job
  coerced to bool).
- upsert_wechat_account: now ALSO upserts the embedded Staff row via
  upsert_staff_from_account (best-effort; skips when account has no user_id).
- upsert_staff_from_account: refactored to delegate to upsert_staff and
  forward phone/sale_phone/is_on_job when present.

Tests (TDD red->green): test_scrape_departments (nested + flat + wrapped-data
+ empty), test_scrape_wechat_accounts (2-page pagination + body advance +
empty first page), test_repos extended (dept idempotency/root parent, direct
staff upsert + bool coercion, account-also-upserts-staff, re-upsert updates
both, no-user_id skips staff). Cassettes: dept_tree_fixture (same 4 nodes in
both nested + flat shapes), wxUser_fixture (4 capture-shaped accounts incl.
departed + orphan).

Suite: 97 -> 111 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 13:46:23 +08:00
浅念 eebff9f786 feat: RSA login encryption + confirmed wechat type codes
Replace the Phase-0 placeholder password encryption with real RSA
(PKCS#1 v1.5). encrypt_password now builds an RSA public key from the
getSecureKey modulus/exponent (RSAPublicNumbers, e=int(hex,16),
n=int(hex,16)), encrypts the UTF-8 password, and returns hex (documented
assumption -- base64 is the one-line fallback noted in the docstring).
get_secure_key now returns the inner data dict to match the new contract;
pc_login keeps {account, password} with a comment on the username/password
and account/pwd fallbacks if pcLogin rejects.

Extend normalize_wechat_type with the Phase-0 confirmed codes:
2 -> image, 3 -> image_legacy, 4 -> video, 10 -> system (1 -> text and
-2 -> divider unchanged; 6/9/13/15/21/22 stay unknown until confirmed).

Tests:
- encrypt_password round-trip: generate an RSA keypair, build secure_key
  from its public numbers, encrypt, then decrypt with the private key and
  assert the original password comes back. Pins 512-hex output for a
  2048-bit key. No network.
- Parser parametrize table extended for the new codes (and a downstream
  message-fixture assertion updated from unknown -> image_legacy).

Suite: 87 -> 97 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 13:37:42 +08:00
浅念 8dc8caa749 chore: wechat end-to-end smoke script
Wires the existing pieces together into a runnable integration check:
scrape ~10 WeChat friends + ~50 of their messages (last 7 days) from the
real yunke CRM (read-only), upsert into a local SQLite smoke DB, then
print a summary that verifies rows landed.

Two auth modes:
* Existing-session (preferred): YK_USERID/YK_COMPANY[/YK_COOKIE] -> YunkeClient
  constructed from a captured session, login bypassed.
* Login mode (Phase-0 blocked): YUNKE_USERNAME/YUNKE_PASSWORD -> attempts
  client.login(), which the real CRM will reject until the placeholder
  password encryption is replaced. Prints a clear message and exits 3.

build_client_from_env() is split out as a pure, network-free helper so
auth-mode selection is unit-testable. main() is async (asyncio.run in
__main__). HTTPStatusError from a 401/expired session is wrapped into a
helpful message + exit 4 (no traceback). client.aclose() + engine.dispose()
always run in finally.

Defaults to SMOKE_DB_URL=sqlite+aiosqlite:///./smoke.db so no MySQL is
required. Run from yunke-server/ with the venv active:

    python -m scripts.smoke_wechat

Tests cover the entrypoint shape and auth-mode selection (no network).
Suite: 77 -> 87 passing.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 12:16:02 +08:00