xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
363 Commits 1 Branch 0 Tags
9dc22cd1e6cfc316fbb5bb33acd61fead575b5ba
Commit Graph

182 Commits

Author SHA1 Message Date
浅念 f9ed7dd821 feat(scraper): 微信消息翻页取完整历史(paginate_chats hasLast/end游标+200页上限,移除临时skip;全量回填3478对话+95361条至2025-10) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-18 09:15:04 +08:00
浅念 aeb8a89b03 fix(scraper): talker_id用发送的friend_id优先(API返回的talker可能脱敏导致消息查不到) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-17 09:29:39 +08:00
浅念 bcacf80a7b fix(scraper): 过滤type=-2日期分隔线 + 清理DB已有4062条分隔线 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 15:07:43 +08:00
浅念 9eb747f2b3 feat(scripts): 全量同步改纯页面方式(10步零API零风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:45:50 +08:00
浅念 4a148706b9 feat(scraper): 页面抓取器支持全实体(账号/WA客户/语音/长语音/WA消息,全浏览器方式) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:44:20 +08:00
浅念 7beae8cf55 feat(scripts): 全渠道全量重同步脚本(页面抓取+API全覆盖) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:38:01 +08:00
浅念 2d1943a74e fix(api): 客户/群排序改回last_chat_time DESC(云客按最近沟通优先) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:18:42 +08:00
浅念 854da6e566 fix(api): 客户/群列表排序改为friend_id/room_id ASC(对齐云客API自然顺序) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:14:24 +08:00
浅念 1f89eeddb0 feat(api): /api/customers 加 friend_id 筛选(用于查单个客户昵称) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:54:22 +08:00
浅念 cf858a00c2 fix(scraper): 消息页面抓取器改用浏览器fetch(可靠)+空数据安全 2026-08-14 15:30:02 +08:00
浅念 c6217ab1a9 feat(scraper): 页面抓取器支持消息(导航聊天窗口拦截msg/chats API) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-14 15:04:21 +08:00
浅念 bead7fdf43 fix(scraper): 页面抓取器支持员工列表翻页(之前只抓第一页10个) 2026-08-14 09:25:10 +08:00
浅念 004e6e4d6b feat(scraper): 页面浏览器抓取器(Playwright拦截SPA API响应,绕过风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:38:42 +08:00
浅念 2154d050e5 fix(media): image_legacy渲染+下载覆盖+WA raw_json URL回退 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:02:42 +08:00
浅念 9a953553ec fix(xss+perf): SearchView XSS消除(无v-html)+room_id索引+pool_recycle 
- SearchView.vue: 用 highlightSegments() 返回 {text,hit} 数组替代 v-html+escapeHtml+regex,模板用 <template v-for> + <mark>/<span> 渲染,从结构上消除 XSS 风险(4 处 v-html 全部移除: 消息片段/客户昵称/realName/手机号)
- models.py: Message 新增 ix_message_channel_owner_room_time(channel,owner_account_id,room_id,timestamp),群聊查询过滤 room_id 而非 talker_id,原 talker 索引无法覆盖
- migrations/d8eee96cd017: 新增 room_id 索引的 upgrade/downgrade (offline SQL 已验证)
- base.py + scheduler.py + cli.py: 所有生产 create_async_engine 增加 pool_recycle=3600,在 MySQL wait_timeout(默认 8h)断连前主动回收

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:50:45 +08:00
浅念 c261b2bc9a fix(security): CORS限制+SSRF防护+PATCH乐观锁+accounts安全cap(4 important bugs) 
- main.py + config.py: CORS origins read from Settings.cors_origins (comma-separated).
  Empty falls back to dev origin localhost:5173; never wildcard. Methods locked
  to GET/PATCH/POST; allow_credentials=False.
- scraper/media.py: download_url now passes follow_redirects=False to httpx.
  OSS URLs return 200 directly; a 3xx is treated as a download failure (caught
  by existing except clauses) instead of opening an SSRF vector to internal hosts.
- routers/voice_clips.py: PATCH /voice-clips/{id}/match accepts an optional
  If-Match header carrying the client's last-seen updated_at. On mismatch the
  row was modified by another request and the server returns 409. Without the
  header the behaviour is unchanged (backward compatible). Response now echoes
  the post-update updated_at so the client can fold it into the next request.
- routers/organization.py: /api/accounts now UNION ALLs the wechat_account and
  whatsapp_account projections at the SQL level and pushes COUNT + pagination
  (offset/limit) down to the DB. Stable ordering on (channel, account_id).
  Removes the in-memory materialization that broke at scale.

Tests: 395 passed. Added optimistic-lock coverage (If-Match accepted / stale 409 /
absent header backward-compat). Updated existing PATCH tests for the new
updated_at response field.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:41:30 +08:00
浅念 c7d54a135e fix(tests): 更新风控参数断言匹配新默认值(2.0/2.5/300/5-120/6) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 10:41:46 +08:00
浅念 caef32891e fix(security): 路径穿越防护+导出行数上限+并发同步锁(3 critical/important) 
- media.py: GET /api/media/{hash} 现强制 file 必须位于 Settings.media_dir
  沙箱内(resolve + relative_to),杜绝 local_path 越权读取任意文件。
- export.py: /api/export/chat-html 加 EXPORT_MAX_ROWS=5000 硬上限(.limit()),
  超出时仅渲染前 5000 条并插入 .export-warning 横幅,防止 OOM。
- scheduler.py: _safe_run 增加 SyncRun advisory lock(60min TTL),发现同
  entity 存在 status='running' 且在 TTL 内的记录即返回 skipped SyncResult;
  同时为本轮 stake 一行 wrapping SyncRun,让并发 CLI/scheduler/手动触发
  能相互避让。崩溃场景由 TTL 自动恢复。
- tests: 新增 path-traversal 拒绝/export 截断警告/advisory lock skip+TTL
  恢复 5 个测试;test_api_media 的 fixture 改为 MEDIA_DIR=tmp_path 以适配
  新沙箱语义。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 10:37:13 +08:00
浅念 461e9ff081 tune(anti-risk): 加固防风控策略(间隔2s+抖动2.5s+上限300+退避5-120s×6次) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 09:45:33 +08:00
浅念 5ce33232c1 feat(sync): 聊天图片本地化(message media_hash+download_message_images+local-first rendering) 
- Message 模型新增 media_hash 列 (sha256 FK -> Media.media_hash)
- Alembic 迁移 e5f6a7b8c9d0_message_media_hash (batch_alter_table, SQLite/MySQL 通用)
- app/scraper/media.py:download_message_images 批量下载器，镜像 download_call_recordings
  - 候选: media_hash IS NULL AND channel IN ('wechat','whatsapp') AND type IN ('image','video','file') AND 任一 file_*_url LIKE 'http%'
  - image 类型优先 file_th_url (缩略图，小快省)，video/file 优先 file_url
  - per-row try/except + per-run tried set，失败不挂循环
- sync_message_images orchestrator 入口 + ENTITY_MESSAGE_IMAGES 常量 (提交自身 SyncRun，镜像 call_recordings)
- scheduler: _run_message_images + ENTITY_FUNCS['message_images'] + 加入 daily all job (尾部)
- MessageOut schema 暴露 media_hash 字段
- MessageBubble.vue local-first: media_hash 优先 /api/media/{hash}，否则回退 OSS file_url/file_th_url/file_hd_url
- types.ts: Message 接口添加 media_hash 字段
- 新增 test_message_images_downloader.py (6 tests, 镜像 test_recording_downloader)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 18:24:11 +08:00
浅念 af7f8d71c8 feat(export): 导出 HTML 媒体内嵌(图片/语音/视频/附件, OSS URL 直接嵌入) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 17:22:11 +08:00
浅念 0e1cb87156 feat(scraper): WA 群 scraper(businessFriendList+type=1,待风控重置后实测) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 16:04:03 +08:00
浅念 538a0dc635 feat(sync): 长语音音频本地化(voice_clip media_hash+download+local-first playback) 
Mirror download_call_recordings pattern for voice_clips:
- VoiceClip.media_hash column + Alembic migration (c4d5e6f7a8b9)
- VoiceClipOut.media_hash + frontend VoiceClip.media_hash type
- download_voice_clip_audio in app/scraper/media.py (resilient, sha256-dedup,
  same download_url/upsert_media helpers as call recordings)
- sync_voice_clip_audio orchestrator wrapper (commits own SyncRun, mirrors
  sync_call_recordings semantics)
- scheduler: _run_voice_clip_audio + ENTITY_FUNCS['voice_clip_audio'] +
  appended to daily _run_all job (after voice_clips)
- VoiceClipModal playClip local-first: /api/media/{media_hash} with OSS
  oss_file_name fallback (mirrors CallList.playRecord)

Tested: 376 pytest pass; vue-tsc clean; migration applied; 1523/1533 clips
localized (10 pending are genuine upstream HTTP 404s — resilient skip).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 12:06:26 +08:00
浅念 9da272b992 feat(calls): voice_clip 手动匹配(voice_clip↔customer 本地关联写操作) 
- VoiceClip: 新增 matched_friend_id 列 (String(128), nullable) + alembic 迁移 b3c4d5e6f7a8
- PATCH /api/voice-clips/{id}/match: friend_id 关联 / null 取消 (本地写, 不回写云客)
- GET /api/voice-clips: 批量解析 matched_friend_name (remark>nickname>real>id)
- VoiceClipModal: 已匹配客户列 + 匹配/取消匹配按钮 + 嵌套客户选择 picker (默认按员工列客户, 名搜索走 /search)
- 测试: 17 passed (新增 7 个 match 用例覆盖 set/clear/name 解析/404)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 10:43:34 +08:00
浅念 d9060b6920 feat(api): 长语音 voice_clip 后端(model+migration+scraper+sync+endpoint) 
- VoiceClip model (BigInteger PK voice_clip_id, structured cols + raw_json)
- Alembic migration a1b2c3d4e5f6 (chains from 8c4f1d2a9e6b)
- scrape_voice_clips(client, user_id, *, channel, page_size) per-employee scraper
  over POST /wechatweb/call/list {pageNum, pageSize, userId}; channel stamped
  locally (wechat/whatsapp), startTime/endTime ms-epoch string -> int
- upsert_voice_clip repo + start_time/end_time/mp3_duration int coercion
- sync_voice_clips orchestrator: iterates distinct user_id across
  WechatAccount + WhatsappAccount, stamps channel per account table
- scheduler ENTITY_FUNCS['voice_clips'] + added to daily 'all' job
- GET /api/voice-clips (user_id exact beats user_real_name fuzzy; channel;
  company_code; start/end window on end_time; ordered end_time desc)
- VoiceClipOut schema + router registered in main.py
- tests: scraper (5) + API (11) = 16 new; full suite 370 green

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 18:12:28 +08:00
浅念 a0db4535f1 feat(sync): WA messages 自动同步(per-conversation 增量水位,镜像 sync_wechat_messages) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 17:50:01 +08:00
浅念 e8885efcc6 fix(sync): 补 import upsert_whatsapp_account(WA accounts sync NameError) 2026-08-10 17:02:28 +08:00
浅念 ceca5bda4c feat(sync): 接 WA 自动同步(accounts+friends wiring to orchestrator/scheduler) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 16:48:44 +08:00
浅念 d220fa1a42 fix(scraper): 群抓取改用 friend_list/es+friendTypeYunke=2(替换 getChatroomList) 
getChatroomList 在云客真机返回 0 行(已 Playwright 2026-08-10 验证)。
群与好友共用 /wechatweb/wxFriends/friend_list/es,仅以 friendTypeYunke 区分
(1=好友 / 2=群),响应信封一致(basicResult),复用 _parse_total/_parse_rows。

- 删除 CHATROOM_LIST_PATH / _parse_chatroom_total / _parse_chatroom_rows
  / _build_chatroom_body / _parse_member_ids(改写后已无引用)
- 新增 _build_group_body(pageNum/pageSize/friendTypeYunke="2")
- 重写 _normalize_chat_group 映射 friend_list/es 字段
  (friendWxId->room_id, friendWxNickname->room_name, friendNum->member_count,
   wxId/ownerName/userId, phone||wechatPhone, messageCount/sendNum/lastChatTime)
- friend_list/es 不返回成员明细,member_ids/member_ids_list 固定 None/[]
  (群列表可用;成员镜像延后)
- 删除 tests/cassettes/chatroom_fixture.py,改复用 wxFriends_fixture 信封
- 重写 test_scrape_wechat_groups:断言新端点 URL + friendTypeYunke="2" body
  + 新字段映射;member_ids 边界用例改为"始终 None/[]"

scrape_wechat_groups 签名不变(client + page_size),orchestrator/scheduler 未改。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 15:33:59 +08:00
浅念 a418830a90 feat(api): /api/sessions 账号会话列表(好友+群,今日消息数+最近联系时间,全本地聚合) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-08 14:58:23 +08:00
浅念 628659b849 refactor(api): staff-chat-stats 确定性排序 + last_is_send=None 分支测试 + 类型标注 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 18:37:26 +08:00
浅念 d2b2c0f5b3 feat(api): staff-chat-stats 聚合端点（微信员工 5 数字 + 客户明细，全本地 message） 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 18:26:25 +08:00
浅念 1578db3467 feat(api): PhoneCallOut 暴露 record_media_hash 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 16:52:41 +08:00
浅念 b59e3cd9fe chore(sync): Task 3 review fixes——删未用 import + module docstring 诚实化 + repr(exc) 一致 + 批处理 guard 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 16:51:10 +08:00
浅念 38071e40ba feat(sync): sync_call_recordings + ENTITY_FUNCS['call_recordings'] 独立下载 pass 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 16:40:06 +08:00
浅念 1e399ff8c0 feat(scraper): download_call_recordings 下载通话录音到本地 Media + 回写 record_media_hash 
复用 download_url/put_by_hash/upsert_media；幂等+resilient（tried 集合防死循环）；media_type='audio'。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 16:30:29 +08:00
浅念 8f70ac5220 feat(db): phone_call.record_media_hash 列 + 迁移（录音本地化 FK→Media） 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 16:20:18 +08:00
浅念 21c555a7e4 test: orchestrator voice sync skip/passthrough 路径覆盖 
照 sync_wechat_accounts 的现有测试模式给 WA + 微信 voice 各加 2 个测试：
skip-when-no-root-department（断言 status=skipped + scraper 未调用 + 无 SyncRun）
+ passes-resolved-root-to-scraper（断言 resolved department_id 透传到 scraper）。
顺手补 SyncResult.status 的 'skipped' 第三态文档。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 09:57:34 +08:00
浅念 79b89bc919 fix(scraper): voice body 加 departmentId/isRoom/时间窗 + sync 解析 root department_id 
实测 queryWeChatVoiceList 必须带 callSource+departmentId+isRoom+startTime+endTime，
否则 200 但 data 空（0 数据）。WA callSource=4 / 微信 callSource=0，宽窗抓全量。
sync_*_voice_calls 照 sync_wechat_accounts 模式解析 _find_root_department_id。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 09:49:31 +08:00
浅念 18620870c6 feat(sync): sync_wa_voice_calls + scheduler ENTITY_FUNCS wa_voice_calls 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 09:36:14 +08:00
浅念 169a64ee29 feat(scraper): WA 语音通话抓取 scrape_wa_voice_calls（callSource=4） 
复用云客统一语音接口 queryWeChatVoiceList + callSource=4；channel=wa_voice_call
区分；WA 特有字段落 raw_json；复用 PhoneCall 表 + upsert_voice_call。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 09:26:16 +08:00
浅念 a66c3be00a chore: final Minor polish — export filename/test断言, downloadBlob 类型/清理, plan docs 
address review Minors: drop redundant filename replace, assert 暂无消息,
tighten downloadBlob return type + try/finally, fix plan group time-picker wording.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-06 18:06:17 +08:00
浅念 d85bf64dde feat(api): GET /api/export/chat-html — 导出 1:1/群聊天为 HTML 
支持 DM(talker_id) 与 群(room_id)，可选 start/end 时间范围；
is_send 决定气泡左右。对齐云客微信客户/群列表「导出HTML聊天记录」。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-06 17:19:46 +08:00
浅念 dfdd0595d2 feat(staff): action buttons → filtered customer/group list 
Adds groups.user_id filter; viewCustomers/viewGroups push owner_account_id query; CustomerList/GroupList read it.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 19:06:27 +08:00
浅念 fef176bd90 feat(api): customer filters user_id / add_time_since / last_chat_since 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 18:48:59 +08:00
浅念 8d7774b9f5 feat(scripts): backfill Customer.add_time from raw_json 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 18:45:55 +08:00
浅念 8494d564ef feat(scraper): map customer add_time from raw_json 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 18:44:04 +08:00
浅念 b5733f15d2 feat(db): add Customer.add_time column + migration 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 18:39:17 +08:00
浅念 df29dd027a feat: scheduler + CLI + docker-compose + README 
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.

Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
  message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
  (departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
  all_wechat/all).
- Conversations for the message run are discovered from the local customer
  mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
  so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).

Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
  — long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
  mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.

Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
  image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
  reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
  SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
  backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.

Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).

Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 12:11:29 +08:00
浅念 c8311bb496 fix: sanitize lone surrogate chars in stored data + backfill script 
Upstream API records may contain lone surrogate chars (emoji fragments)
that break SQLite/MySQL utf-8 encoding. Now sanitized to U+FFFD before
storage. Includes backfill_customers.py for full customer backfill.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 11:57:12 +08:00