xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
184 Commits 1 Branch 0 Tags
b522be828a4e9e0554691d94df4346f0e85b3699
Commit Graph

184 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
浅念 b522be828a feat(scraper): 五实体导航拦截全改浏览器fetch(登录墙杀死拦截方式0捕获) 
- friends/groups/friend_list/es 部门级全量(记录自带owner,告别逐员工遍历)
- accounts/wxUser/list form模式(页码字段是page非pageNum,粘页检测保险)
- wa-accounts/wa-friends 直发;wa-friends 200页1万行(疑端点1万行硬上限)
- _fetch_pages 通用翻页助手:cookie失效首页code告警+粘页检测
- 实测:微信客户5916→6563/WA客户8058→8585/群851→873/账号50→55(捞回登录墙期间漏抓)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:41:33 +08:00
浅念 10cdd81d5c fix(scraper): 消息归一化补fileTh/fileHd→列映射(图片缩略图URL进列,媒体下载器可及) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:46:54 +08:00
浅念 5923cd05b1 feat(scheduler): 定时同步全面页面化(子进程跑页面脚本,API直调退出调度防风控) 
- 每日 full: bash scripts/full_sync_all.sh;每4h messages: scrape_via_page --entity messages
- 子进程隔离+asyncio.Lock互斥+非零退出只记JSON日志;自动补DB_URL环境
- ENTITY_FUNCS/run_once/CLI保留为手动调试,不再挂定时触发器
- README同步更新(Docker镜像需Playwright+Chromium);413测试绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:33:53 +08:00
浅念 02dd77a62f feat(scraper): WA群抓取(type=1真机验证,223群)+群成员派生(12840成员/750群) 
- wa-groups 实体:businessFriendList type=1 真机验证返回群,行结构是好友字段名
  (friendUid=@g.us 群ID/nickName=群名),专用 _normalize_wa_group_row
- backfill_group_members.py:无独立成员API,从群消息派生(talker=成员,sender_name=昵称)
- full_sync_all.sh:+4步(5b WA群/语音后录音重链接/8b 群消息+成员/消息上限99999)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:21:20 +08:00
浅念 f92aac1a3d feat(scraper): 微信群消息抓取(368778条/750群)+浏览器守护循环+SQLite忙等 
- 新 group-messages 实体:真机探明群消息=同一 msg/chats 端点 friendId 传 roomId;
  talker=发送者wxid、msgId=null→owner:room:createTime:talker 合成稳定主键(重扫幂等)
- 浏览器守护循环:超大群撑崩渲染进程时重启浏览器跳过毒群继续(实战2次自愈);
  200页上限;恢复路径补 await
- base.py SQLite connect_args timeout=60:并发写忙等而非撞锁报错
  (双抓取器并发写实测 database is locked 崩任务)
- 5个新单测,全量410绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:09:32 +08:00
浅念 123c23242c chore: start_backend.sh启动脚本+plan4a/4b历史计划文档 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-19 16:48:22 +08:00
浅念 ab5a918a3b fix(calls+scraper): 语音群scope服务端过滤+对话发现缺口补全+抓取韧性三连修 
- /api/calls 加 scope=friend|group(@chatroom后缀判群,与isRoom实测等价120=120),CallList去掉「全部」tab改服务端查
- 对话发现:全部customer pair ∪ 语音pair(weChatId,talker),未扫过的排最前——修复纯语音关系客户聊天记录永远空(35%语音客户pair从未扫过)
- voice-calls改浏览器直发(isRoom=0/1双趟+WA callSource=4,拦截方式已两次失效)
- 断连恢复重试30×2min(云客约3h一次断连,已实战扛过多次)
- WA messages upsert改整列表直调(_upsert_all逐条传dict被迭代成key字符串→TypeError)
- 405测试绿含scope新增3个

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-19 16:48:22 +08:00
浅念 f9ed7dd821 feat(scraper): 微信消息翻页取完整历史(paginate_chats hasLast/end游标+200页上限,移除临时skip;全量回填3478对话+95361条至2025-10) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-18 09:15:04 +08:00
浅念 aeb8a89b03 fix(scraper): talker_id用发送的friend_id优先(API返回的talker可能脱敏导致消息查不到) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-17 09:29:39 +08:00
浅念 2cdbdf7283 fix(ui): 微信群scope换列(群信息)+WA语音加操作列(播放/长语音) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 16:01:56 +08:00
浅念 bcacf80a7b fix(scraper): 过滤type=-2日期分隔线 + 清理DB已有4062条分隔线 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 15:07:43 +08:00
浅念 9eb747f2b3 feat(scripts): 全量同步改纯页面方式(10步零API零风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:45:50 +08:00
浅念 4a148706b9 feat(scraper): 页面抓取器支持全实体(账号/WA客户/语音/长语音/WA消息,全浏览器方式) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:44:20 +08:00
浅念 7beae8cf55 feat(scripts): 全渠道全量重同步脚本(页面抓取+API全覆盖) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:38:01 +08:00
浅念 2d1943a74e fix(api): 客户/群排序改回last_chat_time DESC(云客按最近沟通优先) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:18:42 +08:00
浅念 854da6e566 fix(api): 客户/群列表排序改为friend_id/room_id ASC(对齐云客API自然顺序) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:14:24 +08:00
浅念 aa90d9d8f2 feat(ui): 列表页移除tab结构(每渠道独立页面,由侧栏菜单驱动) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 13:41:49 +08:00
浅念 9b21584e9a feat(ui): 左侧菜单改分组式(微信管理/WhatsApp管理+下拉子菜单,对齐云客布局) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 12:09:55 +08:00
浅念 02f95af279 fix(ui): 所有列表页默认分页改为10条(原20条) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 11:32:33 +08:00
浅念 81f4e7168f fix(ui): 员工微信号用owner_account_id+添加类型/好友类型加标签映射 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 11:22:20 +08:00
浅念 1d2cf965e2 fix(ui): 员工微信列改为头像+昵称+微信号+手机号(对齐云客复合单元格) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 11:14:10 +08:00
浅念 8a396131c0 feat(ui): 微信客户列表补全16列(手机/描述/地区/签名/标签/员工/时间/添加类型/来源/好友类型) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 11:08:58 +08:00
浅念 d28936cf94 fix(ui): 微信客户列表列布局对齐云客(头像+昵称+微信号复合单元格) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:59:27 +08:00
浅念 1f89eeddb0 feat(api): /api/customers 加 friend_id 筛选(用于查单个客户昵称) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:54:22 +08:00
浅念 661ecfc8c9 feat(chat): 聊天标题显示昵称(客户→昵称,群→群名,替代原始ID) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:54:21 +08:00
浅念 edd6a27371 feat(ui): 所有列表页加CSV导出按钮(全量拉取+中文BOM兼容Excel) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:31:21 +08:00
浅念 9bec4dd7c8 feat(ui): 去掉顶部渠道筛选(与列表tab重复)+标题移到侧栏 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:19:24 +08:00
浅念 719bbeecce feat(ui): 左侧改垂直菜单(替换部门树)+顶部只留标题和渠道筛选 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-14 18:05:39 +08:00
浅念 cf858a00c2 fix(scraper): 消息页面抓取器改用浏览器fetch(可靠)+空数据安全 2026-08-14 15:30:02 +08:00
浅念 c6217ab1a9 feat(scraper): 页面抓取器支持消息(导航聊天窗口拦截msg/chats API) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-14 15:04:21 +08:00
浅念 bead7fdf43 fix(scraper): 页面抓取器支持员工列表翻页(之前只抓第一页10个) 2026-08-14 09:25:10 +08:00
浅念 004e6e4d6b feat(scraper): 页面浏览器抓取器(Playwright拦截SPA API响应,绕过风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:38:42 +08:00
浅念 2154d050e5 fix(media): image_legacy渲染+下载覆盖+WA raw_json URL回退 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:02:42 +08:00
浅念 9e4a5ee536 fix(ui): StaffDetail分页查找+SmsList日期筛选+SyncStatus自动刷新 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 14:09:43 +08:00
浅念 9a953553ec fix(xss+perf): SearchView XSS消除(无v-html)+room_id索引+pool_recycle 
- SearchView.vue: 用 highlightSegments() 返回 {text,hit} 数组替代 v-html+escapeHtml+regex,模板用 <template v-for> + <mark>/<span> 渲染,从结构上消除 XSS 风险(4 处 v-html 全部移除: 消息片段/客户昵称/realName/手机号)
- models.py: Message 新增 ix_message_channel_owner_room_time(channel,owner_account_id,room_id,timestamp),群聊查询过滤 room_id 而非 talker_id,原 talker 索引无法覆盖
- migrations/d8eee96cd017: 新增 room_id 索引的 upgrade/downgrade (offline SQL 已验证)
- base.py + scheduler.py + cli.py: 所有生产 create_async_engine 增加 pool_recycle=3600,在 MySQL wait_timeout(默认 8h)断连前主动回收

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:50:45 +08:00
浅念 c261b2bc9a fix(security): CORS限制+SSRF防护+PATCH乐观锁+accounts安全cap(4 important bugs) 
- main.py + config.py: CORS origins read from Settings.cors_origins (comma-separated).
  Empty falls back to dev origin localhost:5173; never wildcard. Methods locked
  to GET/PATCH/POST; allow_credentials=False.
- scraper/media.py: download_url now passes follow_redirects=False to httpx.
  OSS URLs return 200 directly; a 3xx is treated as a download failure (caught
  by existing except clauses) instead of opening an SSRF vector to internal hosts.
- routers/voice_clips.py: PATCH /voice-clips/{id}/match accepts an optional
  If-Match header carrying the client's last-seen updated_at. On mismatch the
  row was modified by another request and the server returns 409. Without the
  header the behaviour is unchanged (backward compatible). Response now echoes
  the post-update updated_at so the client can fold it into the next request.
- routers/organization.py: /api/accounts now UNION ALLs the wechat_account and
  whatsapp_account projections at the SQL level and pushes COUNT + pagination
  (offset/limit) down to the DB. Stable ordering on (channel, account_id).
  Removes the in-memory materialization that broke at scale.

Tests: 395 passed. Added optimistic-lock coverage (If-Match accepted / stale 409 /
absent header backward-compat). Updated existing PATCH tests for the new
updated_at response field.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:41:30 +08:00
浅念 c7d54a135e fix(tests): 更新风控参数断言匹配新默认值(2.0/2.5/300/5-120/6) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 10:41:46 +08:00
浅念 caef32891e fix(security): 路径穿越防护+导出行数上限+并发同步锁(3 critical/important) 
- media.py: GET /api/media/{hash} 现强制 file 必须位于 Settings.media_dir
  沙箱内(resolve + relative_to),杜绝 local_path 越权读取任意文件。
- export.py: /api/export/chat-html 加 EXPORT_MAX_ROWS=5000 硬上限(.limit()),
  超出时仅渲染前 5000 条并插入 .export-warning 横幅,防止 OOM。
- scheduler.py: _safe_run 增加 SyncRun advisory lock(60min TTL),发现同
  entity 存在 status='running' 且在 TTL 内的记录即返回 skipped SyncResult;
  同时为本轮 stake 一行 wrapping SyncRun,让并发 CLI/scheduler/手动触发
  能相互避让。崩溃场景由 TTL 自动恢复。
- tests: 新增 path-traversal 拒绝/export 截断警告/advisory lock skip+TTL
  恢复 5 个测试;test_api_media 的 fixture 改为 MEDIA_DIR=tmp_path 以适配
  新沙箱语义。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 10:37:13 +08:00
浅念 461e9ff081 tune(anti-risk): 加固防风控策略(间隔2s+抖动2.5s+上限300+退避5-120s×6次) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 09:45:33 +08:00
浅念 5ce33232c1 feat(sync): 聊天图片本地化(message media_hash+download_message_images+local-first rendering) 
- Message 模型新增 media_hash 列 (sha256 FK -> Media.media_hash)
- Alembic 迁移 e5f6a7b8c9d0_message_media_hash (batch_alter_table, SQLite/MySQL 通用)
- app/scraper/media.py:download_message_images 批量下载器，镜像 download_call_recordings
  - 候选: media_hash IS NULL AND channel IN ('wechat','whatsapp') AND type IN ('image','video','file') AND 任一 file_*_url LIKE 'http%'
  - image 类型优先 file_th_url (缩略图，小快省)，video/file 优先 file_url
  - per-row try/except + per-run tried set，失败不挂循环
- sync_message_images orchestrator 入口 + ENTITY_MESSAGE_IMAGES 常量 (提交自身 SyncRun，镜像 call_recordings)
- scheduler: _run_message_images + ENTITY_FUNCS['message_images'] + 加入 daily all job (尾部)
- MessageOut schema 暴露 media_hash 字段
- MessageBubble.vue local-first: media_hash 优先 /api/media/{hash}，否则回退 OSS file_url/file_th_url/file_hd_url
- types.ts: Message 接口添加 media_hash 字段
- 新增 test_message_images_downloader.py (6 tests, 镜像 test_recording_downloader)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 18:24:11 +08:00
浅念 af7f8d71c8 feat(export): 导出 HTML 媒体内嵌(图片/语音/视频/附件, OSS URL 直接嵌入) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 17:22:11 +08:00
浅念 20cf2c51f0 feat(staff): 员工详情独立页(StaffDetail.vue,替代 drawer 降级,对齐云客 userDetailNew) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 17:05:01 +08:00
浅念 0e1cb87156 feat(scraper): WA 群 scraper(businessFriendList+type=1,待风控重置后实测) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 16:04:03 +08:00
浅念 17ee5a1a6c feat(calls): 微信语音 好友/群 scope 筛选(对齐云客,client-side isRoom 过滤) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 15:25:42 +08:00
浅念 0cc9408371 feat(staff): WA员工 modal 列对齐云客(序号/员工信息/员工WhatsApp/客户WhatsApp/添加时间/最后沟通) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 14:58:38 +08:00
浅念 05d1e41d31 fix(staff): 默认统计窗改近30天(覆盖smoke.db数据范围,数字不再全0) 2026-08-11 13:53:55 +08:00
浅念 538a0dc635 feat(sync): 长语音音频本地化(voice_clip media_hash+download+local-first playback) 
Mirror download_call_recordings pattern for voice_clips:
- VoiceClip.media_hash column + Alembic migration (c4d5e6f7a8b9)
- VoiceClipOut.media_hash + frontend VoiceClip.media_hash type
- download_voice_clip_audio in app/scraper/media.py (resilient, sha256-dedup,
  same download_url/upsert_media helpers as call recordings)
- sync_voice_clip_audio orchestrator wrapper (commits own SyncRun, mirrors
  sync_call_recordings semantics)
- scheduler: _run_voice_clip_audio + ENTITY_FUNCS['voice_clip_audio'] +
  appended to daily _run_all job (after voice_clips)
- VoiceClipModal playClip local-first: /api/media/{media_hash} with OSS
  oss_file_name fallback (mirrors CallList.playRecord)

Tested: 376 pytest pass; vue-tsc clean; migration applied; 1523/1533 clips
localized (10 pending are genuine upstream HTTP 404s — resilient skip).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 12:06:26 +08:00
浅念 8b19c299bb feat(calls): WA语音 单元格可点(客户→WA chat,员工→drawer)+ chatChannel helper 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 11:59:56 +08:00
浅念 2b2f863800 feat(calls): VoiceClipModal 录音异常检查(本地元数据标记+仅看异常筛选) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 10:50:49 +08:00
浅念 9da272b992 feat(calls): voice_clip 手动匹配(voice_clip↔customer 本地关联写操作) 
- VoiceClip: 新增 matched_friend_id 列 (String(128), nullable) + alembic 迁移 b3c4d5e6f7a8
- PATCH /api/voice-clips/{id}/match: friend_id 关联 / null 取消 (本地写, 不回写云客)
- GET /api/voice-clips: 批量解析 matched_friend_name (remark>nickname>real>id)
- VoiceClipModal: 已匹配客户列 + 匹配/取消匹配按钮 + 嵌套客户选择 picker (默认按员工列客户, 名搜索走 /search)
- 测试: 17 passed (新增 7 个 match 用例覆盖 set/clear/name 解析/404)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 10:43:34 +08:00