xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
339 Commits 1 Branch 0 Tags
1aadc80d6b0c03ddcf95f41be3a9e82a95736bb9
Commit Graph

169 Commits

Author SHA1 Message Date
浅念 b6eea3070a feat(staff): 弹框客户微信/员工微信列改富单元格(头像+昵称+ID+真实姓名,对齐云客) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 10:24:07 +08:00
浅念 31d22ddd45 fix(media): 修复 Windows 反斜杠路径致 Linux 容器 404(49536 行 local_path 全量改为 /data/media 前缀) 
SQLite 迁移时保留了 Windows 路径(media\xx\),Linux 后端 Path.resolve()
失败 → 404。一次性修复全部 media 行 + API 恢复 200。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 15:16:18 +08:00
浅念 5effaac00b fix(scraper): 补_room_watermarks函数定义(增量同步时引用但定义在编辑中丢失,导致群消息扫描崩溃) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 14:15:37 +08:00
浅念 8a99940554 feat(groups): chat_group改复合主键(channel,owner,room)——多员工同群不再去重(云客张士蓓64群vs原42) 
原 room_id 单列 PK 把多员工同群合并成一行(末位覆盖owner),其他员工
群列表缺数据。改复合主键后同群每个员工各一行,对齐云客按员工查看。
重扫后 1590 行/865 去重群/张士蓓 64 群(与云客一致)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 12:01:25 +08:00
浅念 804addf725 feat(staff): 员工列表排序对齐云客API返回顺序(list_order列+抓取记序+API按序排) 
- wechat_account 加 list_order 列(云客 wxUser/list 返回顺序)
- 抓取器记录每行在API响应中的序号
- /api/accounts ORDER BY list_order(对齐云客员工列表排列)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 11:25:33 +08:00
浅念 abc97fe217 feat(search): MySQL ngram FULLTEXT分支(LIKE 13.5s→索引毫秒级;索引缺失自动回退LIKE) 
MySQL LIKE 全扫698k行13.5s撞axios 15s超时→前端ERR_ABORTED;
MATCH...AGAINST(ngram)毫秒级;本地C盘不足索引待服务器建
(索引SQL: CREATE FULLTEXT INDEX ft_msg_text ON message(text) WITH PARSER ngram)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 17:52:26 +08:00
浅念 a8b8d9d81c fix(export+calls): 同类owner错位问题全面修复(群导出不按owner过滤+群语音客户点击走群聊路由) 
- /api/export/chat-html: room 查询忽略 owner(chat_group owner 错位同
  ChatTimeline,群列表/聊天页的导出按钮原来会导出空文件)
- CallList viewCallCustomer: talker 以 @chatroom 结尾→group-chat 路由
  (原走 1:1 路由按 talker_id 查@chatroom 恒空)
- TDD: 群消息在 ownerB 名下用 ownerA+room 导出仍能拿到;428 测试绿
- 实测:错位 owner 导出 43304 群返回 15002 个消息元素

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 17:00:43 +08:00
浅念 a290e39523 feat(customers): 发送/接收列改今日口径(对齐云客;本地消息表Asia/Shanghai今日聚合) 
- CustomerOut 加 send_today/receive_today(路由赋值,非ORM列)
- /api/customers 每页一次 GROUP BY(owner IN 走复合索引前缀)
- 前端列优先今日值,无消息数据回退原raw字段;列名对齐云客「发送条数/接收条数」
- TDD:今日3发2收+昨日不计入+无消息客户0

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 14:39:17 +08:00
浅念 00ae13b5c2 feat(deploy): 完成数据迁移与MySQL全链路(14表698k消息100%一致+5方言补丁+bind mount) 
- migrate_sqlite_to_mysql.py: 逐表清空重导(幂等),复用repos强转,JSON列
  解析,67万消息按msg_id键集分批,467s跑完14表全一致
- 修4个MySQL实测bug: NULLS LAST语法(compat.py编译补丁)、message.text
  TEXT→MEDIUMTEXT(64KB字节上限,实测20.8万字符车源清单)、URL列
  String(1024)→Text(分享链接超1KB)、垃圾超长member_id源头清理
- compose: media改bind mount(不占C盘Docker虚拟磁盘,实测2.5GB卷把
  C盘撑爆引擎500);mysql不映射宿主机3306
- 前端Dockerfile: npm ci→install(本地/容器npm版本锁漂移)
- 五服务实测:nginx→backend→MySQL全链路返回真实数据(客户6859/A**杰25条
  /搜索56条),媒体49536文件bind mount直读

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 12:40:44 +08:00
浅念 2175cb0169 feat(deploy): MySQL容器实测打通(3个方言bug修复)+部署文档 
- repos: on_duplicate只更新实际插入列(fetched_at等Python-default列,
  SQLite excluded容忍/MySQL new.*直接报Unknown column)
- repos: _coerce_int增加日期字符串→ms epoch分支 + _INT_FIELDS补
  last_update/last_chat_time/timestamp/update_time(字符串直通BigInteger)
- models: wechat_account.db_size Integer→BigInteger(实测2.17GB超32位上限)
- compose: mysql不映射宿主机3306(本机已占;容器内服务名互联)
- docs/DEPLOY.md: 服务器部署全指南(架构/步骤/数据迁移/运维命令/注意事项)
- 容器实测:Playwright抓取55账号→MySQL upsert全通

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 10:40:04 +08:00
浅念 3dc80805fa feat(deploy): Docker镜像支持页面抓取(Playwright+Chromium+scripts/)+full_sync可移植 
- Dockerfile:Chromium headless 系统依赖 + playwright install chromium
  + COPY scripts/(调度器 _page_subprocess 的入口,之前镜像里没有)
- full_sync_all.sh:DB_URL 外部可覆盖(容器 MySQL)、python 路径可移植
  (容器 PATH / 本地 venv 双兼容)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 10:15:58 +08:00
浅念 94c2df4612 test(scheduler): 更新messages job断言(四类消息命令序列) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:17:40 +08:00
浅念 d4bc3f5476 feat(scheduler): 4h增量接入全部四类消息(wa-messages/group-messages/wa-group-messages,水位后单对话1页可承受) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:15:24 +08:00
浅念 ef2df073a8 feat(ops): 服务健康检查脚本(检测+拉起后端/前端/调度器) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:10:20 +08:00
浅念 e8c77412fa feat(search): FTS5 trigram全文索引(中文搜索3.95s→0.003s,1300倍) 
- message_fts 外部内容虚拟表+trigram分词(中文子串可MATCH,无需分词库)
- 3个触发器随ORM upsert自动同步索引(抓取器零改动)
- 搜索路由:≥3字走FTS MATCH,<3字/MySQL回退LIKE
- 实测:搜「大众途安」56条,LIKE全扫3.95s→FTS 0.003s;索引+300MB
- 7个新测试(建表幂等/触发器同步/中文匹配/短查询回退/未建索引回退),426绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:08:26 +08:00
浅念 621c2cc929 feat(scheduler): 每周日05:00全量对齐job(四消息实体--full-history,修撤回/编辑+消除滞后窗口) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 13:52:59 +08:00
浅念 0a161c7b6a fix(sync): 增量跳过条件去掉+60s缓冲(会造成永久漏消息) 
若新消息恰好落在水位后60s内,云客act永远≤wm+60s→每轮跳过→消息永久漏。
act==wm(云客最新==我们最新,同源ms时间戳精确相等)才跳,消除该窗口。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 13:43:42 +08:00
浅念 9bca5fc08a feat(sync): 消息全面增量同步(水位+跳过+触底停,4h周期从小时级降到分钟级) 
- 每对话水位=本地max(timestamp);请求带start=水位只拉新消息
- 云客last_chat_time≤水位的对话整跳过(0请求)——实测7046对话3150被跳过(45%)
- 翻页遇≤水位消息即停(已同步区),页数从几十页降到1页
- 微信1:1/WA 1:1/微信群/WA群四个抓取器全接入;CLI --full-history强制全量
- 实测:每对话1页(~3s),3895待访对话约3.5h(含补积压的首次全量);
  稳态后每4h只访有新活动的对话(几十~几百个,15-30min)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 12:20:21 +08:00
浅念 10baa22acb fix(media): tried集合分片进NOT IN(3万+msg_id超SQLite变量上限,42k后崩;每晚全量媒体步同bug) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 11:26:50 +08:00
浅念 d41eb2431f fix(wa): WA消息file_url优先直连OSS的file字段(1:1图片93%有URL)+存量回填7373行(带URL 6028/6804) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 15:40:50 +08:00
浅念 ff73c056e9 feat(chat): 聊天排版升级微信标准(左右分侧+头像+绿气泡)+is_send数据修复 
- is_send 根因:云客消息方向在 mine 字段(isSend 全 null),归一化加 fallback
  +存量回填 60.7 万行(51万收/16万发)——分侧数据首次完整
- MessageBubble:头像(对方=raw headUrl 在左/自己=默认在右)、自己侧隐藏昵称、
  接收白底描边/发送微信绿 #95ec69、气泡列 max-width 78%
- 实测混合对话:13发(绿+头像右)/7收(白+头像左)全部正确

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 15:26:04 +08:00
浅念 ad0bfeba16 fix(messages): 微信图片type=2映射漏掉(存成unknown前端不渲染)——补image映射+存量回填49504行 
真机A**杰对话发现:新版云客图片走type=2且fileTh带完整OSS URL(旧映射只认
3=image_legacy)。存量回填:type_raw=2→image(49504行,其中47126带URL可渲染/
本地化)。A**杰对话实测图片正常显示(OSS直连渲染,本地化批后台进行中)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 15:16:26 +08:00
浅念 8a8c91a6a3 fix(customers): 客户列表排序实时对齐云客(全局实时body刷新last_chat_time,前5位逐位一致) 
- scrape_friends 双阶段:逐员工全量保底 + 全局实时body(friendTab=all/isOnJob/ymd窗)
  刷 last_chat_time(该参数集跨员工按最近沟通实时排序,实测比**赐14:10实时值)
- 阶段2只刷last_chat_time且只增不减(全局结果含旧ES副本,upsert全字段会被后写覆盖)
- 调度4h增量改为 friends→wa-friends→messages(排序键每4h实时化)
- 实测我们top5=云客top5逐位一致(A**杰14:13/比**赐14:10/P**6/A**邱/J**n)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:46:02 +08:00
浅念 edca9b4c77 fix(customers): WA客户破1万上限(逐员工uid过滤,8585→11931全量)+双渠道页脚云客口径+消息同步刷新最近沟通时间 
- wa-friends 逐员工抓取:全局查询~1万行硬上限(云客11931只翻到10000),
  uid=员工WA号过滤(真机验证 declared 1145→1149),total-aware翻页
- declared-total 分渠道:Σ wechat/whatsapp_account.contacts_num(8590/11915)
- 消息增量同步顺带刷新 customer.last_chat_time(云客列表按最近沟通实时
  排序,快照每日一次导致顺序滞后;微信1:1+WA都加)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:04:06 +08:00
浅念 fd3cf0003b feat(customers): 微信客户页脚改云客口径(Σ员工好友数声明值8590,用户选B方案) 
- GET /api/customers/declared-total: Σ wechat_account.contacts_num
- 页脚优先显示云客口径(回退真实total);列表内容保持去重真实行
- 云客列表声明值含其ES索引重复doc(连行id都相同无法物理区分),仅对齐计数

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 13:44:39 +08:00
浅念 0d04f68c07 fix(scraper): 微信客户抓取三连修(wechatId过滤+完整body+total-aware翻页,6563→6846全量) 
- 真机抓包确认过滤字段是 wechatId(原 userId 被无视,每员工拉回部门前1000行)
- 补 ymdStart/ymdEnd/friendTypeYunke=1/isDel=0 完整 body(与云客页面一致)
- total-aware 翻页:端点分页中途掉短页,翻到声明数满或连续空页(重复行幂等)
- 验证:csf 员工声明1999→distinct1846全量到手;云客total含跨页重复水分(1999声明vs1846真实)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 12:32:03 +08:00
浅念 5d22d2b5f5 fix(login): pcLogin协议真机修正(phone/loginType/companyCode)+secureKey换usercenter端点 
浏览器实登抓包确认:POST /usercenter/login/getSecureKey(非newuc GET) +
body {phone,password(RSA-hex),loginType:'yunkecn',companyCode}
实测从500系统错误推进到10300密码错误=协议通,剩加密细节/密码核对(余5次重试暂停试错防锁号)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 11:40:16 +08:00
浅念 4aeed003a4 feat(calls): 微信语音列表加今日统计卡片+通话时间范围筛选+客户跳转带通话时间定位(对齐云客) 
- GET /api/calls/stats:今日(Asia/Shanghai)好友呼入/外呼量+群量+时长聚合(@chatroom判群)
- CallList:5张统计卡片(仅微信语音tab)+datetimerange筛选(服务端start/end)+viewCallCustomer带?start定位
- TDD新增stats聚合测试,全量绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 11:08:13 +08:00
浅念 6c9245a682 fix(groups): /api/groups加room_id精确直查(群聊页群名解析,修page_size=200超限422) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:58:39 +08:00
浅念 1d90829754 feat(scraper): WA群消息抓取(chatRecords friendUid=@g.us,chatRoomTalker=群内发送者) 
- 真机探明:同 chatRecords 端点 friendUid 直接传 @g.us 群id(与微信群同套路)
- 群行 chatRoomTalker=发送者@lid、keyId 有值(缺失时合成稳定主键)
- file 字段直连 OSS URL 优先于 mediaName(WA群媒体 URL 现成!)
- 守护循环同微信群;full_sync 加 9b 步;3 新测试,416 绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:54:09 +08:00
浅念 b522be828a feat(scraper): 五实体导航拦截全改浏览器fetch(登录墙杀死拦截方式0捕获) 
- friends/groups/friend_list/es 部门级全量(记录自带owner,告别逐员工遍历)
- accounts/wxUser/list form模式(页码字段是page非pageNum,粘页检测保险)
- wa-accounts/wa-friends 直发;wa-friends 200页1万行(疑端点1万行硬上限)
- _fetch_pages 通用翻页助手:cookie失效首页code告警+粘页检测
- 实测:微信客户5916→6563/WA客户8058→8585/群851→873/账号50→55(捞回登录墙期间漏抓)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:41:33 +08:00
浅念 10cdd81d5c fix(scraper): 消息归一化补fileTh/fileHd→列映射(图片缩略图URL进列,媒体下载器可及) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:46:54 +08:00
浅念 5923cd05b1 feat(scheduler): 定时同步全面页面化(子进程跑页面脚本,API直调退出调度防风控) 
- 每日 full: bash scripts/full_sync_all.sh;每4h messages: scrape_via_page --entity messages
- 子进程隔离+asyncio.Lock互斥+非零退出只记JSON日志;自动补DB_URL环境
- ENTITY_FUNCS/run_once/CLI保留为手动调试,不再挂定时触发器
- README同步更新(Docker镜像需Playwright+Chromium);413测试绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:33:53 +08:00
浅念 02dd77a62f feat(scraper): WA群抓取(type=1真机验证,223群)+群成员派生(12840成员/750群) 
- wa-groups 实体:businessFriendList type=1 真机验证返回群,行结构是好友字段名
  (friendUid=@g.us 群ID/nickName=群名),专用 _normalize_wa_group_row
- backfill_group_members.py:无独立成员API,从群消息派生(talker=成员,sender_name=昵称)
- full_sync_all.sh:+4步(5b WA群/语音后录音重链接/8b 群消息+成员/消息上限99999)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:21:20 +08:00
浅念 f92aac1a3d feat(scraper): 微信群消息抓取(368778条/750群)+浏览器守护循环+SQLite忙等 
- 新 group-messages 实体:真机探明群消息=同一 msg/chats 端点 friendId 传 roomId;
  talker=发送者wxid、msgId=null→owner:room:createTime:talker 合成稳定主键(重扫幂等)
- 浏览器守护循环:超大群撑崩渲染进程时重启浏览器跳过毒群继续(实战2次自愈);
  200页上限;恢复路径补 await
- base.py SQLite connect_args timeout=60:并发写忙等而非撞锁报错
  (双抓取器并发写实测 database is locked 崩任务)
- 5个新单测,全量410绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:09:32 +08:00
浅念 123c23242c chore: start_backend.sh启动脚本+plan4a/4b历史计划文档 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-19 16:48:22 +08:00
浅念 ab5a918a3b fix(calls+scraper): 语音群scope服务端过滤+对话发现缺口补全+抓取韧性三连修 
- /api/calls 加 scope=friend|group(@chatroom后缀判群,与isRoom实测等价120=120),CallList去掉「全部」tab改服务端查
- 对话发现:全部customer pair ∪ 语音pair(weChatId,talker),未扫过的排最前——修复纯语音关系客户聊天记录永远空(35%语音客户pair从未扫过)
- voice-calls改浏览器直发(isRoom=0/1双趟+WA callSource=4,拦截方式已两次失效)
- 断连恢复重试30×2min(云客约3h一次断连,已实战扛过多次)
- WA messages upsert改整列表直调(_upsert_all逐条传dict被迭代成key字符串→TypeError)
- 405测试绿含scope新增3个

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-19 16:48:22 +08:00
浅念 f9ed7dd821 feat(scraper): 微信消息翻页取完整历史(paginate_chats hasLast/end游标+200页上限,移除临时skip;全量回填3478对话+95361条至2025-10) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-18 09:15:04 +08:00
浅念 aeb8a89b03 fix(scraper): talker_id用发送的friend_id优先(API返回的talker可能脱敏导致消息查不到) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-17 09:29:39 +08:00
浅念 bcacf80a7b fix(scraper): 过滤type=-2日期分隔线 + 清理DB已有4062条分隔线 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 15:07:43 +08:00
浅念 9eb747f2b3 feat(scripts): 全量同步改纯页面方式(10步零API零风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:45:50 +08:00
浅念 4a148706b9 feat(scraper): 页面抓取器支持全实体(账号/WA客户/语音/长语音/WA消息,全浏览器方式) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:44:20 +08:00
浅念 7beae8cf55 feat(scripts): 全渠道全量重同步脚本(页面抓取+API全覆盖) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:38:01 +08:00
浅念 2d1943a74e fix(api): 客户/群排序改回last_chat_time DESC(云客按最近沟通优先) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:18:42 +08:00
浅念 854da6e566 fix(api): 客户/群列表排序改为friend_id/room_id ASC(对齐云客API自然顺序) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:14:24 +08:00
浅念 1f89eeddb0 feat(api): /api/customers 加 friend_id 筛选(用于查单个客户昵称) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:54:22 +08:00
浅念 cf858a00c2 fix(scraper): 消息页面抓取器改用浏览器fetch(可靠)+空数据安全 2026-08-14 15:30:02 +08:00
浅念 c6217ab1a9 feat(scraper): 页面抓取器支持消息(导航聊天窗口拦截msg/chats API) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-14 15:04:21 +08:00
浅念 bead7fdf43 fix(scraper): 页面抓取器支持员工列表翻页(之前只抓第一页10个) 2026-08-14 09:25:10 +08:00
浅念 004e6e4d6b feat(scraper): 页面浏览器抓取器(Playwright拦截SPA API响应,绕过风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:38:42 +08:00