xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
389 Commits 1 Branch 0 Tags
9fbd572bb77ccdca82ce0953fef4e3e269cd31d2
Commit Graph

49 Commits

Author SHA1 Message Date
浅念 b0ebe8a15a feat(group-stats): 微信群列表对齐云客「微信群统计」——getChatroomList 数据源+窗内统计 
用户报微信群列表数据/内容不对,应对齐云客「沟通→微信号管理→微信群统计」
(/cms/wechat/wechatgroup)。真机探明(表单登录+页面导航抓 network):

- 数据源=POST /wechatweb/chatroom/getChatroomList(form:departmentId+
  startTime/endTime 窗+分页;信封 data.overviews[]/totalCount 1,854 行)
- 云客窗是任意时间窗实时算:沟通消息数实测同群全史 417/7天 36/单日 6
- **修正存量映射错误**:member_count 原存 friendNum(员工好友数),真群成员
  数是 friendCount/getChatroomList.memberCount
- 新增/退出成员数云客按窗实时算,本地无事件流 → 按天积累

实现:
- ChatGroup 补 group_remark 列(备注仅 friend_list/es 携带,两归一化均
  「有值才写」防互清);迁移含存量 raw_json 回填(须先于基线抓取执行)
- 新表 chat_group_member_stats(渠道×账号×群×天一行 added/lost)
- 新实体 chatroom-stats:全窗刷档案列(真成员数/群备注/员工信息,owner
  经 wechat_account alias 解析)+ --day 单日窗落进出日行;full_sync 5c 步
- /api/groups 扩展:msg_start/msg_end 窗内聚合(window_msg_count=
  msg_daily_stats 按天和、window_added/lost=日表和)、last_start/end
  最新消息窗、group_remark 子串;行补员工微信/员工信息关联列
- GroupList 微信 tab:统计时段(默认近7天)/最新消息时间段/群名称/群备注/
  员工下拉/查询 + 9 列(群信息|群备注|员工微信|员工信息|群成员数量|
  新增成员数|退出成员数|沟通消息数|操作);WA tab 不动

700 测试绿(新增窗聚合/筛选/员工列 3 例)。本地实测:基线 1,854 群=云客
total 一字不差,今日日窗 added 407/lost 166 落表,页面 9 列/员工列渲染正常。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-09 18:09:20 +08:00
浅念 d42895337d feat(link): customer_account_link 账号↔客户多对多——共享好友员工维度对齐云客 
用户报「Lily 好友数 36,点进员工客户页只有 12」。真机探明根因:那 24 个
是共享客户(同一人加多个员工),云客 per-employee 各算一个;而 customer
表 PK=friend_id 全局单行、归属 last-wins,其他员工名下必然「消失」。
抓取器本身每员工都拿全了(日志 declared=36 rows=36),丢在入库模型层。

- 新表 customer_account_link(渠道×账号×客户一行,per-relation 快照:
  昵称/备注/头像/手机号/收发计数/最近沟通/add_time/fromType/isDel)
- 写:repos.upsert_customer 随档案同事务维护(所有好友抓取路径一处挂全);
  新增 prune_customer_links 对账
- 读:/api/customers?owner_account_id= 改读 link(与云客 contacts_num 口径
  一致);该账号 link 为空时回退档案路径(过渡安全);与 dept/user 等
  过滤组合时维持旧行为;CustomerOut 透传 from_type
- 抓取器:friends/wa-friends 逐员工整批 upsert 后按名单对账(取满声明
  total 才删,防半份名单误删)
- 前端:StaffFriends 数据来源列优先 link 的 from_type

611 测试绿(link 新 6 例);本地验证空表回退/造数接管/from_type 透传。
上线后需重跑 friends+wa-friends 回填 link(存量关系无法从档案反推)。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-08 17:17:06 +08:00
浅念 8499a74a72 fix(analytics): run_full水位前置防回填窗口漏扫+README/DEPLOY/脚本号收尾 
Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-04 23:27:41 +08:00
浅念 6b99202e9b feat(analytics): 4h 调度分析链+CLI 回填+full_sync 夜间自愈步 
- scheduler: yunke_analytics_tick job(与消息增量同节奏 IntervalTrigger),
  _analytics_tick 纯 DB——stat_daily_staff 滚动3天刷新 + 敏感词增量扫描,
  不抢 _page_lock 无子进程,异常只记日志不断链;today 用 time.time()
  UTC epoch 喂 day_key,规避服务器非上海时区的 naive now 误判
- cli: staff-daily-stats(全量重建 rollup,幂等)+ sensitive-scan
  (默认增量水位扫描,--full 首跑建水位/新词回填历史;description 明示
  加词不回填历史)+ 幂等补种起步词库
- full_sync_all.sh: daily-stats 后追加 staff-daily-stats(rollup 随
  msg_daily_stats 重建同步自愈)
- 测试: tick 行为测试(今日 seed→stat_daily_staff 出行)+ 异常吞噬
  (monkeypatch 抛错不外抛);同步更新注册断言(3→4 jobs)

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-04 21:02:41 +08:00
浅念 5cfc09121a feat(scraper): cookie 过期自动续期——浏览器表单重登+缓存+整实体重跑 
撞 CookieExpiredError → PageScraper 表单登录(密码登录tab+placeholder选择器,
2026-09-03 实测一次成功)→ 新 cookie 落 MEDIA_DIR/.yk_cookie_cache(优先于
compose 注入的 env,永远比它新)→ 实体重跑一次(幂等+水位,安全);二次过期
直接抛出防循环;重登失败(无账密/云客改版)报可读原因退出码 3。

main 瘦身:cookie 加载收敛 _load_cookie(缓存>env>settings),分发链移
_dispatch_entity。测试 8 新增,全量 525 绿。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-03 18:11:26 +08:00
浅念 e365108244 fix(scraper): 云客鉴权失败信封统一拦截——cookie 过期响亮报错不再静默 
事故(2026-09-03 手动同步实测):cookie 过期时 WA 路径对 str data 调 .get
崩 AttributeError,微信路径更糟——auth 错误信封被当空数据,55 员工全部
rows=0 无任何告警,整轮同步一无所获。

守卫:_browser_fetch 收口统一走 _check_auth_envelope,识别 success=false
且 code=±20020(实测信封)抛 CookieExpiredError;main 入口捕获后输出
一句话可读信息 + 退出码 3(不再天书堆栈)。业务错误(-90000 等)不受影响
照旧放行。测试 5 新增,全量 517 绿。

后续(等确认真实密码):CookieExpiredError 触发自动重新登录换 cookie,
过期彻底自愈。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-03 17:52:09 +08:00
浅念 421998be54 perf(api): 会话/5数字弹窗超时修复——日粒度汇总层 msg_daily_stats 
根因(服务器实测):sessions 对 owner 全量 message GROUP BY,重账号 5.2万条
3~5s;staff-chat-stats 全时段窗 6.9s——128MB buffer pool 下被推过前端
15s 超时(ECONNABORTED)。EXPLAIN ANALYZE 证实最优 covering 索引也要
18.9s,瓶颈是磁盘不是查询形状。

修复(预聚合,云客同款思路):
- msg_daily_stats 表:每「渠道×账号×会话×上海自然日」一行(收发计数/
  当日最后时间与方向);迁移 a7d3e9f1c2b4(幂等建表)
- 写路径:repos.upsert_messages 落库自动重算本批触及的会话×天(所有
  抓取路径汇到该函数,一处挂全;失败不阻塞写入)
- 读路径:sessions/staff-chat-stats 表有行时读汇总(namedtuple 同形,
  端点主体零改动),空表守卫回退旧聚合(部署回填前不空白)
- 自愈:CLI daily-stats 全量重建(幂等);full_sync_all.sh 夜间追加第 11
  步——顺带修掉尾部 sqlite3 直连 smoke.db 的硬编码(服务器 MySQL 上会炸)
- 口径统一:day 一律 Asia/Shanghai Python 计算(旧 from_unixtime 跟随
  DB 会话时区与今日数割裂);统计窗口对齐自然日(文档化语义变化)

测试:12 新增(汇总层 8 + 端点判别/对拍 4,判别手法=回填后删光 message
仍出完整数据),全量 512 绿。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-03 16:37:36 +08:00
浅念 b2ccbee1ab fix(scripts): backfill_group_members 窗口函数化过 MySQL only_full_group_by(单库收敛) 
max(timestamp)+裸 sender_name 的 GROUP BY 是 SQLite 方言行为,MySQL 8 默认
only_full_group_by 直接 1055 拒绝——2026-08-27 单库收敛(本地+调度器同连
Docker MySQL)时在 MySQL 上实跑触发。改 row_number() OVER 取每 (room_id,
talker_id) 最新一行的 sender_name + count() OVER 计数:两库通用,且语义从
「碰巧取 max 行」变为严格最新一条。另 gitignore 补 *.db.retired
(smoke.db 退役改名后 *.db 规则罩不住)。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-27 14:51:05 +08:00
浅念 35bfebd3e3 chore(backend): list_order迁移补齐+Task14类型修正落链+moments每页日志+MsgTypeParam派生+3处文档修正+Dockerfile打包migrations+DEPLOY补迁移步骤 
- 9b0d0c133035: wechat_account.list_order 进迁移链(此前仅模型声明,
  干净库 /api/accounts 即 500);sa.inspect 幂等守卫兼容已手动加列的
  smoke.db
- fe63c6789e94(验证中发现的预存缺口,超出原清单): Task 14 的 5 列模型
  类型修正(4×URL String(1024)→Text, db_size Integer→BigInteger)从未落
  链——纯 alembic 新库会复活 >1KB 链接截断;对齐后干净库 alembic check
  全绿。SQLite message 表 batch 重建会掉 FTS 触发器+rowid 脱节(docstring
  记明 ensure_fts+rebuild_fts 修复,smoke.db 已修复验证)
- scrape_via_page: _iter_moments_pages 每页一行进度日志(镜像 _fetch_pages)
- messages: MsgTypeParam=Literal[tuple(_MSG_TYPE_FILTER)] 派生,杜绝
  双处罗列漂移(openapi enum 验证一致,492 绿含 422 行为)
- 文档: ChatGroup 复合PK docstring/groups room_id 多行表述/测试名去
  单行假设;Dockerfile 补 alembic.ini+migrations;DEPLOY 增「数据库
  迁移」节(新库/旧库stamp路径/回填脚本显式DB_URL)+运维表升级行

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 20:32:25 +08:00
浅念 64577a1b02 feat(msg): 类型归一化扩展(spike判别+49拆红包/转账)+存量回填+API msg_type/q过滤 
- _normalize_msg_type 升级 (raw_type, row): messageType 经典码优先 →
  49/appmsg 行级拆分(红包/转账/聊天记录) → 云客码兜底
- spike 修正: 云客 type=3 实为语音(messageType=34, 8754/8955 行),
  原 image_legacy 是误标; type_raw=3 的 201 行旧数据才是真图片
- 新映射: 5红包 6/7通话 9文件 10/18/20链接 11转账 13名片 14位置
  15/10000系统 21引用回复→text 22/19 证据不足留 unknown
- 回填 smoke.db: 620279 行重扫, 94268 行修正, unknown 87198→1885, 幂等
- GET /api/messages 加 msg_type(Literal 422)+q(LIKE 转义)参数
- 新增 test_scrape_msg_type.py(11) + test_api_messages 追加(8), 全量 466 绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 15:17:21 +08:00
浅念 6ff288f1d3 feat(groups): chat_group 补 6 群主/建群列 + 复合PK迁移修复 + 存量回填 
对齐云客员工聊天窗口·微信群 tab(2026-08-25 真机验证 raw_json 键):
- ChatGroup 加列 owner_name/owner_wxid/owner_alias/owner_avatar/
  friend_count/create_time(全 nullable)
- scraper _normalize_chat_group 同名提取;createTime 兼容 ms-int 与
  'YYYY-MM-DD HH:mm:ss'(smoke.db 实测 1105 行全为后者),复用
  _parse_add_time 归一 ms-epoch
- 迁移 099ac2097fa2:加 6 列 + 修复 Task1 评审遗留 drift——PK 由单列
  room_id 变更为复合 (room_id, channel, owner_account_id) 且
  owner_account_id NOT NULL(对齐模型 8a99940;smoke.db 1105 行无 NULL,
  未兜底编造数据);SQLite 走 batch 重建(copy_from 显式旧结构),
  MySQL 渲染为直接 ALTER(离线 --sql 已验证)
- 回填脚本 scripts/backfill_chat_group_fields.py(幂等,逐列只补空);
  smoke.db 实跑 backfilled 1105/1105,重跑 0/1105
- GroupOut 加 6 字段(model_validate 自动输出,路由无改)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 14:32:12 +08:00
浅念 90c28f4c7b feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media) 
- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+
  _iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/
  信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account,
  form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments
- media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无
  回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit)
  接入scrape_media第4源
- full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数
- scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页
  由每日full承担);测试断言7条命令序列
- tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核
  (两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化
  (一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 14:07:39 +08:00
浅念 31d22ddd45 fix(media): 修复 Windows 反斜杠路径致 Linux 容器 404(49536 行 local_path 全量改为 /data/media 前缀) 
SQLite 迁移时保留了 Windows 路径(media\xx\),Linux 后端 Path.resolve()
失败 → 404。一次性修复全部 media 行 + API 恢复 200。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 15:16:18 +08:00
浅念 5effaac00b fix(scraper): 补_room_watermarks函数定义(增量同步时引用但定义在编辑中丢失,导致群消息扫描崩溃) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 14:15:37 +08:00
浅念 804addf725 feat(staff): 员工列表排序对齐云客API返回顺序(list_order列+抓取记序+API按序排) 
- wechat_account 加 list_order 列(云客 wxUser/list 返回顺序)
- 抓取器记录每行在API响应中的序号
- /api/accounts ORDER BY list_order(对齐云客员工列表排列)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 11:25:33 +08:00
浅念 00ae13b5c2 feat(deploy): 完成数据迁移与MySQL全链路(14表698k消息100%一致+5方言补丁+bind mount) 
- migrate_sqlite_to_mysql.py: 逐表清空重导(幂等),复用repos强转,JSON列
  解析,67万消息按msg_id键集分批,467s跑完14表全一致
- 修4个MySQL实测bug: NULLS LAST语法(compat.py编译补丁)、message.text
  TEXT→MEDIUMTEXT(64KB字节上限,实测20.8万字符车源清单)、URL列
  String(1024)→Text(分享链接超1KB)、垃圾超长member_id源头清理
- compose: media改bind mount(不占C盘Docker虚拟磁盘,实测2.5GB卷把
  C盘撑爆引擎500);mysql不映射宿主机3306
- 前端Dockerfile: npm ci→install(本地/容器npm版本锁漂移)
- 五服务实测:nginx→backend→MySQL全链路返回真实数据(客户6859/A**杰25条
  /搜索56条),媒体49536文件bind mount直读

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 12:40:44 +08:00
浅念 3dc80805fa feat(deploy): Docker镜像支持页面抓取(Playwright+Chromium+scripts/)+full_sync可移植 
- Dockerfile:Chromium headless 系统依赖 + playwright install chromium
  + COPY scripts/(调度器 _page_subprocess 的入口,之前镜像里没有)
- full_sync_all.sh:DB_URL 外部可覆盖(容器 MySQL)、python 路径可移植
  (容器 PATH / 本地 venv 双兼容)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 10:15:58 +08:00
浅念 ef2df073a8 feat(ops): 服务健康检查脚本(检测+拉起后端/前端/调度器) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:10:20 +08:00
浅念 0a161c7b6a fix(sync): 增量跳过条件去掉+60s缓冲(会造成永久漏消息) 
若新消息恰好落在水位后60s内,云客act永远≤wm+60s→每轮跳过→消息永久漏。
act==wm(云客最新==我们最新,同源ms时间戳精确相等)才跳,消除该窗口。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 13:43:42 +08:00
浅念 9bca5fc08a feat(sync): 消息全面增量同步(水位+跳过+触底停,4h周期从小时级降到分钟级) 
- 每对话水位=本地max(timestamp);请求带start=水位只拉新消息
- 云客last_chat_time≤水位的对话整跳过(0请求)——实测7046对话3150被跳过(45%)
- 翻页遇≤水位消息即停(已同步区),页数从几十页降到1页
- 微信1:1/WA 1:1/微信群/WA群四个抓取器全接入;CLI --full-history强制全量
- 实测:每对话1页(~3s),3895待访对话约3.5h(含补积压的首次全量);
  稳态后每4h只访有新活动的对话(几十~几百个,15-30min)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 12:20:21 +08:00
浅念 ff73c056e9 feat(chat): 聊天排版升级微信标准(左右分侧+头像+绿气泡)+is_send数据修复 
- is_send 根因:云客消息方向在 mine 字段(isSend 全 null),归一化加 fallback
  +存量回填 60.7 万行(51万收/16万发)——分侧数据首次完整
- MessageBubble:头像(对方=raw headUrl 在左/自己=默认在右)、自己侧隐藏昵称、
  接收白底描边/发送微信绿 #95ec69、气泡列 max-width 78%
- 实测混合对话:13发(绿+头像右)/7收(白+头像左)全部正确

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 15:26:04 +08:00
浅念 ad0bfeba16 fix(messages): 微信图片type=2映射漏掉(存成unknown前端不渲染)——补image映射+存量回填49504行 
真机A**杰对话发现:新版云客图片走type=2且fileTh带完整OSS URL(旧映射只认
3=image_legacy)。存量回填:type_raw=2→image(49504行,其中47126带URL可渲染/
本地化)。A**杰对话实测图片正常显示(OSS直连渲染,本地化批后台进行中)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 15:16:26 +08:00
浅念 8a8c91a6a3 fix(customers): 客户列表排序实时对齐云客(全局实时body刷新last_chat_time,前5位逐位一致) 
- scrape_friends 双阶段:逐员工全量保底 + 全局实时body(friendTab=all/isOnJob/ymd窗)
  刷 last_chat_time(该参数集跨员工按最近沟通实时排序,实测比**赐14:10实时值)
- 阶段2只刷last_chat_time且只增不减(全局结果含旧ES副本,upsert全字段会被后写覆盖)
- 调度4h增量改为 friends→wa-friends→messages(排序键每4h实时化)
- 实测我们top5=云客top5逐位一致(A**杰14:13/比**赐14:10/P**6/A**邱/J**n)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:46:02 +08:00
浅念 edca9b4c77 fix(customers): WA客户破1万上限(逐员工uid过滤,8585→11931全量)+双渠道页脚云客口径+消息同步刷新最近沟通时间 
- wa-friends 逐员工抓取:全局查询~1万行硬上限(云客11931只翻到10000),
  uid=员工WA号过滤(真机验证 declared 1145→1149),total-aware翻页
- declared-total 分渠道:Σ wechat/whatsapp_account.contacts_num(8590/11915)
- 消息增量同步顺带刷新 customer.last_chat_time(云客列表按最近沟通实时
  排序,快照每日一次导致顺序滞后;微信1:1+WA都加)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:04:06 +08:00
浅念 0d04f68c07 fix(scraper): 微信客户抓取三连修(wechatId过滤+完整body+total-aware翻页,6563→6846全量) 
- 真机抓包确认过滤字段是 wechatId(原 userId 被无视,每员工拉回部门前1000行)
- 补 ymdStart/ymdEnd/friendTypeYunke=1/isDel=0 完整 body(与云客页面一致)
- total-aware 翻页:端点分页中途掉短页,翻到声明数满或连续空页(重复行幂等)
- 验证:csf 员工声明1999→distinct1846全量到手;云客total含跨页重复水分(1999声明vs1846真实)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 12:32:03 +08:00
浅念 1d90829754 feat(scraper): WA群消息抓取(chatRecords friendUid=@g.us,chatRoomTalker=群内发送者) 
- 真机探明:同 chatRecords 端点 friendUid 直接传 @g.us 群id(与微信群同套路)
- 群行 chatRoomTalker=发送者@lid、keyId 有值(缺失时合成稳定主键)
- file 字段直连 OSS URL 优先于 mediaName(WA群媒体 URL 现成!)
- 守护循环同微信群;full_sync 加 9b 步;3 新测试,416 绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:54:09 +08:00
浅念 b522be828a feat(scraper): 五实体导航拦截全改浏览器fetch(登录墙杀死拦截方式0捕获) 
- friends/groups/friend_list/es 部门级全量(记录自带owner,告别逐员工遍历)
- accounts/wxUser/list form模式(页码字段是page非pageNum,粘页检测保险)
- wa-accounts/wa-friends 直发;wa-friends 200页1万行(疑端点1万行硬上限)
- _fetch_pages 通用翻页助手:cookie失效首页code告警+粘页检测
- 实测:微信客户5916→6563/WA客户8058→8585/群851→873/账号50→55(捞回登录墙期间漏抓)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:41:33 +08:00
浅念 10cdd81d5c fix(scraper): 消息归一化补fileTh/fileHd→列映射(图片缩略图URL进列,媒体下载器可及) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:46:54 +08:00
浅念 02dd77a62f feat(scraper): WA群抓取(type=1真机验证,223群)+群成员派生(12840成员/750群) 
- wa-groups 实体:businessFriendList type=1 真机验证返回群,行结构是好友字段名
  (friendUid=@g.us 群ID/nickName=群名),专用 _normalize_wa_group_row
- backfill_group_members.py:无独立成员API,从群消息派生(talker=成员,sender_name=昵称)
- full_sync_all.sh:+4步(5b WA群/语音后录音重链接/8b 群消息+成员/消息上限99999)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:21:20 +08:00
浅念 f92aac1a3d feat(scraper): 微信群消息抓取(368778条/750群)+浏览器守护循环+SQLite忙等 
- 新 group-messages 实体:真机探明群消息=同一 msg/chats 端点 friendId 传 roomId;
  talker=发送者wxid、msgId=null→owner:room:createTime:talker 合成稳定主键(重扫幂等)
- 浏览器守护循环:超大群撑崩渲染进程时重启浏览器跳过毒群继续(实战2次自愈);
  200页上限;恢复路径补 await
- base.py SQLite connect_args timeout=60:并发写忙等而非撞锁报错
  (双抓取器并发写实测 database is locked 崩任务)
- 5个新单测,全量410绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:09:32 +08:00
浅念 ab5a918a3b fix(calls+scraper): 语音群scope服务端过滤+对话发现缺口补全+抓取韧性三连修 
- /api/calls 加 scope=friend|group(@chatroom后缀判群,与isRoom实测等价120=120),CallList去掉「全部」tab改服务端查
- 对话发现:全部customer pair ∪ 语音pair(weChatId,talker),未扫过的排最前——修复纯语音关系客户聊天记录永远空(35%语音客户pair从未扫过)
- voice-calls改浏览器直发(isRoom=0/1双趟+WA callSource=4,拦截方式已两次失效)
- 断连恢复重试30×2min(云客约3h一次断连,已实战扛过多次)
- WA messages upsert改整列表直调(_upsert_all逐条传dict被迭代成key字符串→TypeError)
- 405测试绿含scope新增3个

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-19 16:48:22 +08:00
浅念 f9ed7dd821 feat(scraper): 微信消息翻页取完整历史(paginate_chats hasLast/end游标+200页上限,移除临时skip;全量回填3478对话+95361条至2025-10) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-18 09:15:04 +08:00
浅念 aeb8a89b03 fix(scraper): talker_id用发送的friend_id优先(API返回的talker可能脱敏导致消息查不到) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-17 09:29:39 +08:00
浅念 bcacf80a7b fix(scraper): 过滤type=-2日期分隔线 + 清理DB已有4062条分隔线 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 15:07:43 +08:00
浅念 9eb747f2b3 feat(scripts): 全量同步改纯页面方式(10步零API零风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:45:50 +08:00
浅念 4a148706b9 feat(scraper): 页面抓取器支持全实体(账号/WA客户/语音/长语音/WA消息,全浏览器方式) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:44:20 +08:00
浅念 7beae8cf55 feat(scripts): 全渠道全量重同步脚本(页面抓取+API全覆盖) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:38:01 +08:00
浅念 cf858a00c2 fix(scraper): 消息页面抓取器改用浏览器fetch(可靠)+空数据安全 2026-08-14 15:30:02 +08:00
浅念 c6217ab1a9 feat(scraper): 页面抓取器支持消息(导航聊天窗口拦截msg/chats API) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-14 15:04:21 +08:00
浅念 bead7fdf43 fix(scraper): 页面抓取器支持员工列表翻页(之前只抓第一页10个) 2026-08-14 09:25:10 +08:00
浅念 004e6e4d6b feat(scraper): 页面浏览器抓取器(Playwright拦截SPA API响应,绕过风控) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:38:42 +08:00
浅念 8d7774b9f5 feat(scripts): backfill Customer.add_time from raw_json 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 18:45:55 +08:00
浅念 c8311bb496 fix: sanitize lone surrogate chars in stored data + backfill script 
Upstream API records may contain lone surrogate chars (emoji fragments)
that break SQLite/MySQL utf-8 encoding. Now sanitized to U+FFFD before
storage. Includes backfill_customers.py for full customer backfill.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 11:57:12 +08:00
浅念 fd86adfb63 chore: seed chat messages for listed customers (seed_chats) 
Iterates smoke.db customers, scrapes each conversation's recent messages
so timelines aren't empty. Bounded per-customer.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 09:54:40 +08:00
浅念 a57ed57616 fix: whatsapp businessFriendList needs page+departmentId+type 
Live validation: businessFriendList uses 'page' (not pageNum) and REQUIRES
departmentId (root) + type or returns empty. Same class of bug as wxUser/list.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-01 17:53:26 +08:00
浅念 0df53c6b1a fix: wechat account scrape (form-data+dates) + frontend staff page 
Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
  isOnJob / departmentId / startdate / enddate + empty userId/salesname/
  weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
  plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
  Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.

Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-01 17:11:17 +08:00
浅念 2508b5aec1 fix: live-validation fixes (list-coercion, .env session loading, smoke summary) 
Cleanup after successful live validation (10 real customers scraped+stored):

- app/config.py: add yk_userid / yk_company / yk_cookie Settings fields so
  existing-session creds can be supplied via .env (bypasses programmatic
  login for validation with a live browser session).
- app/storage/repos.py: coerce list values to ';'-joined strings in
  _build_column_values. Fixes a real bug: the live friends-list API returns
  Customer.deptRoute as a LIST, but dept_route is a scalar String(255)
  column. raw_json still preserves the original list (no data loss).
- scripts/smoke_wechat.py:
  * Revert build_client_from_env to read os.environ ONLY (no get_settings
    fallbacks), so the monkeypatch.setenv unit tests stay hermetic and the
    real .env stops leaking in. 3 previously-red tests green again.
  * Add _push_settings_into_environ() (Settings -> os.environ, real env
    wins, only fills gaps) and call it as the first line of main(); so the
    smoke run still honors .env without the pure builder depending on
    Settings. .env -> Settings -> os.environ (once) -> build_client_from_env.
  * Keep the await _print_summary fix from validation.
- tests/test_repos.py: regression test for list dept_route coercion
  (stored joined 'A;B', raw_json keeps the list).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-01 15:23:43 +08:00
浅念 eebff9f786 feat: RSA login encryption + confirmed wechat type codes 
Replace the Phase-0 placeholder password encryption with real RSA
(PKCS#1 v1.5). encrypt_password now builds an RSA public key from the
getSecureKey modulus/exponent (RSAPublicNumbers, e=int(hex,16),
n=int(hex,16)), encrypts the UTF-8 password, and returns hex (documented
assumption -- base64 is the one-line fallback noted in the docstring).
get_secure_key now returns the inner data dict to match the new contract;
pc_login keeps {account, password} with a comment on the username/password
and account/pwd fallbacks if pcLogin rejects.

Extend normalize_wechat_type with the Phase-0 confirmed codes:
2 -> image, 3 -> image_legacy, 4 -> video, 10 -> system (1 -> text and
-2 -> divider unchanged; 6/9/13/15/21/22 stay unknown until confirmed).

Tests:
- encrypt_password round-trip: generate an RSA keypair, build secure_key
  from its public numbers, encrypt, then decrypt with the private key and
  assert the original password comes back. Pins 512-hex output for a
  2048-bit key. No network.
- Parser parametrize table extended for the new codes (and a downstream
  message-fixture assertion updated from unknown -> image_legacy).

Suite: 87 -> 97 passing. ruff check clean.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-01 13:37:42 +08:00
浅念 8dc8caa749 chore: wechat end-to-end smoke script 
Wires the existing pieces together into a runnable integration check:
scrape ~10 WeChat friends + ~50 of their messages (last 7 days) from the
real yunke CRM (read-only), upsert into a local SQLite smoke DB, then
print a summary that verifies rows landed.

Two auth modes:
* Existing-session (preferred): YK_USERID/YK_COMPANY[/YK_COOKIE] -> YunkeClient
  constructed from a captured session, login bypassed.
* Login mode (Phase-0 blocked): YUNKE_USERNAME/YUNKE_PASSWORD -> attempts
  client.login(), which the real CRM will reject until the placeholder
  password encryption is replaced. Prints a clear message and exits 3.

build_client_from_env() is split out as a pure, network-free helper so
auth-mode selection is unit-testable. main() is async (asyncio.run in
__main__). HTTPStatusError from a 401/expired session is wrapped into a
helpful message + exit 4 (no traceback). client.aclose() + engine.dispose()
always run in finally.

Defaults to SMOKE_DB_URL=sqlite+aiosqlite:///./smoke.db so no MySQL is
required. Run from yunke-server/ with the venv active:

    python -m scripts.smoke_wechat

Tests cover the entrypoint shape and auth-mode selection (no network).
Suite: 77 -> 87 passing.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-01 12:16:02 +08:00