xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
278 Commits 1 Branch 0 Tags
35bfebd3e3e35d4913ab54d080a6bc9bdf094578
Commit Graph

102 Commits

Author SHA1 Message Date
浅念 35bfebd3e3 chore(backend): list_order迁移补齐+Task14类型修正落链+moments每页日志+MsgTypeParam派生+3处文档修正+Dockerfile打包migrations+DEPLOY补迁移步骤 
- 9b0d0c133035: wechat_account.list_order 进迁移链(此前仅模型声明,
  干净库 /api/accounts 即 500);sa.inspect 幂等守卫兼容已手动加列的
  smoke.db
- fe63c6789e94(验证中发现的预存缺口,超出原清单): Task 14 的 5 列模型
  类型修正(4×URL String(1024)→Text, db_size Integer→BigInteger)从未落
  链——纯 alembic 新库会复活 >1KB 链接截断;对齐后干净库 alembic check
  全绿。SQLite message 表 batch 重建会掉 FTS 触发器+rowid 脱节(docstring
  记明 ensure_fts+rebuild_fts 修复,smoke.db 已修复验证)
- scrape_via_page: _iter_moments_pages 每页一行进度日志(镜像 _fetch_pages)
- messages: MsgTypeParam=Literal[tuple(_MSG_TYPE_FILTER)] 派生,杜绝
  双处罗列漂移(openapi enum 验证一致,492 绿含 422 行为)
- 文档: ChatGroup 复合PK docstring/groups room_id 多行表述/测试名去
  单行假设;Dockerfile 补 alembic.ini+migrations;DEPLOY 增「数据库
  迁移」节(新库/旧库stamp路径/回填脚本显式DB_URL)+运维表升级行

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 20:32:25 +08:00
浅念 1b66be6a53 feat(api): /api/customers 加 q 文本搜索(昵称/备注/ID/手机号,LIKE转义) + /api/groups 加 q 群名搜索 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 18:55:53 +08:00
浅念 4baa8639ba fix(api): staff-stats评审修复(GROUP BY显式列防ONLY_FULL_GROUP_BY/金额千分位/detail死参数/2边缘测试) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 16:36:07 +08:00
浅念 b95fc5fa6f feat(api): GET /api/staff-stats 7类计数+分型明细(voice走user_id join,新客户add_time兜底,红包金额提取) 
计数模式(无kind)返回 new_customers/new_groups/red_transfer/del_customers/
blocked_customers/voice_calls/sensitive_words(恒0,spec占位);明细模式
(?kind=)返回{kind,rows},每kind字段不同,前端按kind配列。start/end
缺省近7天。

口径(spec §4.5,全本地聚合):
- new_customers: add_time窗内;IS NULL行用该对话首条消息min(timestamp)兜底
- new_groups: chat_group.create_time窗内(复合PK按owner过滤)
- red_transfer: wechat type IN (red_packet,transfer)窗内;金额三路正则
  (红包XML feedesc/转账纯文本/totalamount),实测真库转账399/399全提取
- del/blocked_customers: 现存量口径忽略窗口,明细time=updated_at毫秒
- voice_calls: phone_call.user_id存员工userId(32HEX)非wxId——先经
  wechat_account.user_id换算,账号无uid返回空防跨员工串数
- voice明细字段语义对齐CallList.vue(connected=call_status==1,
  media_url本地/api/media/{hash}优先OSS兜底)

真机smoke(yunke-mysql):voice 61条/新客67/删除29/拉黑1均通过;
转账金额100%提取。真库chat_group列与message.type回填滞后(运维项,另报)。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 16:22:44 +08:00
浅念 b289aa2650 fix(api): sessions is_del 越界422校验+省略不过滤显式契约测试 
is_del 加 Query(ge=0, le=1)——越界值(如2)422 拒绝而非静默空列表;
test_sessions_is_del_filter 追加第三个请求:不带 is_del 时全部 4 个
talker 都返回(f1/f2/filehelper/gh0st),补上省略=完全不过滤的显式契约。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 15:57:59 +08:00
浅念 2837500bc0 feat(api): GET /api/moments 朋友圈列表(desc分页+账号头部) 
对齐云客员工聊天窗口朋友圈tab:帖子 create_time desc 分页 +
wechat_account 头部(昵称/头像/别名);员工不存在不404(header全None
前端兜底)。头像提取抽共享 helper account_avatar 放 organization.py,
accounts 列表与 moments 头部复用。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 15:57:53 +08:00
浅念 d3d7146887 feat(api): sessions加is_del过滤+filehelper置顶标记,accounts加avatar(raw_json提取) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 15:44:54 +08:00
浅念 64577a1b02 feat(msg): 类型归一化扩展(spike判别+49拆红包/转账)+存量回填+API msg_type/q过滤 
- _normalize_msg_type 升级 (raw_type, row): messageType 经典码优先 →
  49/appmsg 行级拆分(红包/转账/聊天记录) → 云客码兜底
- spike 修正: 云客 type=3 实为语音(messageType=34, 8754/8955 行),
  原 image_legacy 是误标; type_raw=3 的 201 行旧数据才是真图片
- 新映射: 5红包 6/7通话 9文件 10/18/20链接 11转账 13名片 14位置
  15/10000系统 21引用回复→text 22/19 证据不足留 unknown
- 回填 smoke.db: 620279 行重扫, 94268 行修正, unknown 87198→1885, 幂等
- GET /api/messages 加 msg_type(Literal 422)+q(LIKE 转义)参数
- 新增 test_scrape_msg_type.py(11) + test_api_messages 追加(8), 全量 466 绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 15:17:21 +08:00
浅念 f6ddd0263a fix(repos): int 强转跳过 String 声明列(列类型优先于 _INT_FIELDS 按名集合) 
Task 1 评审遗留:_INT_FIELDS 含 type_raw(为 Message int code 加),但
MomentsPost.type_raw 是 String(16)——按名强转会把非数字 code(如
'text')置 None,违背列声明。_build_column_values 现在检查目标列
声明类型:isinstance(col.type, sa.String)(含 Text)则跳过强转,
Message.type_raw(Integer) 照旧。另修正 upsert_chat_group 文档串里
过时的 'PK = room_id'(复合键自 8a99940)。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 14:32:19 +08:00
浅念 6ff288f1d3 feat(groups): chat_group 补 6 群主/建群列 + 复合PK迁移修复 + 存量回填 
对齐云客员工聊天窗口·微信群 tab(2026-08-25 真机验证 raw_json 键):
- ChatGroup 加列 owner_name/owner_wxid/owner_alias/owner_avatar/
  friend_count/create_time(全 nullable)
- scraper _normalize_chat_group 同名提取;createTime 兼容 ms-int 与
  'YYYY-MM-DD HH:mm:ss'(smoke.db 实测 1105 行全为后者),复用
  _parse_add_time 归一 ms-epoch
- 迁移 099ac2097fa2:加 6 列 + 修复 Task1 评审遗留 drift——PK 由单列
  room_id 变更为复合 (room_id, channel, owner_account_id) 且
  owner_account_id NOT NULL(对齐模型 8a99940;smoke.db 1105 行无 NULL,
  未兜底编造数据);SQLite 走 batch 重建(copy_from 显式旧结构),
  MySQL 渲染为直接 ALTER(离线 --sql 已验证)
- 回填脚本 scripts/backfill_chat_group_fields.py(幂等,逐列只补空);
  smoke.db 实跑 backfilled 1105/1105,重跑 0/1105
- GroupOut 加 6 字段(model_validate 自动输出,路由无改)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 14:32:12 +08:00
浅念 90c28f4c7b feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media) 
- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+
  _iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/
  信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account,
  form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments
- media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无
  回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit)
  接入scrape_media第4源
- full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数
- scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页
  由每日full承担);测试断言7条命令序列
- tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核
  (两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化
  (一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 14:07:39 +08:00
浅念 b7669199f8 feat(db): moments_post 表+upsert(朋友圈实体,PK=云客data.id,create_time统一ms) 
- MomentsPost 模型:moments_post 表,PK=id(云客 getFriendsterInfo data.id),
  wechat_id/create_time 建索引,praise/comment/image 存 JSON 列
- 迁移 524da8d2d597(手写 create_table,已应用到 smoke.db 并在临时 SQLite
  全链路 upgrade/downgrade 验证)
- repos.upsert_moments_posts:批量幂等 upsert;附带修 _build_column_values
  对 JSON 列表的误 join(标量列 join 保持不变,dept_route 旧测试仍过)
- 修复 3 个 chat_group 旧测试缺 owner_account_id(2026-08-24 复合主键遗留)
- 全套 430 测试通过

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 13:44:54 +08:00
浅念 b6eea3070a feat(staff): 弹框客户微信/员工微信列改富单元格(头像+昵称+ID+真实姓名,对齐云客) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 10:24:07 +08:00
浅念 8a99940554 feat(groups): chat_group改复合主键(channel,owner,room)——多员工同群不再去重(云客张士蓓64群vs原42) 
原 room_id 单列 PK 把多员工同群合并成一行(末位覆盖owner),其他员工
群列表缺数据。改复合主键后同群每个员工各一行,对齐云客按员工查看。
重扫后 1590 行/865 去重群/张士蓓 64 群(与云客一致)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 12:01:25 +08:00
浅念 804addf725 feat(staff): 员工列表排序对齐云客API返回顺序(list_order列+抓取记序+API按序排) 
- wechat_account 加 list_order 列(云客 wxUser/list 返回顺序)
- 抓取器记录每行在API响应中的序号
- /api/accounts ORDER BY list_order(对齐云客员工列表排列)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-24 11:25:33 +08:00
浅念 abc97fe217 feat(search): MySQL ngram FULLTEXT分支(LIKE 13.5s→索引毫秒级;索引缺失自动回退LIKE) 
MySQL LIKE 全扫698k行13.5s撞axios 15s超时→前端ERR_ABORTED;
MATCH...AGAINST(ngram)毫秒级;本地C盘不足索引待服务器建
(索引SQL: CREATE FULLTEXT INDEX ft_msg_text ON message(text) WITH PARSER ngram)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 17:52:26 +08:00
浅念 a8b8d9d81c fix(export+calls): 同类owner错位问题全面修复(群导出不按owner过滤+群语音客户点击走群聊路由) 
- /api/export/chat-html: room 查询忽略 owner(chat_group owner 错位同
  ChatTimeline,群列表/聊天页的导出按钮原来会导出空文件)
- CallList viewCallCustomer: talker 以 @chatroom 结尾→group-chat 路由
  (原走 1:1 路由按 talker_id 查@chatroom 恒空)
- TDD: 群消息在 ownerB 名下用 ownerA+room 导出仍能拿到;428 测试绿
- 实测:错位 owner 导出 43304 群返回 15002 个消息元素

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 17:00:43 +08:00
浅念 a290e39523 feat(customers): 发送/接收列改今日口径(对齐云客;本地消息表Asia/Shanghai今日聚合) 
- CustomerOut 加 send_today/receive_today(路由赋值,非ORM列)
- /api/customers 每页一次 GROUP BY(owner IN 走复合索引前缀)
- 前端列优先今日值,无消息数据回退原raw字段;列名对齐云客「发送条数/接收条数」
- TDD:今日3发2收+昨日不计入+无消息客户0

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 14:39:17 +08:00
浅念 00ae13b5c2 feat(deploy): 完成数据迁移与MySQL全链路(14表698k消息100%一致+5方言补丁+bind mount) 
- migrate_sqlite_to_mysql.py: 逐表清空重导(幂等),复用repos强转,JSON列
  解析,67万消息按msg_id键集分批,467s跑完14表全一致
- 修4个MySQL实测bug: NULLS LAST语法(compat.py编译补丁)、message.text
  TEXT→MEDIUMTEXT(64KB字节上限,实测20.8万字符车源清单)、URL列
  String(1024)→Text(分享链接超1KB)、垃圾超长member_id源头清理
- compose: media改bind mount(不占C盘Docker虚拟磁盘,实测2.5GB卷把
  C盘撑爆引擎500);mysql不映射宿主机3306
- 前端Dockerfile: npm ci→install(本地/容器npm版本锁漂移)
- 五服务实测:nginx→backend→MySQL全链路返回真实数据(客户6859/A**杰25条
  /搜索56条),媒体49536文件bind mount直读

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 12:40:44 +08:00
浅念 2175cb0169 feat(deploy): MySQL容器实测打通(3个方言bug修复)+部署文档 
- repos: on_duplicate只更新实际插入列(fetched_at等Python-default列,
  SQLite excluded容忍/MySQL new.*直接报Unknown column)
- repos: _coerce_int增加日期字符串→ms epoch分支 + _INT_FIELDS补
  last_update/last_chat_time/timestamp/update_time(字符串直通BigInteger)
- models: wechat_account.db_size Integer→BigInteger(实测2.17GB超32位上限)
- compose: mysql不映射宿主机3306(本机已占;容器内服务名互联)
- docs/DEPLOY.md: 服务器部署全指南(架构/步骤/数据迁移/运维命令/注意事项)
- 容器实测:Playwright抓取55账号→MySQL upsert全通

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-22 10:40:04 +08:00
浅念 d4bc3f5476 feat(scheduler): 4h增量接入全部四类消息(wa-messages/group-messages/wa-group-messages,水位后单对话1页可承受) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:15:24 +08:00
浅念 e8c77412fa feat(search): FTS5 trigram全文索引(中文搜索3.95s→0.003s,1300倍) 
- message_fts 外部内容虚拟表+trigram分词(中文子串可MATCH,无需分词库)
- 3个触发器随ORM upsert自动同步索引(抓取器零改动)
- 搜索路由:≥3字走FTS MATCH,<3字/MySQL回退LIKE
- 实测:搜「大众途安」56条,LIKE全扫3.95s→FTS 0.003s;索引+300MB
- 7个新测试(建表幂等/触发器同步/中文匹配/短查询回退/未建索引回退),426绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:08:26 +08:00
浅念 621c2cc929 feat(scheduler): 每周日05:00全量对齐job(四消息实体--full-history,修撤回/编辑+消除滞后窗口) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 13:52:59 +08:00
浅念 10baa22acb fix(media): tried集合分片进NOT IN(3万+msg_id超SQLite变量上限,42k后崩;每晚全量媒体步同bug) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 11:26:50 +08:00
浅念 d41eb2431f fix(wa): WA消息file_url优先直连OSS的file字段(1:1图片93%有URL)+存量回填7373行(带URL 6028/6804) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 15:40:50 +08:00
浅念 8a8c91a6a3 fix(customers): 客户列表排序实时对齐云客(全局实时body刷新last_chat_time,前5位逐位一致) 
- scrape_friends 双阶段:逐员工全量保底 + 全局实时body(friendTab=all/isOnJob/ymd窗)
  刷 last_chat_time(该参数集跨员工按最近沟通实时排序,实测比**赐14:10实时值)
- 阶段2只刷last_chat_time且只增不减(全局结果含旧ES副本,upsert全字段会被后写覆盖)
- 调度4h增量改为 friends→wa-friends→messages(排序键每4h实时化)
- 实测我们top5=云客top5逐位一致(A**杰14:13/比**赐14:10/P**6/A**邱/J**n)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:46:02 +08:00
浅念 edca9b4c77 fix(customers): WA客户破1万上限(逐员工uid过滤,8585→11931全量)+双渠道页脚云客口径+消息同步刷新最近沟通时间 
- wa-friends 逐员工抓取:全局查询~1万行硬上限(云客11931只翻到10000),
  uid=员工WA号过滤(真机验证 declared 1145→1149),total-aware翻页
- declared-total 分渠道:Σ wechat/whatsapp_account.contacts_num(8590/11915)
- 消息增量同步顺带刷新 customer.last_chat_time(云客列表按最近沟通实时
  排序,快照每日一次导致顺序滞后;微信1:1+WA都加)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:04:06 +08:00
浅念 fd3cf0003b feat(customers): 微信客户页脚改云客口径(Σ员工好友数声明值8590,用户选B方案) 
- GET /api/customers/declared-total: Σ wechat_account.contacts_num
- 页脚优先显示云客口径(回退真实total);列表内容保持去重真实行
- 云客列表声明值含其ES索引重复doc(连行id都相同无法物理区分),仅对齐计数

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 13:44:39 +08:00
浅念 5d22d2b5f5 fix(login): pcLogin协议真机修正(phone/loginType/companyCode)+secureKey换usercenter端点 
浏览器实登抓包确认:POST /usercenter/login/getSecureKey(非newuc GET) +
body {phone,password(RSA-hex),loginType:'yunkecn',companyCode}
实测从500系统错误推进到10300密码错误=协议通,剩加密细节/密码核对(余5次重试暂停试错防锁号)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 11:40:16 +08:00
浅念 4aeed003a4 feat(calls): 微信语音列表加今日统计卡片+通话时间范围筛选+客户跳转带通话时间定位(对齐云客) 
- GET /api/calls/stats:今日(Asia/Shanghai)好友呼入/外呼量+群量+时长聚合(@chatroom判群)
- CallList:5张统计卡片(仅微信语音tab)+datetimerange筛选(服务端start/end)+viewCallCustomer带?start定位
- TDD新增stats聚合测试,全量绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 11:08:13 +08:00
浅念 6c9245a682 fix(groups): /api/groups加room_id精确直查(群聊页群名解析,修page_size=200超限422) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 10:58:39 +08:00
浅念 5923cd05b1 feat(scheduler): 定时同步全面页面化(子进程跑页面脚本,API直调退出调度防风控) 
- 每日 full: bash scripts/full_sync_all.sh;每4h messages: scrape_via_page --entity messages
- 子进程隔离+asyncio.Lock互斥+非零退出只记JSON日志;自动补DB_URL环境
- ENTITY_FUNCS/run_once/CLI保留为手动调试,不再挂定时触发器
- README同步更新(Docker镜像需Playwright+Chromium);413测试绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:33:53 +08:00
浅念 f92aac1a3d feat(scraper): 微信群消息抓取(368778条/750群)+浏览器守护循环+SQLite忙等 
- 新 group-messages 实体:真机探明群消息=同一 msg/chats 端点 friendId 传 roomId;
  talker=发送者wxid、msgId=null→owner:room:createTime:talker 合成稳定主键(重扫幂等)
- 浏览器守护循环:超大群撑崩渲染进程时重启浏览器跳过毒群继续(实战2次自愈);
  200页上限;恢复路径补 await
- base.py SQLite connect_args timeout=60:并发写忙等而非撞锁报错
  (双抓取器并发写实测 database is locked 崩任务)
- 5个新单测,全量410绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:09:32 +08:00
浅念 ab5a918a3b fix(calls+scraper): 语音群scope服务端过滤+对话发现缺口补全+抓取韧性三连修 
- /api/calls 加 scope=friend|group(@chatroom后缀判群,与isRoom实测等价120=120),CallList去掉「全部」tab改服务端查
- 对话发现:全部customer pair ∪ 语音pair(weChatId,talker),未扫过的排最前——修复纯语音关系客户聊天记录永远空(35%语音客户pair从未扫过)
- voice-calls改浏览器直发(isRoom=0/1双趟+WA callSource=4,拦截方式已两次失效)
- 断连恢复重试30×2min(云客约3h一次断连,已实战扛过多次)
- WA messages upsert改整列表直调(_upsert_all逐条传dict被迭代成key字符串→TypeError)
- 405测试绿含scope新增3个

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-19 16:48:22 +08:00
浅念 2d1943a74e fix(api): 客户/群排序改回last_chat_time DESC(云客按最近沟通优先) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:18:42 +08:00
浅念 854da6e566 fix(api): 客户/群列表排序改为friend_id/room_id ASC(对齐云客API自然顺序) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 14:14:24 +08:00
浅念 1f89eeddb0 feat(api): /api/customers 加 friend_id 筛选(用于查单个客户昵称) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-15 10:54:22 +08:00
浅念 2154d050e5 fix(media): image_legacy渲染+下载覆盖+WA raw_json URL回退 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-13 14:02:42 +08:00
浅念 9a953553ec fix(xss+perf): SearchView XSS消除(无v-html)+room_id索引+pool_recycle 
- SearchView.vue: 用 highlightSegments() 返回 {text,hit} 数组替代 v-html+escapeHtml+regex,模板用 <template v-for> + <mark>/<span> 渲染,从结构上消除 XSS 风险(4 处 v-html 全部移除: 消息片段/客户昵称/realName/手机号)
- models.py: Message 新增 ix_message_channel_owner_room_time(channel,owner_account_id,room_id,timestamp),群聊查询过滤 room_id 而非 talker_id,原 talker 索引无法覆盖
- migrations/d8eee96cd017: 新增 room_id 索引的 upgrade/downgrade (offline SQL 已验证)
- base.py + scheduler.py + cli.py: 所有生产 create_async_engine 增加 pool_recycle=3600,在 MySQL wait_timeout(默认 8h)断连前主动回收

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:50:45 +08:00
浅念 c261b2bc9a fix(security): CORS限制+SSRF防护+PATCH乐观锁+accounts安全cap(4 important bugs) 
- main.py + config.py: CORS origins read from Settings.cors_origins (comma-separated).
  Empty falls back to dev origin localhost:5173; never wildcard. Methods locked
  to GET/PATCH/POST; allow_credentials=False.
- scraper/media.py: download_url now passes follow_redirects=False to httpx.
  OSS URLs return 200 directly; a 3xx is treated as a download failure (caught
  by existing except clauses) instead of opening an SSRF vector to internal hosts.
- routers/voice_clips.py: PATCH /voice-clips/{id}/match accepts an optional
  If-Match header carrying the client's last-seen updated_at. On mismatch the
  row was modified by another request and the server returns 409. Without the
  header the behaviour is unchanged (backward compatible). Response now echoes
  the post-update updated_at so the client can fold it into the next request.
- routers/organization.py: /api/accounts now UNION ALLs the wechat_account and
  whatsapp_account projections at the SQL level and pushes COUNT + pagination
  (offset/limit) down to the DB. Stable ordering on (channel, account_id).
  Removes the in-memory materialization that broke at scale.

Tests: 395 passed. Added optimistic-lock coverage (If-Match accepted / stale 409 /
absent header backward-compat). Updated existing PATCH tests for the new
updated_at response field.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:41:30 +08:00
浅念 caef32891e fix(security): 路径穿越防护+导出行数上限+并发同步锁(3 critical/important) 
- media.py: GET /api/media/{hash} 现强制 file 必须位于 Settings.media_dir
  沙箱内(resolve + relative_to),杜绝 local_path 越权读取任意文件。
- export.py: /api/export/chat-html 加 EXPORT_MAX_ROWS=5000 硬上限(.limit()),
  超出时仅渲染前 5000 条并插入 .export-warning 横幅,防止 OOM。
- scheduler.py: _safe_run 增加 SyncRun advisory lock(60min TTL),发现同
  entity 存在 status='running' 且在 TTL 内的记录即返回 skipped SyncResult;
  同时为本轮 stake 一行 wrapping SyncRun,让并发 CLI/scheduler/手动触发
  能相互避让。崩溃场景由 TTL 自动恢复。
- tests: 新增 path-traversal 拒绝/export 截断警告/advisory lock skip+TTL
  恢复 5 个测试;test_api_media 的 fixture 改为 MEDIA_DIR=tmp_path 以适配
  新沙箱语义。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 10:37:13 +08:00
浅念 461e9ff081 tune(anti-risk): 加固防风控策略(间隔2s+抖动2.5s+上限300+退避5-120s×6次) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 09:45:33 +08:00
浅念 5ce33232c1 feat(sync): 聊天图片本地化(message media_hash+download_message_images+local-first rendering) 
- Message 模型新增 media_hash 列 (sha256 FK -> Media.media_hash)
- Alembic 迁移 e5f6a7b8c9d0_message_media_hash (batch_alter_table, SQLite/MySQL 通用)
- app/scraper/media.py:download_message_images 批量下载器，镜像 download_call_recordings
  - 候选: media_hash IS NULL AND channel IN ('wechat','whatsapp') AND type IN ('image','video','file') AND 任一 file_*_url LIKE 'http%'
  - image 类型优先 file_th_url (缩略图，小快省)，video/file 优先 file_url
  - per-row try/except + per-run tried set，失败不挂循环
- sync_message_images orchestrator 入口 + ENTITY_MESSAGE_IMAGES 常量 (提交自身 SyncRun，镜像 call_recordings)
- scheduler: _run_message_images + ENTITY_FUNCS['message_images'] + 加入 daily all job (尾部)
- MessageOut schema 暴露 media_hash 字段
- MessageBubble.vue local-first: media_hash 优先 /api/media/{hash}，否则回退 OSS file_url/file_th_url/file_hd_url
- types.ts: Message 接口添加 media_hash 字段
- 新增 test_message_images_downloader.py (6 tests, 镜像 test_recording_downloader)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 18:24:11 +08:00
浅念 af7f8d71c8 feat(export): 导出 HTML 媒体内嵌(图片/语音/视频/附件, OSS URL 直接嵌入) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 17:22:11 +08:00
浅念 0e1cb87156 feat(scraper): WA 群 scraper(businessFriendList+type=1,待风控重置后实测) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 16:04:03 +08:00
浅念 538a0dc635 feat(sync): 长语音音频本地化(voice_clip media_hash+download+local-first playback) 
Mirror download_call_recordings pattern for voice_clips:
- VoiceClip.media_hash column + Alembic migration (c4d5e6f7a8b9)
- VoiceClipOut.media_hash + frontend VoiceClip.media_hash type
- download_voice_clip_audio in app/scraper/media.py (resilient, sha256-dedup,
  same download_url/upsert_media helpers as call recordings)
- sync_voice_clip_audio orchestrator wrapper (commits own SyncRun, mirrors
  sync_call_recordings semantics)
- scheduler: _run_voice_clip_audio + ENTITY_FUNCS['voice_clip_audio'] +
  appended to daily _run_all job (after voice_clips)
- VoiceClipModal playClip local-first: /api/media/{media_hash} with OSS
  oss_file_name fallback (mirrors CallList.playRecord)

Tested: 376 pytest pass; vue-tsc clean; migration applied; 1523/1533 clips
localized (10 pending are genuine upstream HTTP 404s — resilient skip).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 12:06:26 +08:00
浅念 9da272b992 feat(calls): voice_clip 手动匹配(voice_clip↔customer 本地关联写操作) 
- VoiceClip: 新增 matched_friend_id 列 (String(128), nullable) + alembic 迁移 b3c4d5e6f7a8
- PATCH /api/voice-clips/{id}/match: friend_id 关联 / null 取消 (本地写, 不回写云客)
- GET /api/voice-clips: 批量解析 matched_friend_name (remark>nickname>real>id)
- VoiceClipModal: 已匹配客户列 + 匹配/取消匹配按钮 + 嵌套客户选择 picker (默认按员工列客户, 名搜索走 /search)
- 测试: 17 passed (新增 7 个 match 用例覆盖 set/clear/name 解析/404)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 10:43:34 +08:00
浅念 d9060b6920 feat(api): 长语音 voice_clip 后端(model+migration+scraper+sync+endpoint) 
- VoiceClip model (BigInteger PK voice_clip_id, structured cols + raw_json)
- Alembic migration a1b2c3d4e5f6 (chains from 8c4f1d2a9e6b)
- scrape_voice_clips(client, user_id, *, channel, page_size) per-employee scraper
  over POST /wechatweb/call/list {pageNum, pageSize, userId}; channel stamped
  locally (wechat/whatsapp), startTime/endTime ms-epoch string -> int
- upsert_voice_clip repo + start_time/end_time/mp3_duration int coercion
- sync_voice_clips orchestrator: iterates distinct user_id across
  WechatAccount + WhatsappAccount, stamps channel per account table
- scheduler ENTITY_FUNCS['voice_clips'] + added to daily 'all' job
- GET /api/voice-clips (user_id exact beats user_real_name fuzzy; channel;
  company_code; start/end window on end_time; ordered end_time desc)
- VoiceClipOut schema + router registered in main.py
- tests: scraper (5) + API (11) = 16 new; full suite 370 green

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 18:12:28 +08:00
浅念 a0db4535f1 feat(sync): WA messages 自动同步(per-conversation 增量水位,镜像 sync_wechat_messages) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 17:50:01 +08:00
浅念 e8885efcc6 fix(sync): 补 import upsert_whatsapp_account(WA accounts sync NameError) 2026-08-10 17:02:28 +08:00