xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
403 Commits 1 Branch 0 Tags
602bee8fdfd7a8a8bde9e0e56cd71cb226e5042a
Commit Graph

20 Commits

Author SHA1 Message Date
浅念 a9197f696a fix(sync): 同步链路审计修复——水位边界白拉/全量无超时/群活跃窗判定源 
全项目数据同步审计发现四漏洞(前项为证据):

① 消息增量边界白拉(证据:昨夜 wa-messages fetched=53,682 而真实新消息
   仅 321 条,99.4% 是白拉白写的已入库旧消息)——云客 chatRecords/msg-chats
   的 start 参数是 inclusive(返回 ts>=start 含边界),四个消息实体(微信
   1:1/WA 1:1/微信群/WA 群)统一在归一化后过滤 ts<=watermark 的边界行,
   不计数不写库(省 99% 写放大,计数从此反映真实新增)
② full_sync_all.sh 无超时——极端情况(常态 10-20h)拖进次日相位把 02:00
   全量又挤掉(前两天事故同型);加 22h 上限
③ 周日 full-history 无超时——4 实体全历史可能 24h+,必挤周一;单实体
   加 6h 上限
④ 群消息活跃窗判定源是昨晚 full 的 chat_group.last_chat_time 快照——
   「昨天安静、今天突然活跃」的群会被 24h 窗口漏到明晚;4h 序列在
   group-messages 前加 chatroom-stats 全窗(~1min)刷新群活跃判定源

722 测试绿(4h 序列断言 9→10 命令+chatroom-stats 位序)。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-10 11:08:15 +08:00
浅念 e3fadfe1a6 fix(scheduler): 调度根治——4h 增量只扫活跃对话+账号表进 4h+超时保险丝 
用户报「员工列表时间与云客差 2 天」。根因链:4h 增量序列全对话遍历
(WA 9,357 对×~3.5s≈9h,整序列 15h+)超过自身 4h 间隔→与 02:00 全量在
asyncio.Lock 上背靠背互相排队→近两晚全量被挤掉→账号表(仅全量刷)停更
两天,员工列表「数据更新/最新消息时间」显示 09-08 快照。

三层修复:
- 活跃窗:_discover_conversations/群发现加 active_since_ms——已扫过的
  对话仅保留 last_chat_time 在窗内的(云客 lastChatTime 滞后小时级,
  24h 窗足够容错),从未扫过的(新覆盖优先)不受限;陈旧对话无新消息,
  每日 full 兜底。CLI --active-window-hours,四消息实体透传
- 4h 序列头部加 accounts/wa-accounts(~2min)——员工列表时间最多滞后
  4h,不再依赖每晚全量
- 超时保险丝:_page_subprocess 加 timeout(超时 kill 返回 124,上游
  break 下轮补)——messages 各实体 15-80min 预算,单命令拖不死相位;
  full/full-history 不设限

722 测试绿(序列断言 7→9 命令+窗口参数)。干跑验证:近 24h 活跃私聊/
群相对全量降一个数量级(本地旧库群 2011→537;私聊依赖 friends 先刷,
序列顺序已保证)。

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-10 10:38:47 +08:00
浅念 f839a6db20 docs(analytics): tick docstring锁竞争与时区措辞修正+测试名对齐4job 
Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-04 21:14:52 +08:00
浅念 6b99202e9b feat(analytics): 4h 调度分析链+CLI 回填+full_sync 夜间自愈步 
- scheduler: yunke_analytics_tick job(与消息增量同节奏 IntervalTrigger),
  _analytics_tick 纯 DB——stat_daily_staff 滚动3天刷新 + 敏感词增量扫描,
  不抢 _page_lock 无子进程,异常只记日志不断链;today 用 time.time()
  UTC epoch 喂 day_key,规避服务器非上海时区的 naive now 误判
- cli: staff-daily-stats(全量重建 rollup,幂等)+ sensitive-scan
  (默认增量水位扫描,--full 首跑建水位/新词回填历史;description 明示
  加词不回填历史)+ 幂等补种起步词库
- full_sync_all.sh: daily-stats 后追加 staff-daily-stats(rollup 随
  msg_daily_stats 重建同步自愈)
- 测试: tick 行为测试(今日 seed→stat_daily_staff 出行)+ 异常吞噬
  (monkeypatch 抛错不外抛);同步更新注册断言(3→4 jobs)

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-09-04 21:02:41 +08:00
浅念 90c28f4c7b feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media) 
- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+
  _iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/
  信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account,
  form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments
- media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无
  回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit)
  接入scrape_media第4源
- full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数
- scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页
  由每日full承担);测试断言7条命令序列
- tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核
  (两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化
  (一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-25 14:07:39 +08:00
浅念 d4bc3f5476 feat(scheduler): 4h增量接入全部四类消息(wa-messages/group-messages/wa-group-messages,水位后单对话1页可承受) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 15:15:24 +08:00
浅念 621c2cc929 feat(scheduler): 每周日05:00全量对齐job(四消息实体--full-history,修撤回/编辑+消除滞后窗口) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-21 13:52:59 +08:00
浅念 8a8c91a6a3 fix(customers): 客户列表排序实时对齐云客(全局实时body刷新last_chat_time,前5位逐位一致) 
- scrape_friends 双阶段:逐员工全量保底 + 全局实时body(friendTab=all/isOnJob/ymd窗)
  刷 last_chat_time(该参数集跨员工按最近沟通实时排序,实测比**赐14:10实时值)
- 阶段2只刷last_chat_time且只增不减(全局结果含旧ES副本,upsert全字段会被后写覆盖)
- 调度4h增量改为 friends→wa-friends→messages(排序键每4h实时化)
- 实测我们top5=云客top5逐位一致(A**杰14:13/比**赐14:10/P**6/A**邱/J**n)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 14:46:02 +08:00
浅念 5923cd05b1 feat(scheduler): 定时同步全面页面化(子进程跑页面脚本,API直调退出调度防风控) 
- 每日 full: bash scripts/full_sync_all.sh;每4h messages: scrape_via_page --entity messages
- 子进程隔离+asyncio.Lock互斥+非零退出只记JSON日志;自动补DB_URL环境
- ENTITY_FUNCS/run_once/CLI保留为手动调试,不再挂定时触发器
- README同步更新(Docker镜像需Playwright+Chromium);413测试绿

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-20 09:33:53 +08:00
浅念 9a953553ec fix(xss+perf): SearchView XSS消除(无v-html)+room_id索引+pool_recycle 
- SearchView.vue: 用 highlightSegments() 返回 {text,hit} 数组替代 v-html+escapeHtml+regex,模板用 <template v-for> + <mark>/<span> 渲染,从结构上消除 XSS 风险(4 处 v-html 全部移除: 消息片段/客户昵称/realName/手机号)
- models.py: Message 新增 ix_message_channel_owner_room_time(channel,owner_account_id,room_id,timestamp),群聊查询过滤 room_id 而非 talker_id,原 talker 索引无法覆盖
- migrations/d8eee96cd017: 新增 room_id 索引的 upgrade/downgrade (offline SQL 已验证)
- base.py + scheduler.py + cli.py: 所有生产 create_async_engine 增加 pool_recycle=3600,在 MySQL wait_timeout(默认 8h)断连前主动回收

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 11:50:45 +08:00
浅念 caef32891e fix(security): 路径穿越防护+导出行数上限+并发同步锁(3 critical/important) 
- media.py: GET /api/media/{hash} 现强制 file 必须位于 Settings.media_dir
  沙箱内(resolve + relative_to),杜绝 local_path 越权读取任意文件。
- export.py: /api/export/chat-html 加 EXPORT_MAX_ROWS=5000 硬上限(.limit()),
  超出时仅渲染前 5000 条并插入 .export-warning 横幅,防止 OOM。
- scheduler.py: _safe_run 增加 SyncRun advisory lock(60min TTL),发现同
  entity 存在 status='running' 且在 TTL 内的记录即返回 skipped SyncResult;
  同时为本轮 stake 一行 wrapping SyncRun,让并发 CLI/scheduler/手动触发
  能相互避让。崩溃场景由 TTL 自动恢复。
- tests: 新增 path-traversal 拒绝/export 截断警告/advisory lock skip+TTL
  恢复 5 个测试;test_api_media 的 fixture 改为 MEDIA_DIR=tmp_path 以适配
  新沙箱语义。

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-12 10:37:13 +08:00
浅念 5ce33232c1 feat(sync): 聊天图片本地化(message media_hash+download_message_images+local-first rendering) 
- Message 模型新增 media_hash 列 (sha256 FK -> Media.media_hash)
- Alembic 迁移 e5f6a7b8c9d0_message_media_hash (batch_alter_table, SQLite/MySQL 通用)
- app/scraper/media.py:download_message_images 批量下载器，镜像 download_call_recordings
  - 候选: media_hash IS NULL AND channel IN ('wechat','whatsapp') AND type IN ('image','video','file') AND 任一 file_*_url LIKE 'http%'
  - image 类型优先 file_th_url (缩略图，小快省)，video/file 优先 file_url
  - per-row try/except + per-run tried set，失败不挂循环
- sync_message_images orchestrator 入口 + ENTITY_MESSAGE_IMAGES 常量 (提交自身 SyncRun，镜像 call_recordings)
- scheduler: _run_message_images + ENTITY_FUNCS['message_images'] + 加入 daily all job (尾部)
- MessageOut schema 暴露 media_hash 字段
- MessageBubble.vue local-first: media_hash 优先 /api/media/{hash}，否则回退 OSS file_url/file_th_url/file_hd_url
- types.ts: Message 接口添加 media_hash 字段
- 新增 test_message_images_downloader.py (6 tests, 镜像 test_recording_downloader)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 18:24:11 +08:00
浅念 0e1cb87156 feat(scraper): WA 群 scraper(businessFriendList+type=1,待风控重置后实测) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 16:04:03 +08:00
浅念 538a0dc635 feat(sync): 长语音音频本地化(voice_clip media_hash+download+local-first playback) 
Mirror download_call_recordings pattern for voice_clips:
- VoiceClip.media_hash column + Alembic migration (c4d5e6f7a8b9)
- VoiceClipOut.media_hash + frontend VoiceClip.media_hash type
- download_voice_clip_audio in app/scraper/media.py (resilient, sha256-dedup,
  same download_url/upsert_media helpers as call recordings)
- sync_voice_clip_audio orchestrator wrapper (commits own SyncRun, mirrors
  sync_call_recordings semantics)
- scheduler: _run_voice_clip_audio + ENTITY_FUNCS['voice_clip_audio'] +
  appended to daily _run_all job (after voice_clips)
- VoiceClipModal playClip local-first: /api/media/{media_hash} with OSS
  oss_file_name fallback (mirrors CallList.playRecord)

Tested: 376 pytest pass; vue-tsc clean; migration applied; 1523/1533 clips
localized (10 pending are genuine upstream HTTP 404s — resilient skip).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-11 12:06:26 +08:00
浅念 d9060b6920 feat(api): 长语音 voice_clip 后端(model+migration+scraper+sync+endpoint) 
- VoiceClip model (BigInteger PK voice_clip_id, structured cols + raw_json)
- Alembic migration a1b2c3d4e5f6 (chains from 8c4f1d2a9e6b)
- scrape_voice_clips(client, user_id, *, channel, page_size) per-employee scraper
  over POST /wechatweb/call/list {pageNum, pageSize, userId}; channel stamped
  locally (wechat/whatsapp), startTime/endTime ms-epoch string -> int
- upsert_voice_clip repo + start_time/end_time/mp3_duration int coercion
- sync_voice_clips orchestrator: iterates distinct user_id across
  WechatAccount + WhatsappAccount, stamps channel per account table
- scheduler ENTITY_FUNCS['voice_clips'] + added to daily 'all' job
- GET /api/voice-clips (user_id exact beats user_real_name fuzzy; channel;
  company_code; start/end window on end_time; ordered end_time desc)
- VoiceClipOut schema + router registered in main.py
- tests: scraper (5) + API (11) = 16 new; full suite 370 green

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 18:12:28 +08:00
浅念 a0db4535f1 feat(sync): WA messages 自动同步(per-conversation 增量水位,镜像 sync_wechat_messages) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 17:50:01 +08:00
浅念 ceca5bda4c feat(sync): 接 WA 自动同步(accounts+friends wiring to orchestrator/scheduler) 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-10 16:48:44 +08:00
浅念 38071e40ba feat(sync): sync_call_recordings + ENTITY_FUNCS['call_recordings'] 独立下载 pass 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 16:40:06 +08:00
浅念 18620870c6 feat(sync): sync_wa_voice_calls + scheduler ENTITY_FUNCS wa_voice_calls 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-07 09:36:14 +08:00
浅念 df29dd027a feat: scheduler + CLI + docker-compose + README 
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.

Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
  message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
  (departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
  all_wechat/all).
- Conversations for the message run are discovered from the local customer
  mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
  so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).

Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
  — long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
  mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.

Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
  image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
  reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
  SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
  backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.

Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).

Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-08-05 12:11:29 +08:00