feat(analytics): 4h 调度分析链+CLI 回填+full_sync 夜间自愈步

- scheduler: yunke_analytics_tick job(与消息增量同节奏 IntervalTrigger),
  _analytics_tick 纯 DB——stat_daily_staff 滚动3天刷新 + 敏感词增量扫描,
  不抢 _page_lock 无子进程,异常只记日志不断链;today 用 time.time()
  UTC epoch 喂 day_key,规避服务器非上海时区的 naive now 误判
- cli: staff-daily-stats(全量重建 rollup,幂等)+ sensitive-scan
  (默认增量水位扫描,--full 首跑建水位/新词回填历史;description 明示
  加词不回填历史)+ 幂等补种起步词库
- full_sync_all.sh: daily-stats 后追加 staff-daily-stats(rollup 随
  msg_daily_stats 重建同步自愈)
- 测试: tick 行为测试(今日 seed→stat_daily_staff 出行)+ 异常吞噬
  (monkeypatch 抛错不外抛);同步更新注册断言(3→4 jobs)

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
浅念
2026-09-04 21:02:41 +08:00
parent 6a1c904018
commit 6b99202e9b
5 changed files with 212 additions and 4 deletions
+39
View File
@@ -28,6 +28,7 @@ import logging
import os
import shutil
import sys
import time
from collections.abc import Awaitable, Callable
from pathlib import Path
from typing import Any
@@ -535,6 +536,16 @@ class SyncScheduler:
coalesce=True,
max_instances=1,
)
# 4h 分析链:stat_daily_staff 滚动3天刷新 + 敏感词增量扫描(纯 DB,
# 无浏览器子进程;与消息增量同节奏,读滞后语义与「数据截至」一致)
sched.add_job(
self._analytics_tick,
IntervalTrigger(hours=self.message_interval_hours),
id="yunke_analytics_tick",
replace_existing=True,
coalesce=True,
max_instances=1,
)
sched.start()
self._scheduler = sched
logger.info(
@@ -662,6 +673,34 @@ class SyncScheduler:
logger.info(line)
print(line, flush=True)
async def _analytics_tick(self) -> None:
"""4h 分析链(stat_daily_staff 滚动3天 + 敏感词增量)——异常只记日志不断链。
纯 DB 无浏览器子进程,不抢 ``_page_lock``(DB 写毫秒级;
``max_instances=1`` 防自身重叠即可;与夜间 full_sync 的
staff-daily-stats 步骤重叠时 UNIQUE 兜底幂等)。
时区:today 用 ``time.time()``(UTC epoch,无时区歧义)喂
``day_key``——naive ``now().timestamp()`` 按本地时区解释,服务器
时区未必是上海(2026-09-04 部署注意点)。
"""
try:
async with self._session_maker() as session:
from app.analytics.sensitive import scanner
from app.storage.staff_summary import recompute_days, rolling_days
from app.storage.summary import day_key
today = day_key(int(time.time() * 1000))
n = await recompute_days(session, rolling_days(today, 3))
scan = await scanner.run_incremental(session)
await session.commit()
logger.info(
"[analytics-tick] staff_days=%s scanned=%s hits=%s",
n, scan.get("scanned"), scan.get("hits"),
)
except Exception: # noqa: BLE001 — scheduler MUST NOT crash
logger.exception("analytics tick failed")
async def _page_subprocess(self, args: list[str]) -> int:
"""启动页面子进程并等待结束;继承环境 + 补 DB_URL/PYTHONIOENCODING。"""
env = {**os.environ, "PYTHONIOENCODING": "utf-8"}