feat(analytics): 4h 调度分析链+CLI 回填+full_sync 夜间自愈步
- scheduler: yunke_analytics_tick job(与消息增量同节奏 IntervalTrigger), _analytics_tick 纯 DB——stat_daily_staff 滚动3天刷新 + 敏感词增量扫描, 不抢 _page_lock 无子进程,异常只记日志不断链;today 用 time.time() UTC epoch 喂 day_key,规避服务器非上海时区的 naive now 误判 - cli: staff-daily-stats(全量重建 rollup,幂等)+ sensitive-scan (默认增量水位扫描,--full 首跑建水位/新词回填历史;description 明示 加词不回填历史)+ 幂等补种起步词库 - full_sync_all.sh: daily-stats 后追加 staff-daily-stats(rollup 随 msg_daily_stats 重建同步自愈) - 测试: tick 行为测试(今日 seed→stat_daily_staff 出行)+ 异常吞噬 (monkeypatch 抛错不外抛);同步更新注册断言(3→4 jobs) Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
@@ -28,6 +28,7 @@ import logging
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -535,6 +536,16 @@ class SyncScheduler:
|
||||
coalesce=True,
|
||||
max_instances=1,
|
||||
)
|
||||
# 4h 分析链:stat_daily_staff 滚动3天刷新 + 敏感词增量扫描(纯 DB,
|
||||
# 无浏览器子进程;与消息增量同节奏,读滞后语义与「数据截至」一致)
|
||||
sched.add_job(
|
||||
self._analytics_tick,
|
||||
IntervalTrigger(hours=self.message_interval_hours),
|
||||
id="yunke_analytics_tick",
|
||||
replace_existing=True,
|
||||
coalesce=True,
|
||||
max_instances=1,
|
||||
)
|
||||
sched.start()
|
||||
self._scheduler = sched
|
||||
logger.info(
|
||||
@@ -662,6 +673,34 @@ class SyncScheduler:
|
||||
logger.info(line)
|
||||
print(line, flush=True)
|
||||
|
||||
async def _analytics_tick(self) -> None:
|
||||
"""4h 分析链(stat_daily_staff 滚动3天 + 敏感词增量)——异常只记日志不断链。
|
||||
|
||||
纯 DB 无浏览器子进程,不抢 ``_page_lock``(DB 写毫秒级;
|
||||
``max_instances=1`` 防自身重叠即可;与夜间 full_sync 的
|
||||
staff-daily-stats 步骤重叠时 UNIQUE 兜底幂等)。
|
||||
|
||||
时区:today 用 ``time.time()``(UTC epoch,无时区歧义)喂
|
||||
``day_key``——naive ``now().timestamp()`` 按本地时区解释,服务器
|
||||
时区未必是上海(2026-09-04 部署注意点)。
|
||||
"""
|
||||
try:
|
||||
async with self._session_maker() as session:
|
||||
from app.analytics.sensitive import scanner
|
||||
from app.storage.staff_summary import recompute_days, rolling_days
|
||||
from app.storage.summary import day_key
|
||||
|
||||
today = day_key(int(time.time() * 1000))
|
||||
n = await recompute_days(session, rolling_days(today, 3))
|
||||
scan = await scanner.run_incremental(session)
|
||||
await session.commit()
|
||||
logger.info(
|
||||
"[analytics-tick] staff_days=%s scanned=%s hits=%s",
|
||||
n, scan.get("scanned"), scan.get("hits"),
|
||||
)
|
||||
except Exception: # noqa: BLE001 — scheduler MUST NOT crash
|
||||
logger.exception("analytics tick failed")
|
||||
|
||||
async def _page_subprocess(self, args: list[str]) -> int:
|
||||
"""启动页面子进程并等待结束;继承环境 + 补 DB_URL/PYTHONIOENCODING。"""
|
||||
env = {**os.environ, "PYTHONIOENCODING": "utf-8"}
|
||||
|
||||
Reference in New Issue
Block a user