Compare commits
10 Commits
2c7625c3e4
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
| 682249a3fd | |||
| 80b721b84f | |||
| acd6997bf0 | |||
| 253be99eca | |||
| e00d399195 | |||
| 2ac63e3844 | |||
| 0dc5993af7 | |||
| 0b975a8289 | |||
| 8f9ef747cc | |||
| 3d2889bcde |
@@ -372,3 +372,40 @@ action_counts 内部:`unreplied(hours=2)` **2.38s**;`churn(real,30d)` 0.12s
|
||||
| 1 | 五维条行序随接口返回序 | 合规风险/行动执行(待数据)排在三实维之前——非固定语义序;批次E 接线 risk/action 后自然消解,如需固定序前端按 dim 顺序数组排即可 |
|
||||
| 2 | 走查任务字面 Step6 措辞偏差 | 「切视图服务质量→点综合分列排序」按字面不可执行(综合分列仅管理概览视图有);实际动线=管理概览排序→切服务质量视图导出,已按实际动线完成走查 |
|
||||
| 3 | 空值显示 `-` 与 `--` 混用 | format.ts 的 formatMsDuration/formatRatio 按各自注释约定 null 显 `-`,员工总览本地数值/比率格式化显 `--`——同表跨列观感不一;纯观感项,批次E 顺手统一即可 |
|
||||
|
||||
---
|
||||
|
||||
# 批次F-调度接入验证(Task F3,2026-09-10)
|
||||
|
||||
F1(`_analytics_tick` 段2 管理域滚动,0dc5993)/ F2(夜间链管理步,2ac63e3)已提交后,本地真机直调 tick 验证 + 本地 scheduler 镜像重建。方法:临时脚本(已删)按 tests/test_scheduler_analytics_tick.py 的构造方式 `SyncScheduler(engine=create_async_engine(get_settings().db_url), session_maker=...)`,不 start、只 `await scheduler._analytics_tick()`,对六个上游函数挂计时包装(与测试 monkeypatch 同一延迟 import 打点位),连跑两次。
|
||||
|
||||
## 1. tick 两段实测(本地真机库,连跑两次)
|
||||
|
||||
| 段 | 内容 | 第1次(冷) | 第2次(暖) |
|
||||
|---|---|---|---|
|
||||
| 总计 | `_analytics_tick` 全程 | **2.78s** | **0.74s** |
|
||||
| 段1 | recompute_days(staff 滚动3天) | 0.86s | 0.15s |
|
||||
| 段1 | scanner.run_incremental(敏感词增量) | 0.14s | 0.02s |
|
||||
| 段2 | pairs_active_in_days(fact 活跃对发现) | 0.04s | 0.02s |
|
||||
| 段2 | recompute_pairs(fact 重算) | 0.00s | 0.00s |
|
||||
| 段2 | recompute_metric_days(宽表滚动3天) | 1.51s | 0.52s |
|
||||
| 段2 | recompute_call_days(通话滚动3天) | 0.22s | 0.02s |
|
||||
|
||||
(段值不含各自 session.commit;日志输出:`staff_days=0 scanned=10 hits=0` + `mgmt fact_pairs=0 wide_rows=168 call_rows=0`,两次一致。)
|
||||
|
||||
- **远低于 <60s 预算**。本地镜像已停同步 7 天,滚动窗(0908~0910)内无消息/通话活动——`staff_days=0`、`fact_pairs=0`、`call_rows=0` 属停同步后的数据真空而非缺陷;`wide_rows=168`= 3 天 × 56 行/天(39 员工,wechat 38 + whatsapp 18 行/天,双渠道员工两行),宽表对零活跃员工照常出零值行。段2 主成本=宽表重算,随员工数×天数扩展而非会话数,服务器侧活跃会话数百的形态下同量级。
|
||||
- **幂等 ✓**:三表行数跑前 fact=68,476 / wide=12,371 / calls=1,163 → 第1次后 68,476 / **12,539** / 1,163 → 第2次后 68,476 / 12,539 / 1,163。wide 的 +168 为首跑补齐:0908~0910 三天行在表内原本不存在(本地夜间链停摆 7 天没写过),第1次 DELETE+INSERT 补齐后第2次行数纹丝不动——滚动窗语义下幂等成立。
|
||||
|
||||
## 2. 本地 scheduler 容器重建(范围仅本地;服务器镜像 2026-09-09 已重建不碰)
|
||||
|
||||
- `docker compose up -d --build scheduler`:镜像 `yunke-scheduler:latest`(`ded96dfc9625`,2026-09-10 构建)重建成功,容器 recreate + 起来。
|
||||
- 冒烟:`docker exec yunke-scheduler-1 python -c "import app.sync.scheduler"` → ok;容器日志确认 **4 个 job 全注册**——3×`_safe_page_run`(full/messages/full-history)+ **`SyncScheduler._analytics_tick`(F1 新 job 已进夜间/4h 链)**,"Scheduler started" 正常。
|
||||
- **调度状态**:验证完即 `docker stop`——重建前容器本就 Exited 7 天(本地不跑调度,避免与服务器调度器双跑同刷云客触发风控),维持 stopped 现状;容器已绑新镜像,用户随时 `docker compose start scheduler` 即以新代码上线。mysql/redis 未动。
|
||||
|
||||
## 3. 已知边界(归档)
|
||||
|
||||
| # | 项 | 说明 |
|
||||
|---|---|---|
|
||||
| 1 | phone_call 未进页面同步链 | 通话小表(employee_call_daily)暂从既有静态源重生成——本地 call_rows=0 即此边界的体现;审计 P0-5 由并行会话领地,不在此处理 |
|
||||
| 2 | 补同步历史日期 | tick 只滚动当日+前2天;更早日期的修正由夜间全量(full_sync 汇总/管理步)覆盖语义兜底 |
|
||||
| 3 | 本地数据真空 | 本地镜像停摆 7 天,0908~0910 无消息/通话——fact/敏感词扫描的真实活跃量级须在服务器侧(夜间链已跑)观察 |
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# 云客经营管理后台·批次F-调度子批(管理指标接入定时链)实施计划
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** 管理域三张派生表(first_reply_fact / employee_metric_daily / employee_call_daily)进入生产定时链——4h 滚动 + 夜间全量自愈,数据每天活起来(同步审计 2026-09-10 结论 #5 的管理域部分)。
|
||||
|
||||
**Architecture:** 两处接线:`scheduler.py _analytics_tick` 加**独立事务**的管理滚动段(近 3 天,失败不回滚既有 staff/敏感词段);`yunke-server/scripts/full_sync_all.sh` 在 11/12 步后加 13/14 步(fact 全量 → 宽表+通话全量)+ 终局快照补三表行数。CLI 已有(first-reply-facts / employee-metrics),无新端点。**边界:不碰审计文档 P0 的脚本加固/实体补抓(并行会话领地)**。
|
||||
|
||||
**Tech Stack:** 同批次A(APScheduler + SQLAlchemy async + bash)。
|
||||
|
||||
---
|
||||
|
||||
## 执行者必读
|
||||
|
||||
1. **上游函数签名(as-built)**:`first_reply_facts.pairs_active_in_days(session, days) / recompute_pairs(session, pairs)`;`storage/employee_metrics.recompute_metric_days(session, days) / recompute_call_days(session, days)`;`staff_summary.rolling_days(today, 3)`(返回当日+前2天列表)。全部幂等 delete+insert。
|
||||
2. **隔离语义(v2 Task 12:指标失败不中断主流程)**:tick 内管理段用**独立 session+commit**——失败只回滚管理段自身,staff/敏感词段已提交不受影响(当前实现是单 try 单事务,改造为两段)。
|
||||
3. **锁竞争既有结论**(tick docstring 已记录):与夜间 full_sync 重叠时 DELETE+INSERT 可能锁等待,非致命,幂等下轮自愈——管理段沿用同一语义,docstring 注明。
|
||||
4. 脚本在 `yunke-server/scripts/full_sync_all.sh`(注意路径已从根 scripts/ 迁移);测试先例 `tests/test_scheduler_analytics_tick.py`。
|
||||
5. 并行会话活动照旧;中文 conventional 提交。
|
||||
|
||||
## 任务
|
||||
|
||||
### Task F1: _analytics_tick 管理段 + 测试
|
||||
|
||||
**Files:** Modify `yunke-server/app/sync/scheduler.py` + `tests/test_scheduler_analytics_tick.py`
|
||||
|
||||
改造 `_analytics_tick`(:703)为两段:
|
||||
|
||||
```python
|
||||
async def _analytics_tick(self) -> None:
|
||||
"""4h 分析链(staff 滚动3天 + 敏感词增量 + 管理域滚动)——异常只记日志不断链。
|
||||
|
||||
两段独立事务:段1 staff+敏感词;段2 管理域(fact 近3天活跃会话 + 宽表/
|
||||
通话 滚动3天,批次F 2026-09-10)。段2 失败只回滚自身,不牵连段1 已
|
||||
提交结果(v2 Task12:指标失败不中断主分析链)。纯 DB 无浏览器,不抢
|
||||
``_page_lock``;与夜间 full_sync 13/14 步重叠时的锁竞争语义同段1
|
||||
(非致命,幂等下轮自愈)。
|
||||
"""
|
||||
# ── 段1:staff 滚动 + 敏感词增量(既有,原样保留) ──
|
||||
try:
|
||||
async with self._session_maker() as session:
|
||||
from app.analytics.sensitive import scanner
|
||||
from app.storage.staff_summary import recompute_days, rolling_days
|
||||
from app.storage.summary import day_key
|
||||
|
||||
today = day_key(int(time.time() * 1000))
|
||||
n = await recompute_days(session, rolling_days(today, 3))
|
||||
scan = await scanner.run_incremental(session)
|
||||
await session.commit()
|
||||
logger.info(
|
||||
"[analytics-tick] staff_days=%s scanned=%s hits=%s",
|
||||
n, scan.get("scanned"), scan.get("hits"),
|
||||
)
|
||||
except Exception: # noqa: BLE001 — scheduler MUST NOT crash
|
||||
logger.exception("analytics tick failed")
|
||||
|
||||
# ── 段2:管理域滚动(fact 活跃会话 + 宽表/通话,当日+滚动3天) ──
|
||||
try:
|
||||
async with self._session_maker() as session:
|
||||
from app.storage.employee_metrics import (
|
||||
recompute_call_days,
|
||||
recompute_metric_days,
|
||||
)
|
||||
from app.storage.first_reply_facts import (
|
||||
pairs_active_in_days,
|
||||
recompute_pairs,
|
||||
)
|
||||
from app.storage.staff_summary import rolling_days
|
||||
from app.storage.summary import day_key
|
||||
|
||||
rd = rolling_days(day_key(int(time.time() * 1000)), 3)
|
||||
pairs = await pairs_active_in_days(session, rd)
|
||||
f = await recompute_pairs(session, pairs)
|
||||
w = await recompute_metric_days(session, rd)
|
||||
c = await recompute_call_days(session, rd)
|
||||
await session.commit()
|
||||
logger.info(
|
||||
"[analytics-tick] mgmt fact_pairs=%s wide_rows=%s call_rows=%s",
|
||||
f, w, c,
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("management metrics tick failed")
|
||||
```
|
||||
|
||||
(导入合并/去重按文件现状微调;today 变量两段各自取或共享只读。注:`recompute_metric_days` 内部会再查一次 `pairs_active_in_days`(与本段显式调用重复但廉价)——非冗余待修,fact 段需要显式拿 pairs 计数记日志。)
|
||||
|
||||
**测试追加**(照既有文件 mock 模式):
|
||||
1. tick 调用顺序:staff recompute_days → 敏感 scan → pairs_active_in_days(rolling3) → recompute_pairs → recompute_metric_days → recompute_call_days;
|
||||
2. **段2 失败隔离**:recompute_metric_days 抛异常 → staff recompute_days 与敏感 scan 仍被调用且各自 commit 成功、tick 不抛、异常被记(caplog);
|
||||
3. rolling 窗口传参 = rolling_days(today,3)。
|
||||
|
||||
- [ ] `pytest tests/test_scheduler_analytics_tick.py -v` + 全量 -q → commit `feat(management): 4h分析链接入管理域滚动段——独立事务隔离/fact活跃会话+宽表+通话滚动3天`
|
||||
|
||||
### Task F2: full_sync_all.sh 13/14 步 + 快照行数
|
||||
|
||||
**Files:** Modify `yunke-server/scripts/full_sync_all.sh`
|
||||
|
||||
第 12 步之后(COMPLETE 横幅之前)加:
|
||||
|
||||
```bash
|
||||
# 13:首响事实表全量重建(管理域,跨日精确串;批次A)——会话发现读
|
||||
# msg_daily_stats(fact 行直算读 message),故须在 11 步后;与 11/12
|
||||
# 同为夜间自愈链(绕过钩子的写入由全量重算纠正)
|
||||
echo "$(date) --- 13/14 First-reply-facts rebuild ---"
|
||||
$PY -m app.sync.cli first-reply-facts
|
||||
# 14:管理宽表+通话小表全量重建(驾驶舱/员工页读路径;串类列与 fact 同
|
||||
# 算法各自直算,放 fact 后保持派生链顺序语义)
|
||||
echo "$(date) --- 14/14 Employee-metrics rebuild ---"
|
||||
$PY -m app.sync.cli employee-metrics
|
||||
```
|
||||
|
||||
终局快照 TABLES 追加三行(放 daily stats rows 后):
|
||||
|
||||
```python
|
||||
("first reply facts", "SELECT COUNT(*) FROM first_reply_fact"),
|
||||
("employee metric rows", "SELECT COUNT(*) FROM employee_metric_daily"),
|
||||
("employee call rows", "SELECT COUNT(*) FROM employee_call_daily"),
|
||||
```
|
||||
|
||||
- [ ] `bash -n yunke-server/scripts/full_sync_all.sh` 语法过 + commit `feat(management): 夜间全量脚本13/14步——fact与宽表全量自愈+终局快照三表行数`
|
||||
|
||||
### Task F3: 本地 scheduler 容器重建 + 真机验证 + 归档
|
||||
|
||||
1. **真机验证滚动段**(直接跑 tick 逻辑,不等 4h):临时脚本或 `python -c` 用 scheduler 的 session_maker 调 `_analytics_tick()`(或手动等价:按 rolling 3 天调三个 recompute + commit)——记录耗时(预期 fact 活跃会话数十秒级内)+ 三表当日行数变化(recompute 后 updated 行 freshness);
|
||||
2. **幂等复跑**:连跑两次行数一致;
|
||||
3. **本地 scheduler 容器重建**(新代码进夜间链):`docker ps` 确认 scheduler 容器名 → `docker compose -f docker-compose.yml up -d --build scheduler`——**范围仅本地**(服务器 scheduler 镜像 2026-09-09 已重建过,服务器侧等部署时随 DEPLOY.md 流程走,本任务不碰);重建前后 `docker exec <scheduler> python -c "import app.sync.scheduler"` 冒烟;(已知后果不阻塞:phone_call 尚未进 4h/每日页面同步链[审计P0-5,并行会话领地]——通话小表滚动重算在补齐前只是从静态源幂等重生成);
|
||||
4. 归档:耗时/行数/容器重建记录 → 核实文档「批次F-调度」小节 + commit。
|
||||
|
||||
- [ ] commit `docs(management): 批次F调度接入验证——滚动段耗时/幂等/容器重建(清9d3cc8e旧账)`
|
||||
|
||||
## 验收清单
|
||||
|
||||
- [ ] tick 两段隔离测试过(段2 异常不牵连段1);
|
||||
- [ ] 夜间脚本 13/14 步 + 快照三表行数;
|
||||
- [ ] 滚动段真机耗时记录(预期 <60s)+ 幂等;
|
||||
- [ ] 本地 scheduler 容器已重建(新代码),夜间 02:00 链将含管理域;
|
||||
- [ ] 全量测试绿;
|
||||
- [ ] 未触碰审计 P0 领地(脚本加固/实体补抓——并行会话负责)。
|
||||
|
||||
## 后续(不在本子批)
|
||||
|
||||
E:辅导+快照;F 余项:dashboard 性能修复(unreplied 索引)、winsorize 决策、服务器部署(届时服务器 scheduler 镜像重建走 DEPLOY.md)。
|
||||
@@ -101,14 +101,37 @@ async def list_groups(
|
||||
list_stmt = sa.select(ChatGroup)
|
||||
if conditions:
|
||||
list_stmt = list_stmt.where(*conditions)
|
||||
list_stmt = (
|
||||
list_stmt.order_by(
|
||||
sa.nullslast(ChatGroup.last_chat_time.desc()),
|
||||
ChatGroup.room_id.asc(),
|
||||
if (channel or "wechat") == "wechat":
|
||||
# 云客「微信群统计」同款排序(2026-09-10 真机终判):owner_account_id
|
||||
# (=微信号/wxid 混存,如 Nancyjade < csf7169708 < wxid_01gv…——大写
|
||||
# 在小写前=区分大小写)**二进制字节序升序**分组,组内 room_id 字符串
|
||||
# 升序。字节序按方言:MySQL 用 CAST(col AS BINARY)(collate("BINARY")
|
||||
# 会被 SQLAlchemy 反引号成标识符);SQLite 用 COLLATE BINARY(其 CAST
|
||||
# AS BINARY 语义不符)。WA 保持最近活跃优先。
|
||||
dialect = session.bind.dialect.name if session.bind is not None else ""
|
||||
if dialect == "mysql":
|
||||
owner_order = sa.literal_column(
|
||||
"CAST(chat_group.owner_account_id AS BINARY)"
|
||||
).asc()
|
||||
else:
|
||||
owner_order = ChatGroup.owner_account_id.collate("BINARY").asc()
|
||||
list_stmt = (
|
||||
list_stmt.order_by(
|
||||
owner_order,
|
||||
ChatGroup.room_id.asc(),
|
||||
)
|
||||
.limit(page_size)
|
||||
.offset((page - 1) * page_size)
|
||||
)
|
||||
else:
|
||||
list_stmt = (
|
||||
list_stmt.order_by(
|
||||
sa.nullslast(ChatGroup.last_chat_time.desc()),
|
||||
ChatGroup.room_id.asc(),
|
||||
)
|
||||
.limit(page_size)
|
||||
.offset((page - 1) * page_size)
|
||||
)
|
||||
.limit(page_size)
|
||||
.offset((page - 1) * page_size)
|
||||
)
|
||||
rows = (await session.execute(list_stmt)).scalars().all()
|
||||
|
||||
# ---- 员工微信/员工信息列(wechat_account/staff 关联,本页行集) ----
|
||||
|
||||
@@ -537,6 +537,18 @@ class Message(Base):
|
||||
"ix_message_fetched_at",
|
||||
"fetched_at",
|
||||
),
|
||||
# 该回未回决赛圈点查(owner+talker 定位末条消息)在 channel=None
|
||||
# (全平台计数——管理驾驶舱 action_counts 复用)时无渠道前缀条件,
|
||||
# 上面全部以 channel 打头的索引用不上,逐候选退化扫表+filesort:
|
||||
# 2026-09-10 服务器实测 dashboard 34~37s 撞前端 15s 超时
|
||||
# (ECONNABORTED);补无渠道前缀索引后 1.3~1.8s(100 万行在线建
|
||||
# 索引 49s)。「模型+迁移双声明」防 alembic autogenerate 反 DROP。
|
||||
Index(
|
||||
"ix_message_owner_talker_time",
|
||||
"owner_account_id",
|
||||
"talker_id",
|
||||
"timestamp",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -701,25 +701,27 @@ class SyncScheduler:
|
||||
print(line, flush=True)
|
||||
|
||||
async def _analytics_tick(self) -> None:
|
||||
"""4h 分析链(stat_daily_staff 滚动3天 + 敏感词增量)——异常只记日志不断链。
|
||||
"""4h 分析链(staff 滚动3天 + 敏感词增量 + 管理域滚动)——异常只记日志不断链。
|
||||
|
||||
纯 DB 无浏览器子进程,不抢 ``_page_lock``(DB 写毫秒级;
|
||||
``max_instances=1`` 防自身重叠即可)。与夜间 full_sync 的
|
||||
staff-daily-stats 步骤重叠时,两侧 DELETE+INSERT 在重叠 PK 范围上
|
||||
可能死锁/锁等待——非致命:tick 吞异常下轮重试,夜间脚本无 set -e
|
||||
次日自愈(UNIQUE/PK 幂等只保证串行提交下不重复,不消锁竞争)。
|
||||
两段独立事务:段1 staff+敏感词(既有);段2 管理域(fact 近3天活跃会话
|
||||
+ 宽表/通话 滚动3天,批次F 2026-09-10)。段2 失败只回滚自身,不牵连
|
||||
段1 已提交结果(v2 Task12:指标失败不中断主分析链)。纯 DB 无浏览器
|
||||
子进程,不抢 ``_page_lock``;``max_instances=1`` 防自身重叠即可。与
|
||||
夜间 full_sync 的汇总/管理步重叠时,两侧 DELETE+INSERT 在重叠 PK
|
||||
范围可能锁等待——非致命:tick 吞异常下轮重试,夜间脚本幂等自愈。
|
||||
|
||||
时区:``day_key`` 内部钉死 Asia/Shanghai,喂 UTC epoch
|
||||
(``time.time()*1000``)即无歧义——用 epoch 而非 naive datetime
|
||||
只为读法直接,数学上两者经本地时区往返等价。
|
||||
(``time.time()*1000``)即无歧义。
|
||||
"""
|
||||
from app.storage.summary import day_key
|
||||
|
||||
today = day_key(int(time.time() * 1000))
|
||||
# ── 段1:staff 滚动 + 敏感词增量(既有,语义原样) ──
|
||||
try:
|
||||
async with self._session_maker() as session:
|
||||
from app.analytics.sensitive import scanner
|
||||
from app.storage.staff_summary import recompute_days, rolling_days
|
||||
from app.storage.summary import day_key
|
||||
|
||||
today = day_key(int(time.time() * 1000))
|
||||
n = await recompute_days(session, rolling_days(today, 3))
|
||||
scan = await scanner.run_incremental(session)
|
||||
await session.commit()
|
||||
@@ -730,6 +732,32 @@ class SyncScheduler:
|
||||
except Exception: # noqa: BLE001 — scheduler MUST NOT crash
|
||||
logger.exception("analytics tick failed")
|
||||
|
||||
# ── 段2:管理域滚动(fact 活跃会话 + 宽表/通话,当日+滚动3天) ──
|
||||
try:
|
||||
async with self._session_maker() as session:
|
||||
from app.storage.employee_metrics import (
|
||||
recompute_call_days,
|
||||
recompute_metric_days,
|
||||
)
|
||||
from app.storage.first_reply_facts import (
|
||||
pairs_active_in_days,
|
||||
recompute_pairs,
|
||||
)
|
||||
from app.storage.staff_summary import rolling_days
|
||||
|
||||
rd = rolling_days(today, 3)
|
||||
pairs = await pairs_active_in_days(session, rd)
|
||||
f = await recompute_pairs(session, pairs)
|
||||
w = await recompute_metric_days(session, rd)
|
||||
c = await recompute_call_days(session, rd)
|
||||
await session.commit()
|
||||
logger.info(
|
||||
"[analytics-tick] mgmt fact_pairs=%s wide_rows=%s call_rows=%s",
|
||||
f, w, c,
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("management metrics tick failed")
|
||||
|
||||
async def _page_subprocess(self, args: list[str], *, timeout: float | None = None) -> int:
|
||||
"""启动页面子进程并等待结束;继承环境 + 补 DB_URL/PYTHONIOENCODING。
|
||||
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
"""message 补无渠道前缀索引 (owner_account_id, talker_id, timestamp)
|
||||
|
||||
Revision ID: d8e9f0a1b2c3
|
||||
Revises: c9d0e1f2a3b4
|
||||
Create Date: 2026-09-10
|
||||
|
||||
背景:该回未回决赛圈点查在 channel=None(管理驾驶舱 action_counts)时
|
||||
无渠道前缀条件,现有索引全部以 channel 打头用不上——服务器 100 万行
|
||||
实测 dashboard 34~37s 撞前端 15s 超时;索引落地后 1.3~1.8s。
|
||||
幂等:sa.inspect 先查,存在即跳过(服务器已先行手工建同名索引,本迁移
|
||||
兜底对齐 schema 版本)。「模型+迁移双声明」防 autogenerate 反 DROP。
|
||||
"""
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "d8e9f0a1b2c3"
|
||||
down_revision: str | Sequence[str] | None = "c9d0e1f2a3b4"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
_IDX = "ix_message_owner_talker_time"
|
||||
|
||||
|
||||
def _has_index(bind) -> bool:
|
||||
insp = sa.inspect(bind)
|
||||
if not insp.has_table("message"):
|
||||
return False
|
||||
return _IDX in {ix["name"] for ix in insp.get_indexes("message")}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_index(op.get_bind()):
|
||||
op.create_index(_IDX, "message",
|
||||
["owner_account_id", "talker_id", "timestamp"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_index(op.get_bind()):
|
||||
op.drop_index(_IDX, table_name="message")
|
||||
@@ -59,6 +59,15 @@ $PY -m app.sync.cli daily-stats
|
||||
# A1 员工指标读路径依赖此表,2026-09-04)
|
||||
echo "$(date) --- 12/12 Staff-daily-stats rebuild ---"
|
||||
$PY -m app.sync.cli staff-daily-stats
|
||||
# 13:首响事实表全量重建(管理域,跨日精确串;批次A)——会话发现读
|
||||
# msg_daily_stats(fact 行直算读 message),故须在 11 步后;与 11/12
|
||||
# 同为夜间自愈链(绕过钩子的写入由全量重算纠正)
|
||||
echo "$(date) --- 13/14 First-reply-facts rebuild ---"
|
||||
$PY -m app.sync.cli first-reply-facts
|
||||
# 14:管理宽表+通话小表全量重建(驾驶舱/员工页读路径;串类列与 fact 同
|
||||
# 算法各自直算,放 fact 后保持派生链顺序语义)
|
||||
echo "$(date) --- 14/14 Employee-metrics rebuild ---"
|
||||
$PY -m app.sync.cli employee-metrics
|
||||
|
||||
echo "=============================================="
|
||||
echo "$(date) === FULL RE-SYNC COMPLETE ==="
|
||||
@@ -84,6 +93,9 @@ TABLES = [
|
||||
("wechat messages", "SELECT COUNT(*) FROM message WHERE channel='wechat'"),
|
||||
("whatsapp messages", "SELECT COUNT(*) FROM message WHERE channel='whatsapp'"),
|
||||
("daily stats rows", "SELECT COUNT(*) FROM msg_daily_stats"),
|
||||
("first reply facts", "SELECT COUNT(*) FROM first_reply_fact"),
|
||||
("employee metric rows", "SELECT COUNT(*) FROM employee_metric_daily"),
|
||||
("employee call rows", "SELECT COUNT(*) FROM employee_call_daily"),
|
||||
("voice calls", "SELECT COUNT(*) FROM phone_call WHERE channel IN ('voice_call','wa_voice_call')"),
|
||||
("voice clips", "SELECT COUNT(*) FROM voice_clip"),
|
||||
("moments posts", "SELECT COUNT(*) FROM moments_post"),
|
||||
|
||||
@@ -1889,25 +1889,29 @@ async def scrape_chatroom_stats(cookie_str: str, day: str | None = None):
|
||||
# 误删;只动 wechat(WA 群另一来源);GroupMember 按 room 共享不动。
|
||||
pruned = 0
|
||||
if day_key is None and group_recs and declared and len(group_recs) >= declared:
|
||||
from sqlalchemy import tuple_ as sa_tuple
|
||||
import sqlalchemy as sa
|
||||
|
||||
from app.db.base import async_session_maker
|
||||
from app.db.models import ChatGroup
|
||||
|
||||
keep = {(r["owner_account_id"], r["room_id"]) for r in group_recs}
|
||||
keep_list = list(keep)
|
||||
async with async_session_maker() as session:
|
||||
for i in range(0, len(keep_list), 500):
|
||||
chunk = keep_list[i:i + 500]
|
||||
res = await session.execute(
|
||||
local = (await session.execute(
|
||||
sa.select(ChatGroup.owner_account_id, ChatGroup.room_id)
|
||||
.where(ChatGroup.channel == "wechat")
|
||||
)).all()
|
||||
stale = [(o, r) for (o, r) in local if (o, r) not in keep]
|
||||
for owner, room in stale:
|
||||
# 逐行删(差集常态 0~几行;不用 tuple IN——asyncmy 对其参数
|
||||
# 形态挑剔,实测 TypeError)
|
||||
await session.execute(
|
||||
sa.delete(ChatGroup).where(
|
||||
ChatGroup.channel == "wechat",
|
||||
sa.tuple_(
|
||||
ChatGroup.owner_account_id, ChatGroup.room_id
|
||||
).not_in(sa_tuple(chunk)),
|
||||
ChatGroup.owner_account_id == owner,
|
||||
ChatGroup.room_id == room,
|
||||
)
|
||||
)
|
||||
pruned += int(res.rowcount or 0)
|
||||
pruned = len(stale)
|
||||
await session.commit()
|
||||
|
||||
logger.info(
|
||||
|
||||
@@ -64,17 +64,19 @@ async def _seed(session: AsyncSession) -> None:
|
||||
|
||||
|
||||
async def test_default_list_returns_all_ordered_desc(make_client):
|
||||
"""No filters -> 3 groups, ordered by ``last_chat_time desc`` (nulls last)."""
|
||||
"""No filters -> 3 groups;微信排序=owner 二进制字符串升序分组+room_id
|
||||
升序(云客群统计同款,2026-09-10 终判:区分大小写,大写排小写前)。
|
||||
owner 序:wa_owner_1 < wx_owner_1 < wx_owner_2 → [wa_room_x, room_a, room_b]。
|
||||
"""
|
||||
client = await make_client(_seed)
|
||||
r = await client.get("/api/groups")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["total"] == 3
|
||||
# 2000, 1000, null -> [room_a, room_b, wa_room_x]
|
||||
assert [it["room_id"] for it in body["items"]] == [
|
||||
"wa_room_x@s.whatsapp.net",
|
||||
"room_a@chatroom",
|
||||
"room_b@chatroom",
|
||||
"wa_room_x@s.whatsapp.net",
|
||||
]
|
||||
for key in (
|
||||
"room_id",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""_analytics_tick:滚动刷新 + 增量扫描 + 异常吞掉(数据分析板块 v1, T11)。
|
||||
"""_analytics_tick:滚动刷新 + 增量扫描 + 管理域滚动段 + 异常吞掉(v1 T11;批次F F1)。
|
||||
|
||||
行为测试是底线——start() 的 job 注册断言在 test_sync_scheduler.py 的
|
||||
注册测试里同步更新(那里 spy 断言 add_job 全集)。
|
||||
@@ -15,6 +15,7 @@ from sqlalchemy.pool import StaticPool
|
||||
|
||||
from app.db.base import Base
|
||||
from app.db.models import MsgDailyStats, SensitiveWord, StatDailyStaff
|
||||
from app.storage.staff_summary import rolling_days
|
||||
from app.storage.summary import day_key
|
||||
from app.sync.scheduler import SyncScheduler
|
||||
|
||||
@@ -74,3 +75,110 @@ async def test_analytics_tick_swallows_errors(sched_engine, monkeypatch, caplog)
|
||||
with caplog.at_level(logging.ERROR, logger="app.sync.scheduler"):
|
||||
await sch._analytics_tick() # 不抛
|
||||
assert any("analytics tick failed" in r.getMessage() for r in caplog.records)
|
||||
|
||||
|
||||
async def _seed_today(sched_engine) -> int:
|
||||
"""seed 当日一条 wechat friend 会话 + 敏感词,返回 seed 用的 day 键。"""
|
||||
_, maker = sched_engine
|
||||
today = day_key(int(time.time() * 1000))
|
||||
async with maker() as s:
|
||||
s.add(MsgDailyStats(channel="wechat", owner_account_id="wx1",
|
||||
peer_kind="friend", peer_id="t1", day=today,
|
||||
send_cnt=1, recv_cnt=0, last_ts=1756950000000,
|
||||
last_is_send=True))
|
||||
s.add(SensitiveWord(word="私单"))
|
||||
await s.commit()
|
||||
return today
|
||||
|
||||
|
||||
async def test_tick_calls_management_chain_in_order(sched_engine, monkeypatch):
|
||||
"""两段链完整顺序:recompute_days → scan → pairs_active_in_days →
|
||||
recompute_pairs → recompute_metric_days → recompute_call_days。
|
||||
|
||||
记录桩替换全部六个上游——_analytics_tick 段内函数延迟 import,
|
||||
monkeypatch 模块属性即命中(惯例见 test_analytics_tick_swallows_errors)。
|
||||
"""
|
||||
engine, maker = sched_engine
|
||||
calls: list[str] = []
|
||||
|
||||
def rec(name: str, ret):
|
||||
async def _stub(*args, **kwargs):
|
||||
calls.append(name)
|
||||
return ret
|
||||
return _stub
|
||||
|
||||
monkeypatch.setattr("app.storage.staff_summary.recompute_days",
|
||||
rec("recompute_days", 0))
|
||||
monkeypatch.setattr("app.analytics.sensitive.scanner.run_incremental",
|
||||
rec("scan", {"scanned": 0, "hits": 0}))
|
||||
monkeypatch.setattr("app.storage.first_reply_facts.pairs_active_in_days",
|
||||
rec("pairs_active_in_days", []))
|
||||
monkeypatch.setattr("app.storage.first_reply_facts.recompute_pairs",
|
||||
rec("recompute_pairs", 0))
|
||||
monkeypatch.setattr("app.storage.employee_metrics.recompute_metric_days",
|
||||
rec("recompute_metric_days", 0))
|
||||
monkeypatch.setattr("app.storage.employee_metrics.recompute_call_days",
|
||||
rec("recompute_call_days", 0))
|
||||
|
||||
sch = SyncScheduler(engine=engine, session_maker=maker)
|
||||
await sch._analytics_tick()
|
||||
assert calls == [
|
||||
"recompute_days", "scan", # 段1
|
||||
"pairs_active_in_days", "recompute_pairs", # 段2
|
||||
"recompute_metric_days", "recompute_call_days",
|
||||
]
|
||||
|
||||
|
||||
async def test_tick_management_failure_isolated(sched_engine, monkeypatch, caplog):
|
||||
"""段2 失败隔离:recompute_metric_days 抛 RuntimeError → ①tick 不抛;
|
||||
②记 "management metrics tick failed";③段1 已提交的 StatDailyStaff
|
||||
不被段2 回滚(两段独立事务)。
|
||||
"""
|
||||
engine, maker = sched_engine
|
||||
await _seed_today(sched_engine)
|
||||
|
||||
async def _boom(*args, **kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr("app.storage.employee_metrics.recompute_metric_days", _boom)
|
||||
sch = SyncScheduler(engine=engine, session_maker=maker)
|
||||
with caplog.at_level(logging.ERROR, logger="app.sync.scheduler"):
|
||||
await sch._analytics_tick() # ①不抛
|
||||
assert any( # ②段2 独立记日志
|
||||
"management metrics tick failed" in r.getMessage() for r in caplog.records
|
||||
)
|
||||
async with maker() as s: # ③段1 结果已提交
|
||||
n = (await s.execute(
|
||||
select(func.count()).select_from(StatDailyStaff)
|
||||
)).scalar_one()
|
||||
assert n >= 1
|
||||
|
||||
|
||||
async def test_tick_management_rolling_window(sched_engine, monkeypatch):
|
||||
"""管理域三函数收到的 days == rolling_days(today, 3)(长度 3,当日+前
|
||||
两天),且与 staff 段捕获的窗口一致(两段共用函数顶提取的同一 today)。
|
||||
"""
|
||||
engine, maker = sched_engine
|
||||
got: dict[str, list[int]] = {}
|
||||
|
||||
def cap(name: str, ret):
|
||||
async def _stub(session, days, *args, **kwargs):
|
||||
got[name] = list(days)
|
||||
return ret
|
||||
return _stub
|
||||
|
||||
monkeypatch.setattr("app.storage.staff_summary.recompute_days",
|
||||
cap("staff", 0))
|
||||
monkeypatch.setattr("app.storage.first_reply_facts.pairs_active_in_days",
|
||||
cap("pairs", []))
|
||||
monkeypatch.setattr("app.storage.employee_metrics.recompute_metric_days",
|
||||
cap("wide", 0))
|
||||
monkeypatch.setattr("app.storage.employee_metrics.recompute_call_days",
|
||||
cap("call", 0))
|
||||
|
||||
sch = SyncScheduler(engine=engine, session_maker=maker)
|
||||
await sch._analytics_tick()
|
||||
|
||||
rd = rolling_days(day_key(int(time.time() * 1000)), 3)
|
||||
assert len(rd) == 3 # 当日 + 前两天
|
||||
assert got["staff"] == got["pairs"] == got["wide"] == got["call"] == rd
|
||||
|
||||
@@ -105,7 +105,11 @@ export const GROUP_FIELD_GROUPS: GroupFieldGroup[] = [
|
||||
]
|
||||
|
||||
/** Every key already surfaced via a group (so 其他 can skip duplicates). */
|
||||
const KNOWN_KEYS = new Set(GROUP_FIELD_GROUPS.flatMap((g) => g.keys))
|
||||
const KNOWN_KEYS = new Set([
|
||||
...GROUP_FIELD_GROUPS.flatMap((g) => g.keys),
|
||||
// 原始 wxid 串由群详情成员墙替代,不在任何区(含「其他」)显示(2026-09-10)
|
||||
'memberIds',
|
||||
])
|
||||
|
||||
/** Resolve the 中文 label for a raw_json key (falls back to the key). */
|
||||
export function fieldLabel(key: string): string {
|
||||
|
||||
Reference in New Issue
Block a user