perf(api): 会话/5数字弹窗超时修复——日粒度汇总层 msg_daily_stats

根因(服务器实测):sessions 对 owner 全量 message GROUP BY,重账号 5.2万条
3~5s;staff-chat-stats 全时段窗 6.9s——128MB buffer pool 下被推过前端
15s 超时(ECONNABORTED)。EXPLAIN ANALYZE 证实最优 covering 索引也要
18.9s,瓶颈是磁盘不是查询形状。

修复(预聚合,云客同款思路):
- msg_daily_stats 表:每「渠道×账号×会话×上海自然日」一行(收发计数/
  当日最后时间与方向);迁移 a7d3e9f1c2b4(幂等建表)
- 写路径:repos.upsert_messages 落库自动重算本批触及的会话×天(所有
  抓取路径汇到该函数,一处挂全;失败不阻塞写入)
- 读路径:sessions/staff-chat-stats 表有行时读汇总(namedtuple 同形,
  端点主体零改动),空表守卫回退旧聚合(部署回填前不空白)
- 自愈:CLI daily-stats 全量重建(幂等);full_sync_all.sh 夜间追加第 11
  步——顺带修掉尾部 sqlite3 直连 smoke.db 的硬编码(服务器 MySQL 上会炸)
- 口径统一:day 一律 Asia/Shanghai Python 计算(旧 from_unixtime 跟随
  DB 会话时区与今日数割裂);统计窗口对齐自然日(文档化语义变化)

测试:12 新增(汇总层 8 + 端点判别/对拍 4,判别手法=回填后删光 message
仍出完整数据),全量 512 绿。

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
浅念
2026-09-03 16:37:04 +08:00
parent e3f7aa1304
commit 421998be54
11 changed files with 892 additions and 32 deletions
@@ -1,12 +1,17 @@
"""GET /api/staff-chat-stats — 微信员工 5 数字 + 客户明细(本地 message 表实时聚合)。
"""GET /api/staff-chat-stats — 微信员工 5 数字 + 客户明细。
**双路径(2026-09-03 全时段窗 6.9s 超时修复)**:``msg_daily_stats`` 有行时
读日粒度汇总(窗口对齐到自然日,毫秒级);空表回退旧实时聚合,行为不变。
窗口语义随之统一为「上海自然日」(旧 SQL 日期函数跟随 DB 会话时区,UTC
容器下与今日数口径割裂——顺带修复)。
数据全本地:按 员工微信 wxid × ms-epoch 日期窗 聚合 1:1 私聊(room_id 过滤),
不访问云客。5 汇总数从 customers 在 Python 侧派生(spec §5)。
跨库日期提取(codebase 首例)镜像 app/storage/repos.py 的方言检测写法。
"""
from __future__ import annotations
import time
from collections import namedtuple
from typing import Annotated
import sqlalchemy as sa
@@ -15,7 +20,8 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api.deps import get_session
from app.api.schemas import StaffChatCustomerOut, StaffChatStatsOut, StaffChatStatsSummary
from app.db.models import Customer, Message
from app.db.models import Customer, Message, MsgDailyStats
from app.storage.summary import day_key, has_rows as _summary_has_rows
router = APIRouter(prefix="/api", tags=["staff-chat-stats"])
@@ -81,6 +87,62 @@ async def _aggregate(session: AsyncSession, owner: str, start: int, end: int) ->
return agg_rows, last_map
# ── 汇总路径:msg_daily_stats 日行(namedtuple 与旧 Row 同形)────────────── #
_AggRow = namedtuple(
"_AggRow", ["talker_id", "send_count", "receive_count", "interaction_days", "last_time"]
)
async def _aggregate_summary(
session: AsyncSession, owner: str, start: int, end: int
) -> tuple[list, dict[str, bool | None]]:
"""窗口 → [day_key(start), day_key(end)] 自然日区间,按日行 Python 聚合。
interaction_days = 窗内有消息的自然日数;last_time/last_is_send 取窗内
最后一条(last_ts 最大的日行)。窗口对齐自然日是文档化的语义变化
(前端日期选择器本就按天取值)。
"""
rows = (
await session.execute(
sa.select(
MsgDailyStats.peer_id,
MsgDailyStats.day,
MsgDailyStats.send_cnt,
MsgDailyStats.recv_cnt,
MsgDailyStats.last_ts,
MsgDailyStats.last_is_send,
).where(
MsgDailyStats.channel == "wechat",
MsgDailyStats.owner_account_id == owner,
MsgDailyStats.peer_kind == "friend",
MsgDailyStats.day >= day_key(start),
MsgDailyStats.day <= day_key(end),
)
)
).all()
per: dict[str, dict] = {}
for peer, _day, send, recv, last_ts, last_is in rows:
acc = per.setdefault(
peer, {"send": 0, "recv": 0, "days": 0, "last_ts": None, "last_is": None}
)
acc["send"] += int(send or 0)
acc["recv"] += int(recv or 0)
acc["days"] += 1
if acc["last_ts"] is None or (last_ts is not None and last_ts >= acc["last_ts"]):
acc["last_ts"] = last_ts
acc["last_is"] = last_is
agg_rows = [
_AggRow(talker_id=p, send_count=a["send"], receive_count=a["recv"],
interaction_days=a["days"], last_time=a["last_ts"])
for p, a in per.items()
]
last_map = {p: a["last_is"] for p, a in per.items()}
return agg_rows, last_map
async def _customer_names(session: AsyncSession, owner: str, talker_ids: list[str]) -> dict[str, tuple[str, str, str]]:
if not talker_ids:
return {}
@@ -114,7 +176,10 @@ async def get_staff_chat_stats(
if start is None:
start = end - 3 * _MS_PER_DAY # 默认近 3 天
agg_rows, last_map = await _aggregate(session, owner_account_id, start, end)
if await _summary_has_rows(session):
agg_rows, last_map = await _aggregate_summary(session, owner_account_id, start, end)
else:
agg_rows, last_map = await _aggregate(session, owner_account_id, start, end)
talker_ids = [r.talker_id for r in agg_rows]
names = await _customer_names(session, owner_account_id, talker_ids)