perf(api): 会话/5数字弹窗超时修复——日粒度汇总层 msg_daily_stats

根因(服务器实测):sessions 对 owner 全量 message GROUP BY,重账号 5.2万条
3~5s;staff-chat-stats 全时段窗 6.9s——128MB buffer pool 下被推过前端
15s 超时(ECONNABORTED)。EXPLAIN ANALYZE 证实最优 covering 索引也要
18.9s,瓶颈是磁盘不是查询形状。

修复(预聚合,云客同款思路):
- msg_daily_stats 表:每「渠道×账号×会话×上海自然日」一行(收发计数/
  当日最后时间与方向);迁移 a7d3e9f1c2b4(幂等建表)
- 写路径:repos.upsert_messages 落库自动重算本批触及的会话×天(所有
  抓取路径汇到该函数,一处挂全;失败不阻塞写入)
- 读路径:sessions/staff-chat-stats 表有行时读汇总(namedtuple 同形,
  端点主体零改动),空表守卫回退旧聚合(部署回填前不空白)
- 自愈:CLI daily-stats 全量重建(幂等);full_sync_all.sh 夜间追加第 11
  步——顺带修掉尾部 sqlite3 直连 smoke.db 的硬编码(服务器 MySQL 上会炸)
- 口径统一:day 一律 Asia/Shanghai Python 计算(旧 from_unixtime 跟随
  DB 会话时区与今日数割裂);统计窗口对齐自然日(文档化语义变化)

测试:12 新增(汇总层 8 + 端点判别/对拍 4,判别手法=回填后删光 message
仍出完整数据),全量 512 绿。

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
浅念
2026-09-03 16:37:04 +08:00
parent e3f7aa1304
commit 421998be54
11 changed files with 892 additions and 32 deletions
+65 -6
View File
@@ -1,10 +1,17 @@
"""GET /api/sessions — 账号会话列表(好友+群,今日消息数+最近联系时间),全本地 message 聚合。
"""GET /api/sessions — 账号会话列表(好友+群,今日消息数+最近联系时间)。
镜像 staff_chat_stats 聚合模式(GROUP BY + case 计数 + Python 侧合并昵称/头像)。
today_start 钉死 Asia/Shanghai(UTC+8)当日 0 点,不依赖服务器时区。
**双路径(2026-09-03 会话列表超时修复)**:
- ``msg_daily_stats`` 有行 → 读日粒度汇总(每会话「今日行 + MAX(last_ts)」,
毫秒级;旧实时全量聚合在重账号上实测 3~5s,128MB buffer pool 下更被推过
前端 15s 超时)。
- 空表(部署回填前守卫)→ 回退旧实时聚合,行为不变。
today_start 钉死 Asia/Shanghai(UTC+8)当日 0 点,不依赖服务器时区;汇总层
``day_key`` 同口径。
"""
from __future__ import annotations
from collections import namedtuple
from datetime import datetime
from typing import Annotated
from zoneinfo import ZoneInfo
@@ -15,7 +22,8 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api.deps import get_session
from app.api.schemas import SessionListOut, SessionOut
from app.db.models import ChatGroup, Customer, Message
from app.db.models import ChatGroup, Customer, Message, MsgDailyStats
from app.storage.summary import day_key, has_rows as _summary_has_rows
router = APIRouter(prefix="/api", tags=["sessions"])
@@ -63,6 +71,51 @@ async def _group_rows(session: AsyncSession, owner: str, channel: str, today_sta
return (await session.execute(stmt)).all()
# ── 汇总路径:msg_daily_stats 日行(namedtuple 与旧 Row 同形,端点主体零改动)── #
_FriendRow = namedtuple("_FriendRow", ["talker_id", "last_chat_time", "today_count"])
_GroupRow = namedtuple("_GroupRow", ["room_id", "last_chat_time", "today_count"])
async def _summary_rows(
session: AsyncSession, owner: str, channel: str, kind: str, today: int
) -> dict[str, tuple[int | None, int]]:
"""peer → (last_chat_time, today_count),来自日行聚合。
last = MAX(last_ts)(日行内时间单调,跨日最大即全史最后一条);
today = 当日行的 send+recv。
"""
base = (
MsgDailyStats.channel == channel,
MsgDailyStats.owner_account_id == owner,
MsgDailyStats.peer_kind == kind,
)
last = {
peer: last_ts
for peer, last_ts in (
await session.execute(
sa.select(MsgDailyStats.peer_id, sa.func.max(MsgDailyStats.last_ts))
.where(*base)
.group_by(MsgDailyStats.peer_id)
)
).all()
}
today_counts = {
peer: int(cnt or 0)
for peer, cnt in (
await session.execute(
sa.select(
MsgDailyStats.peer_id,
sa.func.coalesce(sa.func.sum(MsgDailyStats.send_cnt + MsgDailyStats.recv_cnt), 0),
)
.where(*base, MsgDailyStats.day == today)
.group_by(MsgDailyStats.peer_id)
)
).all()
}
return {peer: (last_ts, today_counts.get(peer, 0)) for peer, last_ts in last.items()}
async def _meta(session: AsyncSession, owner: str, channel: str, friend_ids: list[str], room_ids: list[str]):
"""查 customer/chat_group 表合并昵称/头像;好友行额外带 delete_flag(is_del 过滤用)。
@@ -105,8 +158,14 @@ async def list_sessions(
``pinned``:filehelper 会话置顶标记(对齐云客置顶区),其余好友/群恒 False。
"""
today_start = _today_start_ms()
f_rows = await _friend_rows(session, owner_account_id, channel, today_start)
g_rows = await _group_rows(session, owner_account_id, channel, today_start)
if await _summary_has_rows(session):
f_map = await _summary_rows(session, owner_account_id, channel, "friend", day_key(today_start))
g_map = await _summary_rows(session, owner_account_id, channel, "room", day_key(today_start))
f_rows = [_FriendRow(talker_id=p, last_chat_time=t, today_count=c) for p, (t, c) in f_map.items()]
g_rows = [_GroupRow(room_id=p, last_chat_time=t, today_count=c) for p, (t, c) in g_map.items()]
else:
f_rows = await _friend_rows(session, owner_account_id, channel, today_start)
g_rows = await _group_rows(session, owner_account_id, channel, today_start)
fnames, gnames = await _meta(
session, owner_account_id, channel, [r.talker_id for r in f_rows], [r.room_id for r in g_rows]
)