feat(msg): 类型归一化扩展(spike判别+49拆红包/转账)+存量回填+API msg_type/q过滤

- _normalize_msg_type 升级 (raw_type, row): messageType 经典码优先 →
  49/appmsg 行级拆分(红包/转账/聊天记录) → 云客码兜底
- spike 修正: 云客 type=3 实为语音(messageType=34, 8754/8955 行),
  原 image_legacy 是误标; type_raw=3 的 201 行旧数据才是真图片
- 新映射: 5红包 6/7通话 9文件 10/18/20链接 11转账 13名片 14位置
  15/10000系统 21引用回复→text 22/19 证据不足留 unknown
- 回填 smoke.db: 620279 行重扫, 94268 行修正, unknown 87198→1885, 幂等
- GET /api/messages 加 msg_type(Literal 422)+q(LIKE 转义)参数
- 新增 test_scrape_msg_type.py(11) + test_api_messages 追加(8), 全量 466 绿

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-25 15:17:21 +08:00
parent f6ddd0263a
commit 64577a1b02
5 changed files with 537 additions and 17 deletions
+43 -1
View File
@@ -8,6 +8,10 @@ Read-only query over the mirrored ``message`` table. Filters:
may be supplied (AND), but typically the frontend sends exactly one.
* ``start`` / ``end`` — ms-epoch window compared against ``timestamp``
(inclusive on both ends).
* ``msg_type`` — normalized type label (``text`` / ``image`` / ...,
see ``_MSG_TYPE_FILTER``). Unknown values are rejected with 422.
* ``q`` — substring match on ``text`` (``%``/``_`` escaped
as literals).
Ordered by ``timestamp asc`` so the timeline renders oldest-first. The index
``ix_message_channel_owner_talker_time`` covers the hot path
@@ -16,7 +20,7 @@ Ordered by ``timestamp asc`` so the timeline renders oldest-first. The index
from __future__ import annotations
from typing import Annotated
from typing import Annotated, Literal
import sqlalchemy as sa
from fastapi import APIRouter, Depends, Query
@@ -32,6 +36,36 @@ SessionDep = Annotated[AsyncSession, Depends(get_session)]
PageParam = Annotated[int, Query(ge=1)]
PageSizeParam = Annotated[int, Query(ge=1, le=100)]
# msg_type 参数值 → 覆盖的 Message.type 标签集合(对齐云客 10 类下拉;
# mini_program 暂无样本行也保留, 查空是数据现实)
_MSG_TYPE_FILTER: dict[str, tuple[str, ...]] = {
"text": ("text",),
"image": ("image", "image_legacy"),
"voice": ("voice",),
"video": ("video",),
"link": ("link",),
"file": ("file",),
"red_packet": ("red_packet",),
"transfer": ("transfer",),
"voice_call": ("voice_call",),
"video_call": ("video_call",),
"mini_program": ("mini_program",),
}
MsgTypeParam = Literal[
"text",
"image",
"voice",
"video",
"link",
"file",
"red_packet",
"transfer",
"voice_call",
"video_call",
"mini_program",
]
@router.get("/messages")
async def list_messages(
@@ -44,6 +78,8 @@ async def list_messages(
room_id: Annotated[str | None, Query()] = None,
start: Annotated[int | None, Query(ge=0)] = None,
end: Annotated[int | None, Query(ge=0)] = None,
msg_type: Annotated[MsgTypeParam | None, Query()] = None,
q: Annotated[str | None, Query()] = None,
) -> Page[MessageOut]:
"""List messages for a timeline (ordered by ``timestamp asc``)."""
conditions = []
@@ -59,6 +95,12 @@ async def list_messages(
conditions.append(Message.timestamp >= start)
if end is not None:
conditions.append(Message.timestamp <= end)
if msg_type is not None:
conditions.append(Message.type.in_(_MSG_TYPE_FILTER[msg_type]))
if q:
# LIKE 通配符转义: %/_ 按字面匹配
escaped = q.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
conditions.append(Message.text.like(f"%{escaped}%", escape="\\"))
total_stmt = sa.select(sa.func.count()).select_from(Message)
if conditions: