fix: split message type into normalized label + raw code

The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).

- Model (Message): type -> String(32) normalized label; add type_raw Integer
  for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
  (coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
  wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
  migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 12:04:08 +08:00
parent 854efd3119
commit 5a2ff58af3
8 changed files with 176 additions and 29 deletions
@@ -106,7 +106,8 @@ async def test_scrape_wechat_messages_drops_divider_and_paginates_backward():
assert r1["ori_talker"] == "wxid_friend_0001"
assert r1["room_id"] == "" # raw value carried through
assert r1["is_send"] == 0 # pulled from `mine`
assert r1["type"] == "text"
assert r1["type"] == "text" # normalized label (design §4.3)
assert r1["type_raw"] == 1 # original raw code preserved
assert r1["text"] == "hi alice"
assert r1["timestamp"] == E1 + 600000
assert r1["sender_name"] == "Alice Wang"
@@ -119,6 +120,7 @@ async def test_scrape_wechat_messages_drops_divider_and_paginates_backward():
assert r2["ori_talker"] == "room12345@chatroom"
assert r2["is_send"] == 1
assert r2["type"] == "unknown" # unmapped code passes through
assert r2["type_raw"] == 3 # raw image-type code preserved
assert r2["file_url"] == "https://cdn.example.com/img2.jpg"
assert r2["file_th_url"] == "https://cdn.example.com/img2_th.jpg"
assert r2["file_hd_url"] == "https://cdn.example.com/img2_hd.jpg"
@@ -133,6 +135,7 @@ async def test_scrape_wechat_messages_drops_divider_and_paginates_backward():
# m3 — voice / transcription.
assert r3["msg_id"] == "999999"
assert r3["type"] == "unknown"
assert r3["type_raw"] == 34 # raw voice-type code preserved
assert r3["duration"] == 5
assert r3["transcription_text"] == "hello from yesterday"
assert r3["pay_msg_id"] == "pay-abc-123"