fix: split message type into normalized label + raw code
The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -190,6 +190,34 @@ async def scrape_wechat_friends(
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
def _coerce_int_or_none(value: Any) -> int | None:
|
||||
"""Coerce a possibly-string numeric value to ``int``; ``None`` preserved.
|
||||
|
||||
Mirrors :func:`app.storage.repos._coerce_int` so the scraper can emit a
|
||||
ready-to-store ``type_raw`` int and check raw type codes (e.g. the ``-2``
|
||||
divider drop) without depending on the storage layer (the scraper has no
|
||||
DB dependency by design). A non-numeric value (``"text"`` / ``None`` /
|
||||
missing) returns ``None``.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
return int(value)
|
||||
if isinstance(value, str):
|
||||
s = value.strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return int(s)
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
async def _fetch_msg_page(
|
||||
client: YunkeClient,
|
||||
*,
|
||||
@@ -245,9 +273,13 @@ def _normalize_message(record: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Map one upstream ``msg/chats`` record to the ``Message`` column shape.
|
||||
|
||||
Adds ``channel="wechat"`` and preserves the original record under
|
||||
``raw_json``. The ``type`` field is normalized via
|
||||
:func:`normalize_wechat_type` to a stable enum string; the raw code is
|
||||
still recoverable from ``raw_json`` for the persistence layer.
|
||||
``raw_json``. Per design §4.3 the message type is carried as TWO fields:
|
||||
|
||||
* ``type`` — the normalized enum label (``"text"`` / ``"unknown"`` / ...)
|
||||
via :func:`normalize_wechat_type`.
|
||||
* ``type_raw`` — the original upstream type code (``messageType`` falling
|
||||
back to ``type``), coerced to ``int | None`` so the raw code survives
|
||||
end-to-end without round-tripping through ``raw_json``.
|
||||
|
||||
``None``-safe: missing fields become ``None`` rather than raising, so a
|
||||
partial payload does not break the whole scrape. Value coercion
|
||||
@@ -281,6 +313,7 @@ def _normalize_message(record: dict[str, Any]) -> dict[str, Any]:
|
||||
"room_id": record.get("roomid"),
|
||||
"is_send": is_send,
|
||||
"type": normalize_wechat_type(raw_type),
|
||||
"type_raw": _coerce_int_or_none(raw_type),
|
||||
"text": record.get("text"),
|
||||
"file_url": record.get("file"),
|
||||
"file_th_url": record.get("fileTh"),
|
||||
@@ -305,15 +338,19 @@ def _normalize_message(record: dict[str, Any]) -> dict[str, Any]:
|
||||
|
||||
|
||||
def _is_divider(record: dict[str, Any]) -> bool:
|
||||
"""True iff this record is a date-divider (``type``/``messageType`` == -2).
|
||||
"""True iff this record is a date-divider (raw type code ``-2``).
|
||||
|
||||
These are not real messages — the upstream inserts them as visual date
|
||||
bars in the chat UI — and must never be yielded or persisted. Per the
|
||||
Task 1.6 spec we drop a record when EITHER ``type`` or ``messageType`` is
|
||||
Task 1.6 spec we drop a record when EITHER ``messageType`` or ``type`` is
|
||||
``-2``; all other unknown codes pass through.
|
||||
|
||||
The check is on the RAW code (not the normalized label) so the drop
|
||||
decision is robust against any future change to
|
||||
:func:`normalize_wechat_type`'s label vocabulary.
|
||||
"""
|
||||
return normalize_wechat_type(record.get("messageType")) == "divider" or (
|
||||
normalize_wechat_type(record.get("type")) == "divider"
|
||||
return _coerce_int_or_none(record.get("messageType")) == -2 or (
|
||||
_coerce_int_or_none(record.get("type")) == -2
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user