fix: split message type into normalized label + raw code
The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -23,9 +23,9 @@ dict (see :func:`_coerce_bool`, :func:`_coerce_int`):
|
||||
``1`` / ``"1"`` / ``True`` -> ``True``; ``0`` / ``"0"`` / ``False`` ->
|
||||
``False``; ``None`` preserved.
|
||||
* ``gender``: ``"0"`` / ``"1"`` / ``"2"`` (str) -> ``int`` when not ``None``.
|
||||
* ``type`` (Message): coerced to ``int``; a non-numeric string (e.g. a
|
||||
normalized type label like ``"text"``) collapses to ``None`` — the raw code
|
||||
is still recoverable from ``raw_json``.
|
||||
* ``type_raw`` (Message): the raw upstream type code, coerced to
|
||||
``int | None``. ``Message.type`` is a normalized label string and is
|
||||
stored as-is (no int coercion).
|
||||
* Timestamps (``last_chat_time`` / ``timestamp`` / ``last_update`` / ...):
|
||||
passed through as-is (ms-epoch ``int`` or ``None``).
|
||||
* ``raw_json``: the original record dict, preserved verbatim. When the input
|
||||
@@ -72,7 +72,7 @@ _BOOL_FIELDS: frozenset[str] = frozenset(
|
||||
_INT_FIELDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"gender",
|
||||
"type",
|
||||
"type_raw",
|
||||
"pic_type",
|
||||
"friend_state",
|
||||
"delete_flag",
|
||||
@@ -118,8 +118,7 @@ def _coerce_int(value: Any) -> int | None:
|
||||
|
||||
``"2"`` -> ``2``; ``2`` -> ``2``; ``0``/``"0"`` -> ``0``; ``None`` ->
|
||||
``None``. A non-numeric string (e.g. ``"text"``) returns ``None`` — the
|
||||
caller (Message.type) treats that as "no value", and the raw code is still
|
||||
recoverable from ``raw_json``.
|
||||
caller (e.g. Message.type_raw) treats that as "no value".
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
@@ -287,9 +286,10 @@ async def upsert_customers(session: AsyncSession, recs: list[dict[str, Any]]) ->
|
||||
async def upsert_messages(session: AsyncSession, recs: list[dict[str, Any]]) -> int:
|
||||
"""Batch idempotent upsert for :class:`Message` (PK = ``msg_id``).
|
||||
|
||||
Applies the documented coercions (``is_send``/``is_del`` -> bool, ``type``
|
||||
-> int). Returns the number of records processed (input length); the DB
|
||||
collapses duplicate ``msg_id`` values to one row.
|
||||
Applies the documented coercions (``is_send``/``is_del`` -> bool,
|
||||
``type_raw`` -> int). ``Message.type`` is the normalized label string and
|
||||
is stored as-is. Returns the number of records processed (input length);
|
||||
the DB collapses duplicate ``msg_id`` values to one row.
|
||||
|
||||
.. note::
|
||||
For M1 the ``Message`` model maps to a single physical ``message``
|
||||
|
||||
Reference in New Issue
Block a user