fix: split message type into normalized label + raw code
The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -50,6 +50,11 @@ def _sync_url(url: str) -> str:
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode (emit SQL without a live connection)."""
|
||||
url = _sync_url(get_settings().db_url)
|
||||
# SQLite cannot ALTER most column properties directly; Alembic's batch mode
|
||||
# (table-recreate) handles it. Enable it for SQLite so the SAME migration
|
||||
# scripts run on both SQLite (dev/test) and MySQL (prod). MySQL needs no
|
||||
# batch mode and gets direct ALTERs.
|
||||
render_as_batch = url.startswith("sqlite")
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
@@ -57,6 +62,7 @@ def run_migrations_offline() -> None:
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
compare_type=True,
|
||||
compare_server_default=True,
|
||||
render_as_batch=render_as_batch,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
@@ -74,11 +80,14 @@ def run_migrations_online() -> None:
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
# See offline mode for the rationale on batch mode per dialect.
|
||||
render_as_batch = connection.dialect.name == "sqlite"
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
compare_type=True,
|
||||
compare_server_default=True,
|
||||
render_as_batch=render_as_batch,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
"""message type: split normalized label and raw code
|
||||
|
||||
Revision ID: 96975050fa79
|
||||
Revises: d7a9bc760c6b
|
||||
Create Date: 2026-08-01 12:01:31.810063
|
||||
|
||||
Splits the ``message.type`` column to match design §4.3:
|
||||
|
||||
* ``type`` becomes ``String(32)`` holding the NORMALIZED enum label produced
|
||||
by :func:`app.scraper.parsers.normalize_wechat_type` (``"text"`` /
|
||||
``"unknown"`` / ...). Previously this column was ``Integer`` but the
|
||||
scraper was already writing a normalized string into it, so the storage
|
||||
layer silently coerced that string to ``NULL`` and the only way to recover
|
||||
the raw code was via ``raw_json``.
|
||||
* ``type_raw`` (new, ``Integer``) carries the ORIGINAL upstream type code
|
||||
end-to-end (scraper -> repos -> DB) so it no longer has to be round-tripped
|
||||
through ``raw_json``.
|
||||
|
||||
The operations are wrapped in ``batch_alter_table`` so the SAME migration
|
||||
runs on SQLite (dev/test, where ALTER COLUMN is unsupported and Alembic
|
||||
recreates the table) and MySQL (prod, where it emits a direct ``MODIFY
|
||||
COLUMN`` / ``ADD COLUMN``).
|
||||
"""
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '96975050fa79'
|
||||
down_revision: str | Sequence[str] | None = 'd7a9bc760c6b'
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
with op.batch_alter_table('message', schema=None) as batch_op:
|
||||
batch_op.add_column(sa.Column('type_raw', sa.Integer(), nullable=True))
|
||||
batch_op.alter_column('type',
|
||||
existing_type=sa.INTEGER(),
|
||||
type_=sa.String(length=32),
|
||||
existing_nullable=True)
|
||||
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Downgrade schema."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
with op.batch_alter_table('message', schema=None) as batch_op:
|
||||
batch_op.alter_column('type',
|
||||
existing_type=sa.String(length=32),
|
||||
type_=sa.INTEGER(),
|
||||
existing_nullable=True)
|
||||
batch_op.drop_column('type_raw')
|
||||
|
||||
# ### end Alembic commands ###
|
||||
Reference in New Issue
Block a user