The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
Implements the 14 core ORM models for the yunke data-mirror system using
SQLAlchemy 2.0 Mapped/mapped_column style, driven by the field lists in
docs/superpowers/specs/2026-08-01-yunke-data-dictionary.md:
Department, Staff, WechatAccount, WhatsappAccount, Customer, ChatGroup,
GroupMember, Message, Media, PhoneCall, Sms, SyncState, YkAccount, SyncRun.
Conventions:
- Natural source PKs where they exist (friend_id, wx_id, msg_id, room_id,
media_hash, call_id, sms_id, dept_id, user_id); surrogate Integer PKs
otherwise (SyncState, YkAccount, SyncRun). GroupMember uses a composite
(channel, room_id, member_id) PK.
- Common audit columns version/raw_json/fetched_at/updated_at on every
source-mirrored table.
- channel only where meaningful: Customer/Message/Media/ChatGroup/
GroupMember/Sms use messaging channels; PhoneCall.channel is call-type;
SyncState.channel scopes the cursor. Omitted on Staff/Department/
WechatAccount/WhatsappAccount/YkAccount/SyncRun.
- Message composite index on (channel, owner_account_id, talker_id,
timestamp) for the hot per-talker time-order query path.
- message_table_name(channel, timestamp) helper documents the future
message_<channel>_<YYYYMM> sharding convention; the Message model still
maps to a single physical table for now (sharding is a later task).
All types are portable generic SQLAlchemy types so tests run on in-memory
SQLite while prod targets MySQL. Tests verify Customer round-trip,
raw_json JSON round-trip, Message natural PK, GroupMember composite PK,
the Message composite index, and full table creation on SQLite.
Alembic wired via migrations/env.py to Base.metadata + get_settings().db_url
with an async->sync driver swap (mysql+asyncmy -> mysql+pymysql,
sqlite+aiosqlite -> sqlite). Initial migration autogenerated against an
empty SQLite DB; server defaults normalized to sa.func.now() so the file
is MySQL-portable. upgrade head is intentionally not run (no MySQL yet).
Co-Authored-By: Claude <noreply@anthropic.com>
浅念