The scraper was writing a normalized type label ("text"/"unknown"/...) into
Message.type, but the column was Integer, so the storage layer coerced it to
NULL and the raw code was only recoverable from raw_json. Align with design
§4.3, which specifies two fields: `type` (normalized enum) and `type_raw`
(original code).
- Model (Message): type -> String(32) normalized label; add type_raw Integer
for the raw upstream type code.
- Scraper: _normalize_message sets both `type` (label) and `type_raw`
(coerced int from messageType/type); _is_divider now checks raw -2.
- Repos: type_raw joins _INT_FIELDS; Message.type stored as-is (string).
- Migration 96975050fa79: add type_raw + alter type Integer->String(32),
wrapped in batch_alter_table so it runs on SQLite and MySQL.
- env.py: enable render_as_batch on SQLite (dev/test) for portable batch
migrations; MySQL still uses direct ALTER.
- Tests updated end-to-end (scraper, repos, models).
Co-Authored-By: Claude <noreply@anthropic.com>
Adds the storage layer that bridges scrapers (which yield normalized dicts)
to the DB and the local media store.
app/storage/repos.py
* upsert_customer / upsert_customers / upsert_messages / upsert_wechat_account
/ upsert_staff_from_account, all taking an AsyncSession + scraper-shaped dicts.
* Dialect-aware Core upsert: SQLite on_conflict_do_update for tests,
MySQL on_duplicate_key_update for prod; dialect detected via session.bind.name.
_build_upsert_stmt(model, values, dialect) is the reusable helper.
* Type coercion: _coerce_bool / _coerce_int (module-level, unit-tested) map
is_send/is_del/is_block/is_on_job int-as-bool to bool and gender/type
numeric strings to int; None preserved; raw_json kept verbatim.
* For M1 the Message model maps to a single physical `message` table;
per-month sharding (message_table_name helper) is a LATER task (YAGNI).
app/storage/object_store.py
* Local-filesystem ObjectStore with put_bytes / put_stream / put_by_hash /
exists / get_path, all dispatched via asyncio.to_thread so disk I/O is
genuinely async.
* put_by_hash computes sha256 and writes only if absent (content-addressed
dedup); layout is base_dir/<first 2 hash chars>/<hash><ext>.
* Default base dir is Settings.media_dir; the interface is the seam a
future MinIO/S3 backend will swap in behind.
tests: 24 new tests (in-memory SQLite for repos; tmp_path for object_store)
covering idempotency (same PK twice -> 1 row, updated), type coercion, batch
dedup (3 messages / 1 dup msg_id -> 2 rows), and hash dedup (corrupt-file
probe proves second put_by_hash does not rewrite). ruff clean; full suite
67/67 green.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念