浅念
854efd3119
Adds the storage layer that bridges scrapers (which yield normalized dicts)
to the DB and the local media store.
app/storage/repos.py
* upsert_customer / upsert_customers / upsert_messages / upsert_wechat_account
/ upsert_staff_from_account, all taking an AsyncSession + scraper-shaped dicts.
* Dialect-aware Core upsert: SQLite on_conflict_do_update for tests,
MySQL on_duplicate_key_update for prod; dialect detected via session.bind.name.
_build_upsert_stmt(model, values, dialect) is the reusable helper.
* Type coercion: _coerce_bool / _coerce_int (module-level, unit-tested) map
is_send/is_del/is_block/is_on_job int-as-bool to bool and gender/type
numeric strings to int; None preserved; raw_json kept verbatim.
* For M1 the Message model maps to a single physical `message` table;
per-month sharding (message_table_name helper) is a LATER task (YAGNI).
app/storage/object_store.py
* Local-filesystem ObjectStore with put_bytes / put_stream / put_by_hash /
exists / get_path, all dispatched via asyncio.to_thread so disk I/O is
genuinely async.
* put_by_hash computes sha256 and writes only if absent (content-addressed
dedup); layout is base_dir/<first 2 hash chars>/<hash><ext>.
* Default base dir is Settings.media_dir; the interface is the seam a
future MinIO/S3 backend will swap in behind.
tests: 24 new tests (in-memory SQLite for repos; tmp_path for object_store)
covering idempotency (same PK twice -> 1 row, updated), type coercion, batch
dedup (3 messages / 1 dup msg_id -> 2 rows), and hash dedup (corrupt-file
probe proves second put_by_hash does not rewrite). ruff clean; full suite
67/67 green.
Co-Authored-By: Claude <noreply@anthropic.com>
35 lines
1.1 KiB
Python
35 lines
1.1 KiB
Python
"""Storage layer for the yunke data-mirror system.
|
|
|
|
Bridges the scrapers (``app.scraper.*``, which yield normalized dicts) to the
|
|
database (``app.db.models.*``) and the local media store.
|
|
|
|
Public surface (M1 / Task 1.7):
|
|
|
|
* :mod:`app.storage.repos` — idempotent Core upsert helpers that take an
|
|
``AsyncSession`` plus scraper-shaped dicts and write to the target table.
|
|
The upsert is dialect-aware (SQLite for tests, MySQL for prod) so the same
|
|
call path works in both environments.
|
|
* :mod:`app.storage.object_store` — a local-filesystem object store with
|
|
sha256-content-addressed dedup. The interface (``put_bytes`` / ``put_stream``
|
|
/ ``put_by_hash`` / ``exists`` / ``get_path``) is the seam a future MinIO/S3
|
|
backend will swap in behind.
|
|
"""
|
|
|
|
from app.storage.object_store import ObjectStore
|
|
from app.storage.repos import (
|
|
upsert_customer,
|
|
upsert_customers,
|
|
upsert_messages,
|
|
upsert_staff_from_account,
|
|
upsert_wechat_account,
|
|
)
|
|
|
|
__all__ = [
|
|
"ObjectStore",
|
|
"upsert_customer",
|
|
"upsert_customers",
|
|
"upsert_messages",
|
|
"upsert_staff_from_account",
|
|
"upsert_wechat_account",
|
|
]
|