浅念
7b49f94dcb
M2 Task 2.6 (media downloader) + Task 2.8 (media serve API).
Downloader (app/scraper/media.py):
- download_url(client, url): fetches bytes via the YunkeClient's underlying
httpx client (carries session cookies; respx-mockable; follows redirects).
- download_message_media(client, msg, store, session): URL-driven fetcher that
picks the best http(s) URL (file_url -> file_hd_url -> file_th_url), dedups
by md5 (skips the network when a Media row with local file already exists),
falls back to sha256(bytes) when no md5, stores via ObjectStore.put_by_hash,
and upserts a Media row. Skips type-10 system messages (non-media per
phase0-findings §2) and type-3 opaque-id / text (no http url). Resilient:
download errors are logged and return None (no raise).
- media_type_from_message / guess_mime_type helpers.
Repo (app/storage/repos.py): upsert_media — idempotent Core upsert into Media by
media_hash. Volatile download artifacts (local_path/size_bytes/mime_type) use
COALESCE(incoming, existing) on conflict so a re-scrape that did not re-download
does not clobber previously stored values. Exported from app.storage.
Serve API (app/api/routers/media.py): GET /api/media/{media_hash} streams the
local file via FileResponse (handles HTTP Range / 206). 404 when the row is
absent, has no local file, or the file is missing on disk. Content-type from the
row's mime_type, falling back to ext-derived guess.
Tests (+19, suite 130 -> 149, all green; ruff clean):
- test_media_downloader.py: image/video download + upsert; md5 dedup skips
second network hit; sha256 fallback when no md5; type-3/text/system -> None;
download failure -> None (no raise); upsert_media idempotency +
null-local_path preservation + 3x-same-key-one-row.
- test_api_media.py: 200 + content-type + bytes; 404 unknown hash; 404 missing
file on disk; Range -> 206 partial; ext-derived content-type when mime NULL.
Co-Authored-By: Claude <noreply@anthropic.com>
37 lines
1.1 KiB
Python
37 lines
1.1 KiB
Python
"""Storage layer for the yunke data-mirror system.
|
|
|
|
Bridges the scrapers (``app.scraper.*``, which yield normalized dicts) to the
|
|
database (``app.db.models.*``) and the local media store.
|
|
|
|
Public surface (M1 / Task 1.7):
|
|
|
|
* :mod:`app.storage.repos` — idempotent Core upsert helpers that take an
|
|
``AsyncSession`` plus scraper-shaped dicts and write to the target table.
|
|
The upsert is dialect-aware (SQLite for tests, MySQL for prod) so the same
|
|
call path works in both environments.
|
|
* :mod:`app.storage.object_store` — a local-filesystem object store with
|
|
sha256-content-addressed dedup. The interface (``put_bytes`` / ``put_stream``
|
|
/ ``put_by_hash`` / ``exists`` / ``get_path``) is the seam a future MinIO/S3
|
|
backend will swap in behind.
|
|
"""
|
|
|
|
from app.storage.object_store import ObjectStore
|
|
from app.storage.repos import (
|
|
upsert_customer,
|
|
upsert_customers,
|
|
upsert_media,
|
|
upsert_messages,
|
|
upsert_staff_from_account,
|
|
upsert_wechat_account,
|
|
)
|
|
|
|
__all__ = [
|
|
"ObjectStore",
|
|
"upsert_customer",
|
|
"upsert_customers",
|
|
"upsert_media",
|
|
"upsert_messages",
|
|
"upsert_staff_from_account",
|
|
"upsert_wechat_account",
|
|
]
|