feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media)

- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+
  _iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/
  信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account,
  form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments
- media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无
  回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit)
  接入scrape_media第4源
- full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数
- scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页
  由每日full承担);测试断言7条命令序列
- tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核
  (两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化
  (一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-25 14:06:45 +08:00
parent b7669199f8
commit 90c28f4c7b
6 changed files with 549 additions and 15 deletions
+3
View File
@@ -41,6 +41,8 @@ echo "$(date) --- 9/10 WA messages ---"
$PY scripts/scrape_via_page.py --entity wa-messages --max-conversations 99999
echo "$(date) --- 9b/10 WA group messages ---"
$PY scripts/scrape_via_page.py --entity wa-group-messages --max-conversations 99999
echo "$(date) --- 9c/10 Moments (员工朋友圈) ---"
$PY scripts/scrape_via_page.py --entity moments
echo "$(date) --- 10/10 Media download (audio+images) ---"
$PY scripts/scrape_via_page.py --entity media
@@ -65,6 +67,7 @@ for label, sql in [
('whatsapp messages', \"SELECT COUNT(*) FROM message WHERE channel='whatsapp'\"),
('voice calls', \"SELECT COUNT(*) FROM phone_call WHERE channel IN ('voice_call','wa_voice_call')\"),
('voice clips', 'SELECT COUNT(*) FROM voice_clip'),
('moments posts', 'SELECT COUNT(*) FROM moments_post'),
('media files', 'SELECT COUNT(*) FROM media'),
]:
print(f' {label:22s} {c.execute(sql).fetchone()[0]:>6}')