feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media)
- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+ _iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/ 信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account, form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments - media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无 回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit) 接入scrape_media第4源 - full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数 - scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页 由每日full承担);测试断言7条命令序列 - tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核 (两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化 (一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -41,6 +41,8 @@ echo "$(date) --- 9/10 WA messages ---"
|
||||
$PY scripts/scrape_via_page.py --entity wa-messages --max-conversations 99999
|
||||
echo "$(date) --- 9b/10 WA group messages ---"
|
||||
$PY scripts/scrape_via_page.py --entity wa-group-messages --max-conversations 99999
|
||||
echo "$(date) --- 9c/10 Moments (员工朋友圈) ---"
|
||||
$PY scripts/scrape_via_page.py --entity moments
|
||||
echo "$(date) --- 10/10 Media download (audio+images) ---"
|
||||
$PY scripts/scrape_via_page.py --entity media
|
||||
|
||||
@@ -65,6 +67,7 @@ for label, sql in [
|
||||
('whatsapp messages', \"SELECT COUNT(*) FROM message WHERE channel='whatsapp'\"),
|
||||
('voice calls', \"SELECT COUNT(*) FROM phone_call WHERE channel IN ('voice_call','wa_voice_call')\"),
|
||||
('voice clips', 'SELECT COUNT(*) FROM voice_clip'),
|
||||
('moments posts', 'SELECT COUNT(*) FROM moments_post'),
|
||||
('media files', 'SELECT COUNT(*) FROM media'),
|
||||
]:
|
||||
print(f' {label:22s} {c.execute(sql).fetchone()[0]:>6}')
|
||||
|
||||
Reference in New Issue
Block a user