feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media)
- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+ _iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/ 信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account, form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments - media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无 回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit) 接入scrape_media第4源 - full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数 - scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页 由每日full承担);测试断言7条命令序列 - tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核 (两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化 (一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -578,7 +578,8 @@ class SyncScheduler:
|
||||
|
||||
* ``full`` → ``bash scripts/full_sync_all.sh``(页面方式全量序列)
|
||||
* ``messages`` → ``python scripts/scrape_via_page.py --entity messages
|
||||
--max-conversations N``(增量:未扫对话优先,其余按最近沟通)
|
||||
--max-conversations N``(增量:未扫对话优先,其余按最近沟通),末尾
|
||||
追加 ``--entity moments --max-pages 3``(朋友圈浅扫,每员工最多 3 页)
|
||||
|
||||
子进程隔离——浏览器崩溃/断连不影响调度进程;非零退出只记 JSON 日志
|
||||
不抛异常(保住下一个周期)。asyncio.Lock 保证 full/messages 互斥。
|
||||
@@ -594,7 +595,9 @@ class SyncScheduler:
|
||||
elif kind == "messages":
|
||||
# 先刷客户列表(最近沟通时间是云客列表的实时排序键,friends 快照
|
||||
# 滞后会让本地排序与云客第一页完全不同,2026-08-20),再抓全部四类
|
||||
# 消息增量(2026-08-21 全接入:水位增量后单对话 1 页,可承受)
|
||||
# 消息增量(2026-08-21 全接入:水位增量后单对话 1 页,可承受);
|
||||
# 末尾追加员工朋友圈浅扫(2026-08-25,max-pages 3 即每员工最多
|
||||
# 3 页,新帖通常落第一页;全量翻页由每日 full 走 full_sync_all.sh)
|
||||
commands = [
|
||||
[sys.executable, "scripts/scrape_via_page.py", "--entity", "friends"],
|
||||
[sys.executable, "scripts/scrape_via_page.py", "--entity", "wa-friends"],
|
||||
@@ -608,6 +611,15 @@ class SyncScheduler:
|
||||
"99999",
|
||||
]
|
||||
for e in ("messages", "wa-messages", "group-messages", "wa-group-messages")
|
||||
] + [
|
||||
[
|
||||
sys.executable,
|
||||
"scripts/scrape_via_page.py",
|
||||
"--entity",
|
||||
"moments",
|
||||
"--max-pages",
|
||||
"3",
|
||||
]
|
||||
]
|
||||
elif kind == "full-history":
|
||||
# 每周全量对齐(周日 05:00):强制全量翻历史,对齐增量模式抓不到的
|
||||
|
||||
Reference in New Issue
Block a user