 浅念
|
90c28f4c7b
|
feat(scraper): moments 实体(浏览器fetch getFriendsterInfo,秒→ms,hashNext翻页,接入full_sync+4h调度+图片media)
- scrape_via_page:_normalize_moments(createTime秒×1000,缺id丢弃)+
_iter_moments_pages 翻页内核(fetch回调注入,sleep可注入,hashNext假/
信封异常/max_pages 上限即止)+ scrape_moments(逐员工遍历wechat_account,
form小驼峰,首页code≠10000告警cookie失效,换页重试惯例)+CLI注册moments
- media:download_moments_images(image_urls JSON数组展开去重,MomentsPost无
回链列→用Media.original_url做已下载标记,分片IN防SQLite变量上限,批间commit)
接入scrape_media第4源
- full_sync_all.sh:9c步moments(media前,同轮图片即可本地化)+汇总加moments计数
- scheduler:4h messages job 末尾追加 moments --max-pages 3 浅扫(全量翻页
由每日full承担);测试断言7条命令序列
- tests/test_scrape_moments.py:归一化(秒→ms/空安全/坏createTime)+翻页内核
(两页累积停止/hashNext假单页/max_pages防死循环/丢id行/坏信封)+图片本地化
(一帖两图/幂等跳过/非http跳过+500容错);全套440测试通过
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-08-25 14:07:39 +08:00 |
|
|
浅念
|
02dd77a62f
|
feat(scraper): WA群抓取(type=1真机验证,223群)+群成员派生(12840成员/750群)
- wa-groups 实体:businessFriendList type=1 真机验证返回群,行结构是好友字段名
(friendUid=@g.us 群ID/nickName=群名),专用 _normalize_wa_group_row
- backfill_group_members.py:无独立成员API,从群消息派生(talker=成员,sender_name=昵称)
- full_sync_all.sh:+4步(5b WA群/语音后录音重链接/8b 群消息+成员/消息上限99999)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-08-20 09:21:20 +08:00 |
|