feat(scraper): 微信群消息抓取(368778条/750群)+浏览器守护循环+SQLite忙等

- 新 group-messages 实体:真机探明群消息=同一 msg/chats 端点 friendId 传 roomId;
  talker=发送者wxid、msgId=null→owner:room:createTime:talker 合成稳定主键(重扫幂等)
- 浏览器守护循环:超大群撑崩渲染进程时重启浏览器跳过毒群继续(实战2次自愈);
  200页上限;恢复路径补 await
- base.py SQLite connect_args timeout=60:并发写忙等而非撞锁报错
  (双抓取器并发写实测 database is locked 崩任务)
- 5个新单测,全量410绿

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-20 09:09:32 +08:00
parent 123c23242c
commit f92aac1a3d
3 changed files with 228 additions and 2 deletions
+5
View File
@@ -23,10 +23,15 @@ class Base(DeclarativeBase):
settings = get_settings()
# SQLite: 写锁竞争时忙等 60s 而不是立刻报 "database is locked"
# (多进程并发写的场景:页面抓取器 × 2 + 后端 API 同时跑,2026-08-19 实测撞锁崩任务)。
_connect_args = {"timeout": 60} if settings.db_url.startswith("sqlite") else {}
async_engine = create_async_engine(
settings.db_url,
pool_pre_ping=True,
pool_recycle=3600, # recycle connections before MySQL wait_timeout (default 8h) closes them
connect_args=_connect_args,
future=True,
)