feat(scraper): 微信群消息抓取(368778条/750群)+浏览器守护循环+SQLite忙等
- 新 group-messages 实体:真机探明群消息=同一 msg/chats 端点 friendId 传 roomId; talker=发送者wxid、msgId=null→owner:room:createTime:talker 合成稳定主键(重扫幂等) - 浏览器守护循环:超大群撑崩渲染进程时重启浏览器跳过毒群继续(实战2次自愈); 200页上限;恢复路径补 await - base.py SQLite connect_args timeout=60:并发写忙等而非撞锁报错 (双抓取器并发写实测 database is locked 崩任务) - 5个新单测,全量410绿 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -23,10 +23,15 @@ class Base(DeclarativeBase):
|
||||
|
||||
settings = get_settings()
|
||||
|
||||
# SQLite: 写锁竞争时忙等 60s 而不是立刻报 "database is locked"
|
||||
# (多进程并发写的场景:页面抓取器 × 2 + 后端 API 同时跑,2026-08-19 实测撞锁崩任务)。
|
||||
_connect_args = {"timeout": 60} if settings.db_url.startswith("sqlite") else {}
|
||||
|
||||
async_engine = create_async_engine(
|
||||
settings.db_url,
|
||||
pool_pre_ping=True,
|
||||
pool_recycle=3600, # recycle connections before MySQL wait_timeout (default 8h) closes them
|
||||
connect_args=_connect_args,
|
||||
future=True,
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user