feat(scheduler): 定时同步全面页面化(子进程跑页面脚本,API直调退出调度防风控)
- 每日 full: bash scripts/full_sync_all.sh;每4h messages: scrape_via_page --entity messages - 子进程隔离+asyncio.Lock互斥+非零退出只记JSON日志;自动补DB_URL环境 - ENTITY_FUNCS/run_once/CLI保留为手动调试,不再挂定时触发器 - README同步更新(Docker镜像需Playwright+Chromium);413测试绿 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -99,7 +99,7 @@ That brings up:
|
||||
| `mysql` | `mysql:8.0` | Persistent data store | 3306 |
|
||||
| `redis` | `redis:7-alpine` | Cache / queue (future use; not yet wired into the app) | — |
|
||||
| `backend` | `yunke-server/` | FastAPI viewing API (uvicorn) | 8000 |
|
||||
| `scheduler` | `yunke-server/` | APScheduler loop — daily full sync + every-4h messages | — |
|
||||
| `scheduler` | `yunke-server/` | APScheduler loop — page-based sync (daily full + every-4h messages; needs Playwright/Chromium in the image) | — |
|
||||
| `frontend`| `yunkes/` | SPA served by nginx, reverse-proxies `/api` to backend:8000 | 5173 → 80 |
|
||||
|
||||
Open <http://localhost:5173>. The scheduler emits one JSON-line log per
|
||||
@@ -137,6 +137,12 @@ python -m app.sync.cli scheduler # uses FULL_SYNC_HOUR + MESSAG
|
||||
python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6
|
||||
```
|
||||
|
||||
> **2026-08-20 起:定时同步全部页面方式(Playwright 浏览器抓取),不走 API 直调**
|
||||
> —— 每日全量跑 `scripts/full_sync_all.sh`,消息增量跑
|
||||
> `scripts/scrape_via_page.py --entity messages`。API 直调
|
||||
> (`sync --entity ...` / `run_once`) 保留为手动调试入口,不再挂任何定时触发器。
|
||||
> 容器部署需保证镜像内有 Playwright + Chromium。
|
||||
|
||||
Entity names accepted by `sync --entity` and `run_once`:
|
||||
`departments`, `accounts`, `friends`, `groups`, `voice_calls`,
|
||||
`phone_calls`, `sms`, `messages`, `all_wechat`, `all` (default).
|
||||
|
||||
Reference in New Issue
Block a user