feat(scheduler): 定时同步全面页面化(子进程跑页面脚本,API直调退出调度防风控)

- 每日 full: bash scripts/full_sync_all.sh;每4h messages: scrape_via_page --entity messages
- 子进程隔离+asyncio.Lock互斥+非零退出只记JSON日志;自动补DB_URL环境
- ENTITY_FUNCS/run_once/CLI保留为手动调试,不再挂定时触发器
- README同步更新(Docker镜像需Playwright+Chromium);413测试绿

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-20 09:33:53 +08:00
parent 02dd77a62f
commit 5923cd05b1
3 changed files with 170 additions and 34 deletions
+7 -1
View File
@@ -99,7 +99,7 @@ That brings up:
| `mysql` | `mysql:8.0` | Persistent data store | 3306 |
| `redis` | `redis:7-alpine` | Cache / queue (future use; not yet wired into the app) | — |
| `backend` | `yunke-server/` | FastAPI viewing API (uvicorn) | 8000 |
| `scheduler` | `yunke-server/` | APScheduler loop — daily full sync + every-4h messages | — |
| `scheduler` | `yunke-server/` | APScheduler loop — page-based sync (daily full + every-4h messages; needs Playwright/Chromium in the image) | — |
| `frontend`| `yunkes/` | SPA served by nginx, reverse-proxies `/api` to backend:8000 | 5173 → 80 |
Open <http://localhost:5173>. The scheduler emits one JSON-line log per
@@ -137,6 +137,12 @@ python -m app.sync.cli scheduler # uses FULL_SYNC_HOUR + MESSAG
python -m app.sync.cli scheduler --full-sync-hour 3 --message-interval-hours 6
```
> **2026-08-20 起:定时同步全部页面方式(Playwright 浏览器抓取),不走 API 直调**
> —— 每日全量跑 `scripts/full_sync_all.sh`,消息增量跑
> `scripts/scrape_via_page.py --entity messages`。API 直调
> (`sync --entity ...` / `run_once`) 保留为手动调试入口,不再挂任何定时触发器。
> 容器部署需保证镜像内有 Playwright + Chromium。
Entity names accepted by `sync --entity` and `run_once`:
`departments`, `accounts`, `friends`, `groups`, `voice_calls`,
`phone_calls`, `sms`, `messages`, `all_wechat`, `all` (default).