- GET /api/calls/stats:今日(Asia/Shanghai)好友呼入/外呼量+群量+时长聚合(@chatroom判群)
- CallList:5张统计卡片(仅微信语音tab)+datetimerange筛选(服务端start/end)+viewCallCustomer带?start定位
- TDD新增stats聚合测试,全量绿
Co-Authored-By: Claude <noreply@anthropic.com>
- main.py + config.py: CORS origins read from Settings.cors_origins (comma-separated).
Empty falls back to dev origin localhost:5173; never wildcard. Methods locked
to GET/PATCH/POST; allow_credentials=False.
- scraper/media.py: download_url now passes follow_redirects=False to httpx.
OSS URLs return 200 directly; a 3xx is treated as a download failure (caught
by existing except clauses) instead of opening an SSRF vector to internal hosts.
- routers/voice_clips.py: PATCH /voice-clips/{id}/match accepts an optional
If-Match header carrying the client's last-seen updated_at. On mismatch the
row was modified by another request and the server returns 409. Without the
header the behaviour is unchanged (backward compatible). Response now echoes
the post-update updated_at so the client can fold it into the next request.
- routers/organization.py: /api/accounts now UNION ALLs the wechat_account and
whatsapp_account projections at the SQL level and pushes COUNT + pagination
(offset/limit) down to the DB. Stable ordering on (channel, account_id).
Removes the in-memory materialization that broke at scale.
Tests: 395 passed. Added optimistic-lock coverage (If-Match accepted / stale 409 /
absent header backward-compat). Updated existing PATCH tests for the new
updated_at response field.
Co-Authored-By: Claude <noreply@anthropic.com>
address review Minors: drop redundant filename replace, assert 暂无消息,
tighten downloadBlob return type + try/finally, fix plan group time-picker wording.
Co-Authored-By: Claude <noreply@anthropic.com>
M6 — the scheduling/deployment layer. Makes the system runnable as a
scheduled service + deployable via Docker.
Part A — SyncScheduler (app/sync/scheduler.py):
- APScheduler (AsyncIOScheduler) wraps the orchestrator's sync_* functions.
- Daily full sync at FULL_SYNC_HOUR (default 02:00 local) + incremental
message sync every MESSAGE_INTERVAL_HOURS (default 4).
- ENTITY_FUNCS registry dispatches run_once("<entity>") to the right runner
(departments/accounts/friends/groups/voice_calls/phone_calls/sms/messages/
all_wechat/all).
- Conversations for the message run are discovered from the local customer
mirror (no extra upstream call).
- Failure policy: every run is wrapped — a blown entity is logged + swallowed
so the scheduler keeps its next appointment.
- Per-run JSON-line log (level=info/warning/error, totals, failed entities).
Part B — CLI entry point (app/sync/cli.py):
- `python -m app.sync.cli sync [--entity <name>] [--quiet]` — one-shot sync.
- `python -m app.sync.cli scheduler [--full-sync-hour H] [--message-interval-hours N]`
— long-running scheduler (blocks until SIGINT/SIGTERM).
- Uses _push_settings_into_environ + build_client_from_env (session-cookie
mode preferred; pcLogin deferred).
- Base.metadata.create_all on every command (fresh-checkout friendly).
- Exit codes: 0 success, 1 generic failure, 2 bad CLI / auth misconfig.
Part C — Docker stack:
- yunke-server/Dockerfile: python:3.12-slim, installs deps + app, single
image serves both API (uvicorn) + scheduler (CLI).
- yunkes/Dockerfile: multi-stage node:20 build → nginx serves dist/ +
reverse-proxies /api → backend:8000.
- yunkes/nginx.conf: /api proxy with proxy_buffering off (streaming media),
SPA fallback, asset caching.
- docker-compose.yml: mysql (8.0, healthcheck, volume) + redis (7-alpine) +
backend + scheduler + frontend (port 5173 → 80). Env-driven creds + cadence.
Part D — README.md at repo root: dev mode + docker mode quick-start, config
table, architecture overview, current limitations (pcLogin deferred, WA voice/
media deferred, conservative pacing, conversation discovery is local-only).
Verification:
- pytest: 323 passed (was 294; +29 new for scheduler + CLI).
- npm run build: clean.
- python -m app.sync.cli --help: prints usage.
- docker compose config: VALID.
Co-Authored-By: Claude <noreply@anthropic.com>
Live validation: businessFriendList uses 'page' (not pageNum) and REQUIRES
departmentId (root) + type or returns empty. Same class of bug as wxUser/list.
Co-Authored-By: Claude <noreply@anthropic.com>
Replicates the Customer all-fields pattern across the remaining three
entities. Backend now exposes `raw_json` on MessageOut / StaffOut /
AccountOut so the frontend can render every 云客 source field without a
second endpoint. Each entity gains a Chinese-label field utility + an
el-drawer surfacing all fields (structured-first, raw_json fallback,
unknown keys under 其他).
Backend:
- schemas.py: add `raw_json: dict | None` to MessageOut / StaffOut /
AccountOut. MessageOut/StaffOut pick the column up via from_attributes;
AccountOut is built explicitly in the router, which now threads
`raw_json` through from both WechatAccount and WhatsappAccount rows.
- tests: new raw_json serialization tests for messages, staff, accounts
(assert populated dict round-trips verbatim + NULL serializes as null).
Frontend (yunkes/):
- New utils: messageFields.ts, staffFields.ts, accountFields.ts — each
with a 中文 label map, ordered groups, structured-column preference,
and an 其他 fallback for unknown raw_json keys.
- Message: clicking a chat bubble opens an all-fields drawer (referMsg
renders as pretty-printed JSON). Image preview + file download links
stopPropagation so their own interactions still work.
- Staff: row click → drawer; columns relabeled (员工ID, 销售手机).
- Accounts: new AccountList.vue at /accounts with a 账号 nav item,
unified WeChat/WhatsApp table (渠道/账号ID/昵称/微信号/归属员工/好友数/
群数/收发/在职/最后更新), channel filter, pagination, row-click drawer.
- api: add Account type + getAccounts(); add raw_json to Message/Staff.
Co-Authored-By: Claude <noreply@anthropic.com>
Backend:
- CustomerOut now exposes raw_json (verbatim upstream friend record) so the
frontend can render every 云客 field without per-field structured columns.
- The verbatim blob round-trips via from_attributes; no model change needed.
Frontend:
- CustomerList table mirrors the 云客 客户列表 column set (头像/微信昵称/微信号/
备注/真实姓名/手机/性别/地区/描述/销售员工/部门/来源/添加方式/添加时间/最近
聊天/消息数/渠道). A readField helper reads the structured column first and
falls back to raw_json.<camelCase>.
- Row click opens an all-fields drawer (right side) rendering EVERY field of
the record with Chinese labels, grouped (基本信息/联系方式/归属员工与部门/时间
与来源/消息统计/好友状态/其他). Timestamps format to local datetime, flags
render 是/否, image URLs render as el-image thumbnails, empty values show —.
- New src/utils/customerFields.ts holds the camelCase→中文 label map, the
structured-column alias map, group definitions, and value-type formatters;
unknown raw_json keys auto-flow into an 其他 section so new upstream fields
surface without code changes.
- Drawer 查看聊天记录 action navigates to /chat/:channel/:ownerId/:talkerId
(owner_account_id + friend_id + channel), preserving the existing path.
Co-Authored-By: Claude <noreply@anthropic.com>
Backend (scrape_wechat_accounts):
- Send wxUser/list as form-urlencoded (not JSON) with the REQUIRED filters
isOnJob / departmentId / startdate / enddate + empty userId/salesname/
weChatNickName, matching the live request (empty fails with code -90000).
- scrape_wechat_accounts now takes department_id (the company-root dept id)
plus optional start/end defaulting to a recent 7-day UTC window.
- paginate_by_page gains a form=True mode (data= + form content-type).
- Orchestrator sync_wechat_accounts resolves the root dept from the
Department table (or accepts one) and skips with a warning when unknown.
- seed_accounts.py resolves the root via scrape_departments.
Frontend (yunkes):
- Add StaffList.vue (/staff) with is_on_job filter + pagination.
- Add /staff route and a 员工 nav item between 客户 and 群.
Co-Authored-By: Claude <noreply@anthropic.com>
Live validation found two real bugs:
- YunkeClient sent 'source: yunkepc' on every request; msg/chats binds
source as Integer -> 500. Removed (cookie auth suffices, per Phase 0).
- WeChat message talker_id used the masked 'talker' field; the timeline
joins on the full peer id. Now: 1:1 -> oriTalker (full), group -> talker
(member). ori_talker keeps the raw oriTalker (peer/chatroom).
Co-Authored-By: Claude <noreply@anthropic.com>
Backend (Part A):
- MessageOut: add file_th_url, file_hd_url, duration, transcription_text, md5
(room_id + sender_head_url already existed) so the chat timeline can render
image thumbnails, voice duration/transcription, and avatars from a single
response.
- PhoneCallOut: add record_file so the Calls view can offer recording playback.
- SmsOut: add content (full body) so the SMS detail dialog shows the whole text.
- Tests: red->green for each new field (media fields default None + populated,
record_file round-trip, content round-trip). Suite 267 -> 271.
Frontend (Part B):
- api/index.ts: wrappers getGroups/getGroupMembers/getCalls/getSms/getSyncStatus
+ Group/GroupMember/PhoneCall/Sms/SyncState/SyncRun/SyncStatus types in
types.ts (mirror the *Out pydantic schemas).
- views/GroupList.vue (/groups): table + channel filter + pagination; row click
opens the group timeline.
- views/CallList.vue (/calls): table + call-type filter (phone_call/voice_call/
all); "播放录音" action opens an el-dialog with <audio controls>.
- views/SmsList.vue (/sms): table; row click opens a detail dialog with the full
content.
- views/SyncStatus.vue (/sync): two tables (states + recent_runs) with a manual
refresh; run status color-coded.
- ChatTimeline.vue: reused for both 1:1 DM (talkerId) and group (roomId) by
branching the query; group title + per-member sender already handled by
MessageBubble.
- router: add /groups /calls /sms /sync + /group/:channel/:ownerId/:roomId.
- layouts/MainLayout.vue: replace the single 客户列表 button with an el-menu
(客户 / 群 / 通话 / 短信 / 同步); org tree kept on the left.
Co-Authored-By: Claude <noreply@anthropic.com>
Mirrors the WeChat scraper patterns (M3) using the shared paginator +
parser + repo building blocks. Three new async generators yield
snake_case dicts + verbatim raw_json, ready for the channel-generic
upsert_customer / upsert_messages repos.
- app/scraper/whatsapp.py (new): scrape_whatsapp_accounts /
scrape_whatsapp_friends (page-number pagination over the
data.overviews / data.totalCount envelope) and scrape_whatsapp_messages
(backward-cursor pagination, JSON body {uid,friendUid,start,end} —
NOT form-encoded like WeChat msg/chats). Drops type==-2 dividers;
maps mediaName -> file_url as a deferred-download key.
- app/scraper/parsers.py: add normalize_whatsapp_type (0->text,
1->image, -2->divider, else unknown; never raises) and a shared
coerce_int_or_none scraper helper (mirrors repos._coerce_int so the
scraper emits type_raw ints without a DB dependency).
- app/storage/repos.py: add upsert_whatsapp_account (PK uid) which
also upserts the embedded Staff(user_id, name) via the existing
upsert_staff_from_account helper; WhatsappAccount imported.
Tests (respx mocks, no network): 3 cassette fixtures + 3 scraper test
modules (accounts/friends/messages), parsers + repos extended with WA
cases. Divider drop, backward-cursor advance (end == E1-1), JSON body
asserts, embedded-staff upsert, and channel-generic Customer/Message
upserts all covered.
Suite: 182 -> 211 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Cleanup after successful live validation (10 real customers scraped+stored):
- app/config.py: add yk_userid / yk_company / yk_cookie Settings fields so
existing-session creds can be supplied via .env (bypasses programmatic
login for validation with a live browser session).
- app/storage/repos.py: coerce list values to ';'-joined strings in
_build_column_values. Fixes a real bug: the live friends-list API returns
Customer.deptRoute as a LIST, but dept_route is a scalar String(255)
column. raw_json still preserves the original list (no data loss).
- scripts/smoke_wechat.py:
* Revert build_client_from_env to read os.environ ONLY (no get_settings
fallbacks), so the monkeypatch.setenv unit tests stay hermetic and the
real .env stops leaking in. 3 previously-red tests green again.
* Add _push_settings_into_environ() (Settings -> os.environ, real env
wins, only fills gaps) and call it as the first line of main(); so the
smoke run still honors .env without the pure builder depending on
Settings. .env -> Settings -> os.environ (once) -> build_client_from_env.
* Keep the await _print_summary fix from validation.
- tests/test_repos.py: regression test for list dept_route coercion
(stored joined 'A;B', raw_json keeps the list).
Co-Authored-By: Claude <noreply@anthropic.com>
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds a shared-token auth dependency for the viewing API (Task 2.9). Trust
model: when API_TOKEN is empty (the default) auth is disabled — we trust the
network boundary (local/intranet tool). When set, every request must present
the token via `Authorization: Bearer <token>` or `?token=<token>`, else 401.
`/api/health` is exempt so liveness probes work without the token.
- app/config.py: add `api_token: str = ""` to Settings.
- app/api/auth.py: `require_auth` dependency (secrets.compare_digest, reads
get_settings() per-request so token rotation via cache_clear() works).
- app/api/main.py: apply the dependency globally; document the trust model.
- .env.example: document API_TOKEN.
- tests/test_api_auth.py: 8 tests covering disabled/correct/missing/wrong/
malformed-header/query-param/health-exempt paths.
Co-Authored-By: Claude <noreply@anthropic.com>
浅念