yunke/docs/superpowers/plans/2026-08-07-yunke-plan4a-wa-voice-scrape.md

Plan 4a: WA 语音通话抓取 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: 新增 WhatsApp 语音通话抓取（scrape_wa_voice_calls），复用云客统一语音接口 queryWeChatVoiceList + callSource:"4"，存入 PhoneCall 表（channel='wa_voice_call'），并挂进 scheduler + smoke 实跑验证，为 Plan 4b（CallList UI 对齐 WA tab）提供数据。

Architecture: 复用现有 scrape_wechat_voice_calls（app/scraper/wechat.py:1222）的翻页/解析/upsert 模式，在 app/scraper/whatsapp.py 新增 scrape_wa_voice_calls——唯一差异：请求 body 加 callSource:"4"（=WA；微信="0"）、归一化 channel="wa_voice_call"。不加 PhoneCall 列、不做 Alembic 迁移（yunke 用 Base.metadata.create_all，无 alembic 目录）：WA 特有字段（weChatId=ownerWA/callFileId/nextId/isRoom）落 raw_json，员工用现有 user_id 列筛。

Tech Stack: Python 3.12 + httpx + respx（scraper 测试）+ FastAPI/SQLAlchemy（复用）。

Spec/事实依据:

• docs/superpowers/specs/2026-08-06-yunke-comm-operations-audit.md（核对表 §6 WA 语音 5 列 + 操作）
• Phase 0 浏览器采集（2026-08-06/07）：WA wavoicecalllist 页数据请求 = POST /wechatweb/callRecord/queryWeChatVoiceList，body {"pageNum":1,"pageSize":10,"startTime":"...","endTime":"...","departmentId":"...","isRoom":"0","callSource":"4","ids":[],"userId":""}，response data.total=143 + 每行字段（见下）。微信页同接口 callSource:"0"。
• 代码分析（Explore）：scrape_wechat_voice_calls（wechat.py:1222）已实现且 URL 正确；smoke.db phone_call 0 行是因 scripts/smoke_wechat.py 没调语音 scraper（非代码错）；whatsapp.py 无 voice scraper；upsert_voice_call（repos.py:515）复用 _upsert_one(PhoneCall)；PhoneCall 无 owner_account_id 列（weChatId 落 raw_json）。

---

范围

本 plan 覆盖（4a 抓取）：

• 新增 scrape_wa_voice_calls（whatsapp.py）+ TDD 测试
• 新增 sync_wa_voice_calls（sync 编排）+ scheduler ENTITY_FUNCS["wa_voice_calls"] + 接进 _run_all_wechat/_run_all
• smoke 脚本挂载 WA + 微信语音 scraper 实跑，验证填 smoke.db

不在本 plan 范围：

• CallList UI 对齐（WA tab/列结构/长语音 modal/统计卡片/录音异常检查/工具栏）—— Plan 4b（依赖本 plan 产出的真实数据字段）。
• 长语音片段抓取（/wechatweb/call/list）—— 随 Plan 4b 长语音 modal。
• 微信语音 scraper 改造（现状能用；若 smoke 实测返回含 WA 数据，本 plan Task 4 补 callSource:"0" 加固）。
• PhoneCall 加列 / Alembic 迁移（刻意避免：无 alembic 目录；channel 区分 + raw_json 足够）。

WA 语音字段映射（Phase 0 实测 response → PhoneCall 列）

云客字段	PhoneCall 列	说明
id	call_id (PK)
—	channel = "wa_voice_call"	区分微信(voice_call)/电话(phone_call)
startTime "2026-08-06 11:28:22"	start_call_time	%Y-%m-%d %H:%M:%S → ms epoch（naive UTC，同微信语音）
duration "82"	call_duration	字符串秒 → int
callStatus (1=接通有录音/2=未接通)	call_status	int
isSend (0=呼入/1=呼出)	incoming_call	翻转 1 - isSend（同微信语音，对齐 PhoneCall 极性 1=呼入）
userId	user_id	员工
userName	user_id_name	员工姓名
talker (584142462772@s.whatsapp.net)	call_number	客户 WA id
talkerNickName	plan_customer_name	脱敏昵称
ossFileName (mp3 URL 或 null)	record_file	录音直链
version	version
weChatId/callFileId/nextId/isRoom/from 等	raw_json	WA 特有，无对应列

---

运行命令（验证用）

• 单测：cd yunke-server && .venv/Scripts/python.exe -m pytest tests/test_scrape_wa_voice_calls.py -q
• 全后端：cd yunke-server && .venv/Scripts/python.exe -m pytest tests/ -q
• smoke 实跑（填 smoke.db）：cd yunke-server && DB_URL="sqlite+aiosqlite:///./smoke.db" .venv/Scripts/python.exe -m app.sync.cli sync --entity wa_voice_calls（.env 有 YK_USERID/COMPANY/COOKIE）
• 查 smoke.db：cd yunke-server && .venv/Scripts/python.exe -c "import sqlite3;c=sqlite3.connect('smoke.db');print(c.execute(\"select channel,count(*) from phone_call group by channel\").fetchall())"

---

File Map

• Modify app/scraper/whatsapp.py — 新增 scrape_wa_voice_calls + _build_wa_voice_body/_normalize_wa_voice_call/_parse_wa_voice_total/_parse_wa_voice_rows + 常量 + __all__
• Create tests/test_scrape_wa_voice_calls.py — TDD（respx + cassette，照 test_scrape_wechat_voice_calls.py）
• Create tests/cassettes/wa_voice_fixture.py — WA 响应 fixture（照 voiceCall_fixture.py）
• Modify sync 编排（app/sync/ 下含 sync_wechat_voice_calls 的模块）— 新增 sync_wa_voice_calls
• Modify app/sync/scheduler.py — ENTITY_FUNCS["wa_voice_calls"] + 接进 _run_all_wechat/_run_all
• Modify scripts/smoke_wechat.py（或新 scripts/smoke_calls.py）— 挂载 voice/wa_voice/phone/sms scraper

---

Task 1: WA 语音 scraper + 解析（TDD）

事实：scrape_wechat_voice_calls（wechat.py:1222）用 paginate_by_page + VOICE_LIST_PATH="/wechatweb/callRecord/queryWeChatVoiceList" + _parse_voice_total（读 data.total）/_parse_voice_rows（读 data.data）+ _build_voice_body（{pageNum,pageSize}）+ _normalize_voice_call。WA 用同接口同 envelope，仅 body 加 callSource:"4"、channel 改 wa_voice_call。paginate_by_page 签名见 scraper/base.py。coerce_int_or_none + parse_datetime_to_ms 都从 app.scraper.parsers（whatsapp.py:36 已 import，直接用）。

Files:

• Create: tests/cassettes/wa_voice_fixture.py

• Create: tests/test_scrape_wa_voice_calls.py

• Modify: app/scraper/whatsapp.py

• Step 1: 写 WA 响应 fixture（tests/cassettes/wa_voice_fixture.py，照 tests/cassettes/voiceCall_fixture.py 模式）：导出 WA_VOICE_RECORDS（2-3 条 WA 记录，字段对齐 Phase 0 实测：id/userId/userName/userWeChatNickName/userWeChatAlias/talker/talkerNickName/talkerAlias/callType/isSend/callStatus/ossFileName/duration/startTime/weChatId/callFileId/nextId/isRoom/version）+ make_wa_voice_list_response(records, total, page_num, page_size)（返回 {"message":"调用成功","code":"10000","data":{"pageNum":...,"pageSize":...,"total":total,"data":records}} envelope）。先读 voiceCall_fixture.py 对齐 make_voice_list_response 的确切结构。

• Step 2: 写失败测试（tests/test_scrape_wa_voice_calls.py，照 test_scrape_wechat_voice_calls.py 的 respx 模式）：

from __future__ import annotations
import httpx, respx
from app.scraper.whatsapp import scrape_wa_voice_calls
from app.session.yunke_client import YunkeClient
from tests.cassettes.wa_voice_fixture import WA_VOICE_RECORDS, TOTAL, make_wa_voice_list_response

BASE = "https://crm2.yunkecn.com"
VOICE_URL = f"{BASE}/wechatweb/callRecord/queryWeChatVoiceList"

def _primed_client() -> YunkeClient:
    return YunkeClient(base_url=BASE, userid="a"*32, company="8m0atd")

@respx.mock
async def test_scrape_wa_voice_calls_yields_records_and_paginates():
    """两页 + 空页终止 -> 全部记录；body 含 callSource=4；channel=wa_voice_call。"""
    page1 = make_wa_voice_list_response(WA_VOICE_RECORDS[:2], total=TOTAL, page_num=1, page_size=2)
    page2 = make_wa_voice_list_response(WA_VOICE_RECORDS[2:], total=TOTAL, page_num=2, page_size=2)
    page3 = make_wa_voice_list_response([], total=TOTAL, page_num=3, page_size=2)
    route = respx.post(VOICE_URL).mock(side_effect=[
        httpx.Response(200, json=page1), httpx.Response(200, json=page2), httpx.Response(200, json=page3)])
    client = _primed_client()
    try:
        results = [r async for r in scrape_wa_voice_calls(client, page_size=2)]
    finally:
        await client.aclose()
    assert route.call_count == 3
    assert len(results) == len(WA_VOICE_RECORDS)
    import json as _json
    body1 = _json.loads(route.calls[0].request.content.decode())
    assert body1["callSource"] == "4"   # 关键：WA 区分
    assert body1["pageNum"] == 1 and body1["pageSize"] == 2
    for result, record in zip(results, WA_VOICE_RECORDS, strict=True):
        assert result["channel"] == "wa_voice_call"
        assert result["call_id"] == record["id"]
        assert result["call_number"] == record["talker"]              # friendWA
        assert result["record_file"] == record["ossFileName"]
        assert result["call_status"] == record["callStatus"]
        assert result["raw_json"] == record                            # weChatId/callFileId 保 raw_json
        # incoming_call 翻转
        assert result["incoming_call"] == 1 - record["isSend"]

@respx.mock
async def test_scrape_wa_voice_calls_stops_on_empty_first_page():
    empty = make_wa_voice_list_response([], total=0, page_num=1)
    route = respx.post(VOICE_URL).mock(return_value=httpx.Response(200, json=empty))
    client = _primed_client()
    try:
        results = [r async for r in scrape_wa_voice_calls(client)]
    finally:
        await client.aclose()
    assert results == [] and route.call_count == 1

implementer：参照 test_scrape_wechat_voice_calls.py 的 _ms/断言风格补全（start_call_time 的 ms 断言等）。先确认 voiceCall_fixture.py 的 TOTAL/DEFAULT_PAGE_SIZE 命名，WA fixture 对齐。

• Step 3: 跑测试确认 FAIL

cd yunke-server && .venv/Scripts/python.exe -m pytest tests/test_scrape_wa_voice_calls.py -x -q

预期：FAIL（scrape_wa_voice_calls 不存在，ImportError）。

• Step 4: 实现 WA scraper（app/scraper/whatsapp.py，加在文件末尾，import paginate_by_page 已在 L35）：

# ===========================================================================
# Voice-call scraper — POST /wechatweb/callRecord/queryWeChatVoiceList (callSource=4)
# 复用云客统一语音接口（微信 callSource=0 / WA callSource=4）；envelope 同微信语音
# (data.data / data.total)。channel="wa_voice_call" 区分。
# ===========================================================================

WA_VOICE_LIST_PATH = "/wechatweb/callRecord/queryWeChatVoiceList"


def _parse_wa_voice_total(resp: Any) -> int | None:
    """data.total（同微信语音 envelope）。"""
    try:
        body = resp.json()
    except (AttributeError, ValueError):
        return None
    data = body.get("data") if isinstance(body, dict) else None
    if not isinstance(data, dict):
        return None
    total = data.get("total")
    return int(total) if isinstance(total, (int, float)) and not isinstance(total, bool) else None


def _parse_wa_voice_rows(resp: Any) -> list[dict[str, Any]]:
    """data.data（同微信语音 envelope）。"""
    try:
        body = resp.json()
    except (AttributeError, ValueError):
        return []
    data = body.get("data") if isinstance(body, dict) else None
    if not isinstance(data, dict):
        return []
    rows = data.get("data")
    return rows if isinstance(rows, list) else []


def _build_wa_voice_body(page_num: int, page_size: int) -> dict[str, Any]:
    """WA 语音请求体：分页 + callSource=4（WA）。"""
    return {"pageNum": page_num, "pageSize": page_size, "callSource": "4"}


def _normalize_wa_voice_call(record: dict[str, Any]) -> dict[str, Any]:
    """Map one WA queryWeChatVoiceList record -> PhoneCall shape (channel="wa_voice_call")。

    字段映射同微信语音（_normalize_voice_call），唯二差异：channel 标 wa_voice_call；
    WA 特有字段（weChatId=ownerWA / callFileId / nextId / isRoom / from）无对应列，落 raw_json。
    isSend 翻转极性（0=呼入 -> incoming_call=1）。None-safe。
    """
    is_send = coerce_int_or_none(record.get("isSend"))
    incoming_call = None if is_send is None else 1 - is_send
    return {
        "call_id": record.get("id"),
        "channel": "wa_voice_call",
        "start_call_time": parse_datetime_to_ms(record.get("startTime"), _WA_DATETIME_FORMATS),
        "call_duration": coerce_int_or_none(record.get("duration")),
        "call_status": coerce_int_or_none(record.get("callStatus")),
        "incoming_call": incoming_call,
        "user_id": record.get("userId"),
        "user_id_name": record.get("userName"),
        "call_number": record.get("talker"),
        "plan_customer_name": record.get("talkerNickName"),
        "record_file": record.get("ossFileName"),
        "version": record.get("version"),
        "raw_json": record,
    }


async def scrape_wa_voice_calls(
    client: YunkeClient,
    *,
    page_size: int = 50,
) -> AsyncIterator[dict[str, Any]]:
    """Yield normalized WA voice-call records (queryWeChatVoiceList + callSource=4)."""
    async for record in paginate_by_page(
        client,
        WA_VOICE_LIST_PATH,
        page_size=page_size,
        parse_total=_parse_wa_voice_total,
        parse_rows=_parse_wa_voice_rows,
        build_body=_build_wa_voice_body,
    ):
        yield _normalize_wa_voice_call(record)

注意：coerce_int_or_none/parse_datetime_to_ms 从 app.scraper.parsers（L36 已 import）。_WA_DATETIME_FORMATS（L50）含 %Y-%m-%d %H:%M:%S，WA startTime 是这格式——直接用（比微信的 _parse_voice_start_time 更简，复用 WA 既有格式列表）。把 "scrape_wa_voice_calls" 加入 __all__（L39-43）。

• Step 5: 跑测试确认 PASS

cd yunke-server && .venv/Scripts/python.exe -m pytest tests/test_scrape_wa_voice_calls.py -x -q

• Step 6: 全后端回归

cd yunke-server && .venv/Scripts/python.exe -m pytest tests/ -q

预期：全绿（原 ≥333 + 新增）。

• Step 7: Commit

git add yunke-server/app/scraper/whatsapp.py yunke-server/tests/test_scrape_wa_voice_calls.py yunke-server/tests/cassettes/wa_voice_fixture.py
git commit -m "$(cat <<'EOF'
feat(scraper): WA 语音通话抓取 scrape_wa_voice_calls（callSource=4）

复用云客统一语音接口 queryWeChatVoiceList + callSource=4；channel=wa_voice_call
区分；WA 特有字段落 raw_json；复用 PhoneCall 表 + upsert_voice_call。

Co-Authored-By: Claude <noreply@anthropic.com>
EOF
)"

---

Task 2: sync 编排 + scheduler 注册

事实：scheduler.py:149-152 _run_voice_calls → sync_wechat_voice_calls（在 app/sync/ 下，grep 定位）。ENTITY_FUNCS（scheduler.py:212-225）含 "voice_calls": _run_voice_calls。_run_all_wechat（:182-193）顺序含 voice_calls；_run_all（:196-206）= _run_all_wechat + phone_calls + sms。upsert_voice_call（repos.py:515）。

Files:

• Modify: app/sync/（含 sync_wechat_voice_calls 的模块，grep 定位）

• Modify: app/sync/scheduler.py

• Step 1: 照写 sync_wa_voice_calls（orchestrator.py）。sync_wechat_voice_calls 在 app/sync/orchestrator.py:487（签名 (session: AsyncSession, client: YunkeClient) -> SyncResult，调 scrape_wechat_voice_calls + upsert_voice_call）。读它，紧邻照写 sync_wa_voice_calls(session, client)（调 scrape_wa_voice_calls + upsert_voice_call，其余 session/日志/计数/SyncResult 照搬）。import scrape_wa_voice_calls from app.scraper.whatsapp。

• Step 2: 接进 daily 序列 + scheduler ENTITY_FUNCS。 (a) daily 序列在 orchestrator.py，不是 scheduler.py：sync_all_wechat（orchestrator.py 约 :576-582）的串行序列里，在 await sync_wechat_voice_calls(session, client) 那行后插入 await sync_wa_voice_calls(session, client),。这让 _run_all/daily cron 覆盖 WA voice（scheduler 的 _run_all_wechat 只是委托给 sync_all_wechat，实体顺序在后者）。 (b) scheduler ENTITY_FUNCS（单实体 CLI 用）：scheduler.py 加 helper——照 _run_voice_calls（:149-152）签名 (session, client, store) -> list[SyncResult]，不是 (client)->int：

  async def _run_wa_voice_calls(session, client, store) -> list[SyncResult]:
      return [await sync_wa_voice_calls(session, client)]
  

  ENTITY_FUNCS 加 "wa_voice_calls": _run_wa_voice_calls。import sync_wa_voice_calls（from orchestrator）+ SyncResult。

• Step 3: 回归

cd yunke-server && .venv/Scripts/python.exe -m pytest tests/ -q

（若 scheduler 有测试，确认 ENTITY_FUNCS 含新键；若无 scheduler 测试则全绿即可。）

• Step 4: CLI 单实体冒烟（可选，需 cookie）

cd yunke-server && DB_URL="sqlite+aiosqlite:///./smoke.db" .venv/Scripts/python.exe -m app.sync.cli sync --entity wa_voice_calls

预期：不报错，抓取若干 WA 语音（143 total，按 page_size 翻页）。若 cookie 失效会报 401——则跳过，靠 Task 3 smoke 验证。

• Step 5: Commit

git add yunke-server/app/sync/ yunke-server/app/sync/scheduler.py
git commit -m "$(cat <<'EOF'
feat(sync): sync_wa_voice_calls + scheduler ENTITY_FUNCS wa_voice_calls

Co-Authored-By: Claude <noreply@anthropic.com>
EOF
)"

---

Task 3: smoke 脚本挂载 + 实跑验证

事实：scripts/smoke_wechat.py 只调 scrape_wechat_friends + scrape_wechat_messages（smoke.db voice 0 行根因）。CLI app.sync.cli sync --entity <name> 走 ENTITY_FUNCS。

Files:

• Modify: scripts/smoke_wechat.py（或新建 scripts/smoke_calls.py）

• Step 1: 挂载语音 scraper 到 smoke。读 scripts/smoke_wechat.py，在 friends/messages 调用旁加 voice_calls + wa_voice_calls（+ 可选 phone_calls/sms）的 sync 调用。或新建 scripts/smoke_calls.py 调 sync_wechat_voice_calls + sync_wa_voice_calls + sync_phone_calls + sync_sms。参照现有 smoke 脚本的 client 构造（cookie from .env）+ DB_URL 写入 smoke.db 模式。

• Step 2: 实跑填 smoke.db（需 .env 有效 cookie）：

cd yunke-server && DB_URL="sqlite+aiosqlite:///./smoke.db" .venv/Scripts/python.exe scripts/smoke_calls.py

（或逐实体 CLI：... sync --entity voice_calls 然后 wa_voice_calls）

• Step 3: 验证 smoke.db 有数据

cd yunke-server && .venv/Scripts/python.exe -c "import sqlite3;c=sqlite3.connect('smoke.db');print(c.execute(\"select channel,count(*) from phone_call group by channel\").fetchall())"

预期：含 ('voice_call', N) + ('wa_voice_call', M)（M≈143 若全抓）。若 voice_call 仍 0，说明微信 scraper 不带 callSource 返回空/异常 → 触发 Task 4。

• Step 4: 后端 API 抽查

curl -s "http://localhost:8000/api/calls?channel=wa_voice_call&page_size=3" | head -c 500

（需后端跑：uvicorn app.api.main:app --port 8000）预期：返回 WA 语音通话 items。

• Step 5: Commit

git add yunke-server/scripts/smoke_calls.py yunke-server/scripts/smoke_wechat.py
git commit -m "$(cat <<'EOF'
feat(scripts): smoke 挂载 voice/wa_voice/phone/sms scraper（验证抓取填 smoke.db）

Co-Authored-By: Claude <noreply@anthropic.com>
EOF
)"

---

Task 4（条件）: 微信语音 callSource=0 加固

仅当 Task 3 实测发现：微信 scrape_wechat_voice_calls（不带 callSource）返回的数据含 WA 记录（talker 是 @s.whatsapp.net）或返回空 → 才执行本 task。若微信语音正常返回（纯 wxid talker），跳过本 task。

Files: Modify app/scraper/wechat.py + tests/test_scrape_wechat_voice_calls.py

• Step 1: 给 _build_voice_body 加 call_source 参数（wechat.py:1143）：

def _build_voice_body(page_num: int, page_size: int, *, call_source: str = "0") -> dict[str, Any]:
    return {"pageNum": page_num, "pageSize": page_size, "callSource": call_source}

scrape_wechat_voice_calls 调用处传 call_source="0"（或 paginate_by_page 的 build_body kwargs）。

• Step 2: 更新现有测试（test_scrape_wechat_voice_calls.py 的 body 断言，:232-238）：加 assert body1["callSource"] == "0"。

• Step 3: 回归 + smoke 复跑确认微信纯数据

cd yunke-server && .venv/Scripts/python.exe -m pytest tests/test_scrape_wechat_voice_calls.py -q

• Step 4: Commit

git add yunke-server/app/scraper/wechat.py yunke-server/tests/test_scrape_wechat_voice_calls.py
git commit -m "$(cat <<'EOF'
fix(scraper): 微信语音 body 加 callSource=0 明确区分（防混入 WA）

Co-Authored-By: Claude <noreply@anthropic.com>
EOF
)"

---

收尾验收

• pytest tests/ -q 全绿（≥333 + 新增 WA voice 测试）。
• smoke.db phone_call 表含 channel='wa_voice_call' 行（实跑验证抓取链路通）。
• GET /api/calls?channel=wa_voice_call 返回 WA 语音数据。
• scheduler ENTITY_FUNCS 含 wa_voice_calls，_run_all 覆盖。
• 为 Plan 4b 留好的数据基础：WA 语音字段（talker=friendWA、user_id=员工、record_file=录音URL、call_status/isSend）已结构化进 PhoneCall，raw_json 保留 weChatId/callFileId 供 4b modal/匹配用。

---

不在本 plan 范围（Plan 4b）

• CallList.vue 加 WA tab + WA 列结构（状态/客户信息/员工信息/通话时间和录音/操作）。
• 长语音列表 modal（/wechatweb/call/list 抓取 + voice_clip 存储 + 匹配 UI）。
• 统计卡片（todayStatistics）、录音异常检查（/call/check/{id} 代理）、批量语音下载/全部导出/高级筛选工具栏。
• 微信好友/微信群 tab（isRoom 区分）—— 若 4b 需要，从 raw_json.isRoom 读或加列。