浅念
b0ebe8a15a
用户报微信群列表数据/内容不对,应对齐云客「沟通→微信号管理→微信群统计」 (/cms/wechat/wechatgroup)。真机探明(表单登录+页面导航抓 network): - 数据源=POST /wechatweb/chatroom/getChatroomList(form:departmentId+ startTime/endTime 窗+分页;信封 data.overviews[]/totalCount 1,854 行) - 云客窗是任意时间窗实时算:沟通消息数实测同群全史 417/7天 36/单日 6 - **修正存量映射错误**:member_count 原存 friendNum(员工好友数),真群成员 数是 friendCount/getChatroomList.memberCount - 新增/退出成员数云客按窗实时算,本地无事件流 → 按天积累 实现: - ChatGroup 补 group_remark 列(备注仅 friend_list/es 携带,两归一化均 「有值才写」防互清);迁移含存量 raw_json 回填(须先于基线抓取执行) - 新表 chat_group_member_stats(渠道×账号×群×天一行 added/lost) - 新实体 chatroom-stats:全窗刷档案列(真成员数/群备注/员工信息,owner 经 wechat_account alias 解析)+ --day 单日窗落进出日行;full_sync 5c 步 - /api/groups 扩展:msg_start/msg_end 窗内聚合(window_msg_count= msg_daily_stats 按天和、window_added/lost=日表和)、last_start/end 最新消息窗、group_remark 子串;行补员工微信/员工信息关联列 - GroupList 微信 tab:统计时段(默认近7天)/最新消息时间段/群名称/群备注/ 员工下拉/查询 + 9 列(群信息|群备注|员工微信|员工信息|群成员数量| 新增成员数|退出成员数|沟通消息数|操作);WA tab 不动 700 测试绿(新增窗聚合/筛选/员工列 3 例)。本地实测:基线 1,854 群=云客 total 一字不差,今日日窗 added 407/lost 166 落表,页面 9 列/员工列渲染正常。 Co-Authored-By: Claude Code <noreply@anthropic.com>
109 lines
5.0 KiB
Bash
109 lines
5.0 KiB
Bash
#!/bin/bash
|
|
# FULL page-based re-sync: ALL channels, ALL entities — ZERO API calls, ZERO 风控
|
|
cd "$(dirname "$0")/.."
|
|
# DB_URL:外部已设则沿用(容器里由 docker-compose 注入 MySQL);本地默认 SQLite
|
|
export DB_URL="${DB_URL:-sqlite+aiosqlite:///./smoke.db}"
|
|
# 可移植 python:容器里 python 在 PATH;本地 Windows 用 venv 的解释器(2026-08-21 部署阻塞点#3)
|
|
if [ -f ".venv/Scripts/python.exe" ]; then
|
|
PY=".venv/Scripts/python.exe"
|
|
else
|
|
PY="python"
|
|
fi
|
|
|
|
echo "=============================================="
|
|
echo "$(date) === FULL PAGE-BASED RE-SYNC (NO API) ==="
|
|
echo "=============================================="
|
|
|
|
echo "$(date) --- 1/10 WeChat accounts ---"
|
|
$PY scripts/scrape_via_page.py --entity accounts
|
|
echo "$(date) --- 2/10 WA accounts ---"
|
|
$PY scripts/scrape_via_page.py --entity wa-accounts
|
|
echo "$(date) --- 3/10 WeChat customers ---"
|
|
$PY scripts/scrape_via_page.py --entity friends --max-employees 100
|
|
echo "$(date) --- 4/10 WA customers ---"
|
|
$PY scripts/scrape_via_page.py --entity wa-friends
|
|
echo "$(date) --- 5/10 WeChat groups ---"
|
|
$PY scripts/scrape_via_page.py --entity groups --max-employees 100
|
|
echo "$(date) --- 5b/10 WA groups ---"
|
|
$PY scripts/scrape_via_page.py --entity wa-groups
|
|
# 5c: 微信群统计(getChatroomList,云客「微信群统计」同款)——全窗刷档案列
|
|
# (真成员数/群备注/员工信息)+ 昨日窗落成员进出日行(页面窗内新增/退出)
|
|
echo "$(date) --- 5c/10 WeChat chatroom stats ---"
|
|
$PY scripts/scrape_via_page.py --entity chatroom-stats
|
|
YDAY=$(date -d "yesterday" +%F)
|
|
$PY scripts/scrape_via_page.py --entity chatroom-stats --day "$YDAY"
|
|
echo "$(date) --- 6/10 Voice calls (WeChat + WA) ---"
|
|
$PY scripts/scrape_via_page.py --entity voice-calls
|
|
# 语音重扫会把 record_media_hash 覆盖回 None —— 必须紧跟录音重链接
|
|
$PY -m app.sync.cli sync --entity call_recordings
|
|
echo "$(date) --- 7/10 Voice clips ---"
|
|
$PY scripts/scrape_via_page.py --entity voice-clips
|
|
echo "$(date) --- 8/10 WeChat messages ---"
|
|
$PY scripts/scrape_via_page.py --entity messages --max-conversations 99999
|
|
echo "$(date) --- 8b/10 WeChat group messages + members ---"
|
|
$PY scripts/scrape_via_page.py --entity group-messages --max-conversations 99999
|
|
$PY -m scripts.backfill_group_members
|
|
echo "$(date) --- 9/10 WA messages ---"
|
|
$PY scripts/scrape_via_page.py --entity wa-messages --max-conversations 99999
|
|
echo "$(date) --- 9b/10 WA group messages ---"
|
|
$PY scripts/scrape_via_page.py --entity wa-group-messages --max-conversations 99999
|
|
echo "$(date) --- 9c/10 Moments (员工朋友圈) ---"
|
|
$PY scripts/scrape_via_page.py --entity moments
|
|
echo "$(date) --- 10/10 Media download (audio+images) ---"
|
|
$PY scripts/scrape_via_page.py --entity media
|
|
# 11:会话日汇总全量重建(夜间自愈:绕过 upsert 钩子的写入由它纠正;
|
|
# sessions/staff-chat-stats 读路径依赖此表,2026-09-03)
|
|
echo "$(date) --- 11/12 Daily-stats rebuild ---"
|
|
$PY -m app.sync.cli daily-stats
|
|
# 12:员工×日汇总全量重建(rollup 层随 msg_daily_stats 重建同步自愈;
|
|
# A1 员工指标读路径依赖此表,2026-09-04)
|
|
echo "$(date) --- 12/12 Staff-daily-stats rebuild ---"
|
|
$PY -m app.sync.cli staff-daily-stats
|
|
|
|
echo "=============================================="
|
|
echo "$(date) === FULL RE-SYNC COMPLETE ==="
|
|
echo "=============================================="
|
|
|
|
# 终局数据快照:走 app 引擎(DB_URL 可移植,SQLite/MySQL 通吃;原 sqlite3
|
|
# 直连 smoke.db 的写法在服务器 MySQL 上会炸,2026-09-03 修)
|
|
$PY - <<'PYEOF'
|
|
import asyncio
|
|
|
|
from sqlalchemy import text
|
|
from sqlalchemy.ext.asyncio import create_async_engine
|
|
|
|
from app.config import get_settings
|
|
|
|
TABLES = [
|
|
("wechat accounts", "SELECT COUNT(*) FROM wechat_account"),
|
|
("whatsapp accounts", "SELECT COUNT(*) FROM whatsapp_account"),
|
|
("wechat customers", "SELECT COUNT(*) FROM customer WHERE channel='wechat'"),
|
|
("whatsapp customers", "SELECT COUNT(*) FROM customer WHERE channel='whatsapp'"),
|
|
("wechat groups", "SELECT COUNT(*) FROM chat_group WHERE channel='wechat'"),
|
|
("whatsapp groups", "SELECT COUNT(*) FROM chat_group WHERE channel='whatsapp'"),
|
|
("wechat messages", "SELECT COUNT(*) FROM message WHERE channel='wechat'"),
|
|
("whatsapp messages", "SELECT COUNT(*) FROM message WHERE channel='whatsapp'"),
|
|
("daily stats rows", "SELECT COUNT(*) FROM msg_daily_stats"),
|
|
("voice calls", "SELECT COUNT(*) FROM phone_call WHERE channel IN ('voice_call','wa_voice_call')"),
|
|
("voice clips", "SELECT COUNT(*) FROM voice_clip"),
|
|
("moments posts", "SELECT COUNT(*) FROM moments_post"),
|
|
("media files", "SELECT COUNT(*) FROM media"),
|
|
]
|
|
|
|
|
|
async def main() -> None:
|
|
engine = create_async_engine(get_settings().db_url)
|
|
try:
|
|
async with engine.connect() as conn:
|
|
print()
|
|
print("=== FINAL DATA SUMMARY ===")
|
|
for label, sql in TABLES:
|
|
n = (await conn.execute(text(sql))).scalar_one()
|
|
print(f" {label:22s} {n:>6}")
|
|
finally:
|
|
await engine.dispose()
|
|
|
|
|
|
asyncio.run(main())
|
|
PYEOF
|