xinyouhe/yunke
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(group-stats): 微信群列表对齐云客「微信群统计」——getChatroomList 数据源+窗内统计

Browse Source 
用户报微信群列表数据/内容不对,应对齐云客「沟通→微信号管理→微信群统计」
(/cms/wechat/wechatgroup)。真机探明(表单登录+页面导航抓 network):

- 数据源=POST /wechatweb/chatroom/getChatroomList(form:departmentId+
  startTime/endTime 窗+分页;信封 data.overviews[]/totalCount 1,854 行)
- 云客窗是任意时间窗实时算:沟通消息数实测同群全史 417/7天 36/单日 6
- **修正存量映射错误**:member_count 原存 friendNum(员工好友数),真群成员
  数是 friendCount/getChatroomList.memberCount
- 新增/退出成员数云客按窗实时算,本地无事件流 → 按天积累

实现:
- ChatGroup 补 group_remark 列(备注仅 friend_list/es 携带,两归一化均
  「有值才写」防互清);迁移含存量 raw_json 回填(须先于基线抓取执行)
- 新表 chat_group_member_stats(渠道×账号×群×天一行 added/lost)
- 新实体 chatroom-stats:全窗刷档案列(真成员数/群备注/员工信息,owner
  经 wechat_account alias 解析)+ --day 单日窗落进出日行;full_sync 5c 步
- /api/groups 扩展:msg_start/msg_end 窗内聚合(window_msg_count=
  msg_daily_stats 按天和、window_added/lost=日表和)、last_start/end
  最新消息窗、group_remark 子串;行补员工微信/员工信息关联列
- GroupList 微信 tab:统计时段(默认近7天)/最新消息时间段/群名称/群备注/
  员工下拉/查询 + 9 列(群信息|群备注|员工微信|员工信息|群成员数量|
  新增成员数|退出成员数|沟通消息数|操作);WA tab 不动

700 测试绿(新增窗聚合/筛选/员工列 3 例)。本地实测:基线 1,854 群=云客
total 一字不差,今日日窗 added 407/lost 166 落表,页面 9 列/员工列渲染正常。

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
浅念
2026-09-09 18:09:20 +08:00
parent 03c08881b6
commit b0ebe8a15a
12 changed files with 657 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
yunke-server/scripts/full_sync_all.sh
+6
View File
@@ -26,6 +26,12 @@ echo "$(date) --- 5/10 WeChat groups ---"
$PY scripts/scrape_via_page.py --entity groups --max-employees 100
echo "$(date) --- 5b/10 WA groups ---"
$PY scripts/scrape_via_page.py --entity wa-groups
# 5c: 微信群统计(getChatroomList,云客「微信群统计」同款)——全窗刷档案列
# (真成员数/群备注/员工信息)+ 昨日窗落成员进出日行(页面窗内新增/退出)
echo "$(date) --- 5c/10 WeChat chatroom stats ---"
$PY scripts/scrape_via_page.py --entity chatroom-stats
YDAY=$(date -d "yesterday" +%F)
$PY scripts/scrape_via_page.py --entity chatroom-stats --day "$YDAY"
echo "$(date) --- 6/10 Voice calls (WeChat + WA) ---"
$PY scripts/scrape_via_page.py --entity voice-calls
# 语音重扫会把 record_media_hash 覆盖回 None —— 必须紧跟录音重链接
yunke-server/scripts/scrape_via_page.py
+161
View File
@@ -1631,6 +1631,163 @@ async def scrape_wa_groups(cookie_str: str, max_pages: int = 200):
return total_fetched, total_upserted
# ---------------------------------------------------------------------------
# 微信群统计 getChatroomList(2026-09-09 真机探明,云客同款数据源)
#
# 云客「沟通→微信号管理→微信群统计」页(/cms/wechat/wechatgroup)数据源:
# POST /wechatweb/chatroom/getChatroomList(form-urlencoded)
# departmentId + startTime/endTime('YYYY-MM-DD HH:MM:SS' 时间窗) + 分页;
# 信封 data.overviews[] + totalCount(1,854 行,员工×群维度)。
# 行含:真 memberCount(群成员数量——friend_list/es 的 friendNum 其实是
# 员工好友数,旧映射存反了)、remark(群备注)、员工微信(wechatId=微信号
# alias/wechatNickName/头像)、员工信息(userName/userPhone)、以及**窗内**
# addMembers/lossMembers(新增/退出成员数,任意窗实时算)。
#
# 本地无成员进出事件流 → add/loss 只能按天积累:每夜抓「当日窗」落
# chat_group_member_stats 日行,页面任意日窗=按天求和(历史无数据)。
# ---------------------------------------------------------------------------
CHATROOM_LIST_PATH = "/wechatweb/chatroom/getChatroomList"
def _normalize_chatroom_stat_row(row: dict, owner_wxid: str) -> dict:
"""getChatroomList 行 → ChatGroup 补列记录。
只放该端点有的字段——未出现的列不参与 upsert 的 UPDATE 子句,
friend_list/es 抓的群主信息等旧值得以保留。
"""
rec = {
"room_id": row.get("chatRoomId"),
"channel": "wechat",
"owner_account_id": owner_wxid,
"room_name": row.get("chatRoomName"),
# 真群成员数(修正:旧源 friendNum 是「员工好友数」被误存此列)
"member_count": row.get("memberCount"),
"head_image_url": row.get("headImageUrl"),
"owner_account_nickname": row.get("wechatNickName"),
"owner_user_id": row.get("userId"),
"owner_user_phone": row.get("userPhone"),
"message_count": row.get("messageCount"),
"own_message_count": row.get("ownMessageCount"),
"add_members": row.get("addMembers"), # 本窗快照(日窗=当日进出)
"loss_members": row.get("lossMembers"),
"last_chat_time": row.get("lastChatTime"),
"update_time": row.get("updateTime"),
"raw_json": row,
}
# 群备注:getChatroomList 行通常无 remark 键(云客页面也显示 -);有才写,
# 无则不放该键——upsert 不更新未出现的列,friend_list/es 来源的
# friendWxRemark 备注得以保留
if row.get("remark"):
rec["group_remark"] = row["remark"]
return rec
async def _chatroom_rows(page, dept: str, start_str: str, end_str: str, max_pages: int = 100):
"""翻完 getChatroomList 一个窗,返回全部行(list)。"""
rows_all: list[dict] = []
page_num = 1
while page_num <= max_pages:
body = await _browser_fetch(
page, CHATROOM_LIST_PATH,
form_body={
"departmentId": dept, "startTime": start_str, "endTime": end_str,
"pageNum": page_num, "pageSize": 50,
},
)
data = body.get("data") or {}
rows = data.get("overviews") or []
rows_all.extend(rows)
total = data.get("totalCount") or 0
logger.info(f" chatroom-list page {page_num}: {len(rows)} rows (total {total})")
if not rows or page_num * 50 >= (total or 0):
break
page_num += 1
await asyncio.sleep(1.2)
return rows_all
async def _owner_alias_map(conn):
"""wechat_account 的 alias/nick_name → wx_id 映射。
getChatroomList 行的 wechatId 是员工**微信号**(alias,如 Nancyjade)
而非 wxId,owner 归属需经此解析;miss 时回退存原值(不影响其余列)。
"""
from sqlalchemy import text as sa_text
result = await conn.execute(sa_text(
"SELECT alias, nick_name, wx_id FROM wechat_account "
"WHERE wx_id IS NOT NULL AND wx_id != ''"
))
mapping: dict[str, str] = {}
for alias, nick, wx in result.fetchall():
if alias:
mapping[str(alias)] = wx
if nick:
mapping.setdefault(str(nick), wx)
return mapping
async def scrape_chatroom_stats(cookie_str: str, day: str | None = None):
"""getChatroomList 抓取(云客微信群统计同款)。
* ``day=None``:全窗(2020 起)——chat_group 档案补列基线(真成员数/
群备注/员工信息列;add/loss 存全窗快照);
* ``day='YYYY-MM-DD'``:单日窗——落 chat_group_member_stats 当日行
(页面任意日窗的新增/退出=按天求和)。
"""
import datetime as _dt
from app.db.base import async_engine
from app.storage.repos import upsert_chat_group, upsert_chat_group_member_stats
dept = await _root_dept_id()
async with async_engine.connect() as conn:
alias_map = await _owner_alias_map(conn)
if day:
start_str, end_str = f"{day} 00:00:00", f"{day} 23:59:59"
day_key = int(day.replace("-", ""))
else:
start_str = "2020-01-01 00:00:00"
end_str = f"{_dt.date.today()} 23:59:59"
day_key = None
group_recs: list[dict] = []
stats_recs: list[dict] = []
async with PageScraper(cookie_str) as scraper:
page = await _open_fetch_page(scraper)
for row in await _chatroom_rows(page, dept, start_str, end_str):
rid = row.get("chatRoomId")
if not rid:
continue
owner_wxid = (
alias_map.get(str(row.get("wechatId") or ""))
or alias_map.get(str(row.get("wechatNickName") or ""))
or str(row.get("wechatId") or "")
)
group_recs.append(_normalize_chatroom_stat_row(row, owner_wxid))
if day_key is not None:
stats_recs.append({
"channel": "wechat",
"owner_account_id": owner_wxid,
"room_id": rid,
"day": day_key,
"added": row.get("addMembers") or 0,
"lost": row.get("lossMembers") or 0,
})
if group_recs:
await _upsert_all(upsert_chat_group, group_recs)
if stats_recs:
await _upsert_all(upsert_chat_group_member_stats, stats_recs)
logger.info(
f"DONE: chatroom-stats day={day or '全窗'} groups={len(group_recs)} "
f"stats_rows={len(stats_recs)}"
)
return len(group_recs), len(stats_recs)
# ---------------------------------------------------------------------------
# 员工朋友圈 Moments (browser fetch, per employee)
# ---------------------------------------------------------------------------
@@ -2301,6 +2458,7 @@ async def main():
"wa-accounts", # WA员工账号 (waBusiness/userlist interception)
"wa-friends", # WA客户 (waBusiness/friendlist interception)
"wa-groups", # WA群 (browser fetch businessFriendList type=1)
"chatroom-stats", # 微信群统计 (getChatroomList 带窗,云客同款)
"voice-calls", # 语音通话记录 (voice-call-list + wavoicecalllist)
"voice-clips", # 长语音 (browser fetch call/list per employee)
"wa-messages", # WA消息 (browser fetch chatRecords per conversation)
@@ -2315,6 +2473,7 @@ async def main():
parser.add_argument("--max-pages", type=int, default=200, help="列表翻页上限 (accounts/wa-accounts/wa-friends/voice-calls/moments)")
parser.add_argument("--batch-size", type=int, default=50, help="媒体下载批次大小 (media)")
parser.add_argument("--full-history", action="store_true", help="消息类实体强制全量翻历史(默认增量水位)")
parser.add_argument("--day", type=str, default=None, help="chatroom-stats:单日窗 YYYY-MM-DD(落按天进出表);缺省=全窗档案基线")
args = parser.parse_args()
cookie = _load_cookie()
@@ -2348,6 +2507,8 @@ async def _dispatch_entity(args, cookie: str) -> None:
await scrape_wa_friends(cookie, args.max_pages)
elif args.entity == "wa-groups":
await scrape_wa_groups(cookie, args.max_pages)
elif args.entity == "chatroom-stats":
await scrape_chatroom_stats(cookie, args.day)
elif args.entity == "voice-calls":
await scrape_voice_calls(cookie, args.max_pages)
elif args.entity == "voice-clips":