Files
alibaba/docs/superpowers/plans/2026-07-14-plugin-push.md
T

396 lines
18 KiB
Markdown
Raw Normal View History

2026-08-27 16:12:37 +08:00
# 插件推送改造 实施计划 (Plan 2)
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 在现有 `d:\alibb\` Chrome 插件(已实现抓取 + API 直拉聊天)上**新增推送模块**:抓完一个询盘自动入队,由 service worker 经 `chrome.alarms` 消费、POST 到 Plan 1 后端 `/api/ingest`。带 IndexedDB 队列(防丢)+ 指数退避重试 + 配置页 + 运行时域名授权。**保留**现有本地导出能力(双写)。
**Architecture:**
- `background.js`(SW)改 **ES module**(`manifest` `type:module`),`import` 推送逻辑 `push/*.js`
- 推送逻辑分三个 ES module(可被 node + vitest 测试):`push/queue.js`(IndexedDB 队列)、`push/payload.js`(组装后端 payload)、`push/worker.js`(消费 + POST + 重试)
- `content.js` 保持经典脚本,抓完一条 → `chrome.runtime.sendMessage({type:'enqueue', record, messages})` 发原始数据给 SW;SW 组装 payload + 入队
- IndexedDB 由 **SW 落盘**(content script 的 IDB 属页面源,v3 修订),`chrome.storage.local` 只存 `queue_index` + 元数据
- `popup` 加「后端设置」(URL/token/启用)+ 推送状态显示;`chrome.permissions.request` 运行时授权后端域名
**Tech Stack:** 现有 JS 插件 + vitest 2 + fake-indexeddb(node 测队列)+ Chrome MV3 API(alarms/indexedDB/permissions/storage)。
**关联:** spec `docs/superpowers/specs/2026-07-13-alibaba-inquiry-backend-design.md` §8;后端 Plan 1 已就绪(`d:\alibb-server\backend`,`/api/ingest`)。
**前提:** `d:\alibb` 当前**非 git 仓库**,Task 1 会 `git init`。Plan 1 后端能 `docker compose up` 跑起来供端到端验证。
---
## 文件结构
```
d:\alibb\ # 插件根(本 plan 改这里)
manifest.json # 改:type:module + permissions(alarms/unlimitedStorage) + optional_host_permissions
background.js # 改:ES module,import push/*,alarms worker,接收 enqueue;保留 storeDetail/getBgData(本地导出)
content.js # 改:apiMsgToChat 扩展(messageId/sendTime ISO/translation);fetchChatByApi 扩展(meta);processCurrentDetailPage 加 enqueue
popup.html / popup.js # 改:加后端设置区 + 推送状态显示
push\
queue.js # 新:IndexedDB 队列操作(addItem/getPending/markStatus/remove/getStats)
payload.js # 新:buildIngestPayload(record, messages) → 适配后端 zod
worker.js # 新:processQueue(消费 + POST + 指数退避)
tests\
queue.test.js # vitest + fake-indexeddb
payload.test.js # 纯函数测试
worker.test.js # mock fetch + fake-indexeddb
package.json # 新:vitest + fake-indexeddb
vitest.config.ts
```
---
## Task 1: node 测试基础 + git init
**Files:** `package.json`, `vitest.config.ts`, `.gitignore`
- [ ] **Step 1:** `cd d:/alibb && git init`(若已是 git 仓库则跳过)
- [ ] **Step 2:** `package.json`:`{ "name":"alibaba-inquiry-plugin", "type":"module", "scripts":{"test":"vitest run"} }`
- [ ] **Step 3:** `npm install -D vitest fake-indexeddb`
- [ ] **Step 4:** `vitest.config.ts`:`{ test:{ globals:true, testTimeout:30000, poolOptions:{ forks:{ singleFork:true } }, setupFiles:['fake-indexeddb/auto'] } }`
- [ ] **Step 5:** `.gitignore`:`node_modules/ .env`(注意:**不忽略** push/*.js、content.js 等插件文件)
- [ ] **Step 6:** commit `chore: add node test harness (vitest + fake-indexeddb)`
---
## Task 2: manifest 改造
**Files:** `manifest.json`
- [ ] **Step 1:** permissions 加 `alarms`、`unlimitedStorage` → `["storage","activeTab","alarms","unlimitedStorage"]`
- [ ] **Step 2:** 加 `optional_host_permissions: ["http://*/*","https://*/*"]`(后端域名运行时授权)
- [ ] **Step 3:** background 加 `"type":"module"` → `{ "service_worker":"background.js", "type":"module" }`
- [ ] **Step 4:** commit `feat(manifest): alarms/unlimitedStorage + optional_host_permissions + sw module`
---
## Task 3: push/queue.js(IndexedDB 队列)
**Files:** `push/queue.js`, `tests/queue.test.js`
- [ ] **Step 1:** 写 `tests/queue.test.ts`(TDD):addItem → getPending 返回含它;markStatus('sent') → 不在 pending;getStats → {pending,failed,sent} 计数
- [ ] **Step 2:** 跑测试确认失败
- [ ] **Step 3:** 写 `push/queue.js`:
```js
const DB_NAME = 'inquiry_push', STORE = 'queue', VERSION = 1;
function openDb() {
return new Promise((resolve, reject) => {
const req = indexedDB.open(DB_NAME, VERSION);
req.onupgradeneeded = () => req.result.createObjectStore(STORE, { keyPath: 'id' });
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
}
function tx(db, mode) { return db.transaction(STORE, mode).objectStore(STORE); }
function req2promise(r) { return new Promise((res, rej) => { r.onsuccess = () => res(r.result); r.onerror = () => rej(r.error); }); }
export async function addItem(item) {
const db = await openDb();
const rec = { id: crypto.randomUUID(), payload: item.payload, inquiryId: item.inquiryId,
status: 'pending', retries: 0, nextRetry: Date.now(), createdAt: Date.now() };
await req2promise(tx(db, 'readwrite').add(rec));
db.close();
return rec.id;
}
export async function getPending(limit = 5) {
const db = await openDb();
const all = await req2promise(tx(db, 'readonly').getAll());
db.close();
const now = Date.now();
return all.filter(r => r.status === 'pending' && r.nextRetry <= now)
.sort((a, b) => a.nextRetry - b.nextRetry)
.slice(0, limit);
}
export async function markStatus(id, status, opts = {}) {
const db = await openDb();
const store = tx(db, 'readwrite');
const r = await req2promise(store.get(id));
if (!r) { db.close(); return; }
Object.assign(r, { status, ...opts });
await req2promise(store.put(r));
db.close();
}
export async function remove(id) {
const db = await openDb();
await req2promise(tx(db, 'readwrite').delete(id));
db.close();
}
export async function getStats() {
const db = await openDb();
const all = await req2promise(tx(db, 'readonly').getAll());
db.close();
return all.reduce((a, r) => { a[r.status] = (a[r.status] || 0) + 1; return a; }, {});
}
```
> `crypto.randomUUID()` 在 SW 和 node 18+ 都可用。fake-indexeddb 提供 indexedDB 全局。
- [ ] **Step 4:** 测试通过 + commit `feat(push): IndexedDB queue (add/getPending/markStatus/remove/stats)`
---
## Task 4: push/payload.js(组装后端 payload)
**Files:** `push/payload.js`, `tests/payload.test.js`
- [ ] **Step 1:** 测试:`buildIngestPayload(record, messages)` 输出匹配后端 zod(inquiryId/createTime ISO/updateTime ISO/customer{customerId,country,...}/messages[{messageId,sendTime ISO,direction,content,...}]);空字符串 behavior 归 null;messages 为空时抛错(后端 zod min(1))
- [ ] **Step 2:** 写 `push/payload.js`:
```js
const s = v => (v == null || v === '') ? null : v;
const iso = ms => new Date(ms).toISOString();
export function buildIngestPayload(record, messages) {
if (!messages || messages.length === 0) throw new Error('messages required');
const iq = record || {};
return {
inquiry: {
inquiryId: String(iq.inquiryId),
manager: s(iq.manager) || undefined,
// 优先用 meta 的 gmtCreate/gmtModified(record.createTimeFromApi),回退列表 createTime
createTime: iq.createTimeFromApi || iq.createTime || iso(Date.now()),
updateTime: iq.updateTimeFromApi || iq.updateTime || iq.createTimeFromApi || iso(Date.now()),
status: s(iq.status) || undefined,
msgPreview: s(iq.msgPreview) || undefined,
isFlagged: !!iq.isFlagged,
},
customer: iq.customerId ? {
customerId: String(iq.customerId),
name: s(iq.detailBuyerName) || s(iq.buyerName),
country: s(iq.detailCountry) || s(iq.countryCode) ? (s(iq.detailCountry) || s(iq.countryCode)) : null,
email: s(iq.email), company: s(iq.company),
registerTime: s(iq.registerTime) || null,
avatarUrl: s(iq.detailAvatarUrl) || s(iq.avatarUrl) || undefined,
buyerTags: s(iq.buyerTags) || undefined, buyerFeatures: s(iq.buyerFeatures) || undefined,
behavior: iq.behavior ? {
productViews: n(iq.behavior.productViews), validInquiries: n(iq.behavior.validInquiries),
validRfq: n(iq.behavior.validRfq), loginDays: n(iq.behavior.loginDays),
spamInquiries: n(iq.behavior.spamInquiries), topIndustry: s(iq.behavior.topIndustry),
} : undefined,
} : null,
messages: messages.map(m => ({
messageId: String(m.messageId), sendTime: m.sendTime,
sender: s(m.sender) || undefined, direction: m.direction,
content: m.original ?? m.content ?? '', translation: s(m.translation) || '',
})),
};
}
function n(v) { const x = Number(v); return Number.isFinite(x) ? x : null; }
```
> 注:`messages` 已是 apiMsgToChat 转换后的对象(messageId/sendTime ISO/sender/original/direction/translation)。content 映射用 original(apiMsgToChat 产出 original)。
- [ ] **Step 3:** 测试通过(对照 Plan 1 后端 PayloadSchema 语义)+ commit `feat(push): buildIngestPayload adapter`
---
## Task 5: push/worker.js(消费 + POST + 重试)
**Files:** `push/worker.js`, `tests/worker.test.js`
- [ ] **Step 1:** 测试:mock global.fetch(成功 → markStatus sent 风格的 remove);失败 → retries++ nextRetry 退避;超过 8 次 → markStatus failed。注入 `{addItem,getPending,markStatus,remove}` mock 或用真实 fake-indexeddb + fetch mock
- [ ] **Step 2:** 写 `push/worker.js`:
```js
import { getPending, markStatus, remove } from './queue.js';
import { buildIngestPayload } from './payload.js';
const MAX_RETRIES = 8;
let _config = null; // {serverUrl, token, enabled}
export function configure(cfg) { _config = cfg; }
export async function processQueue() {
if (!_config || !_config.enabled || !_config.serverUrl || !_config.token) return { processed: 0 };
const items = await getPending(3);
let processed = 0;
for (const it of items) {
try {
const payload = buildIngestPayload(it.payload.record, it.payload.messages);
const res = await fetch(`${_config.serverUrl}/api/ingest`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${_config.token}`, 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
});
if (res.ok) { await remove(it.id); processed++; }
else { await bumpRetry(it, `HTTP ${res.status}`); }
} catch (e) {
await bumpRetry(it, String(e && e.message || e));
}
}
return { processed };
}
async function bumpRetry(it, err) {
const retries = it.retries + 1;
if (retries >= MAX_RETRIES) {
await markStatus(it.id, 'failed', { retries, lastError: err });
} else {
const delay = Math.min(Math.pow(2, retries) * 1000, 3600 * 1000);
await markStatus(it.id, 'pending', { retries, nextRetry: Date.now() + delay, lastError: err });
}
}
```
- [ ] **Step 3:** 测试通过 + commit `feat(push): worker (consume + POST + backoff)`
---
## Task 6: background.js(SW,alarms worker + 接收 enqueue)
**Files:** `background.js`
- [ ] **Step 1:** 顶部改 ES module import:
```js
import { addItem, getStats } from './push/queue.js';
import { processQueue, configure } from './push/worker.js';
```
- [ ] **Step 2:** 保留现有 storeDetail/getBgData/clearBgData(本地导出)**不动**
- [ ] **Step 3:** 加配置加载 + alarms 注册 + enqueue 处理:
```js
const CONFIG_KEY = 'push_config';
const ALARM = 'push-tick';
async function loadConfig() {
const d = await chrome.storage.local.get(CONFIG_KEY);
return d[CONFIG_KEY] || { enabled: false, serverUrl: '', token: '' };
}
// 启动 + 配置变更时刷新 worker 配置 + alarm
async function refresh() {
const cfg = await loadConfig();
configure(cfg);
if (cfg.enabled) {
await chrome.alarms.create(ALARM, { periodInMinutes: 1 });
processQueue(); // 立即跑一次
} else {
await chrome.alarms.clear(ALARM);
}
}
chrome.runtime.onInstalled.addListener(refresh);
chrome.runtime.onStartup.addListener(refresh);
chrome.alarms.onAlarm.addListener(a => { if (a.name === ALARM) processQueue(); });
chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
if (msg.type === 'enqueue') {
(async () => {
const cfg = await loadConfig();
if (!cfg.enabled) { sendResponse({ ok: false, reason: 'disabled' }); return; }
const id = await addItem({ payload: { record: msg.record, messages: msg.messages }, inquiryId: msg.record?.inquiryId });
sendResponse({ ok: true, id });
processQueue(); // 立即尝试
})();
return true;
}
if (msg.type === 'getPushStats') {
(async () => sendResponse(await getStats()))();
return true;
}
if (msg.type === 'setPushConfig') {
(async () => {
await chrome.storage.local.set({ [CONFIG_KEY]: msg.config });
await refresh();
sendResponse({ ok: true });
})();
return true;
}
});
```
- [ ] **Step 4:** commit `feat(bg): push worker + alarms + enqueue handler (keep local export)`
---
## Task 7: content.js — apiMsgToChat 扩展
**Files:** `content.js`(apiMsgToChat ~434)
- [ ] **Step 1:** apiMsgToChat 改产出 `{ messageId, sendTime(ISO), sender, original, translation, direction }`:
- `messageId: String(m.messageId)`
- `sendTime: new Date(m.sendTime).toISOString()`(替代原来的本地时区 "YYYY-MM-DD HH:MM:SS")
- `translation`:从 `m.extInfo?.icbuData?.extend` 解析(extend 是 JSON 字符串,内含 extParams.inputTranslateInfo.translate);解析失败留 `''`
- sender/original/direction 保持
- [ ] **Step 2:** translation 解析加双重 JSON.parse 兜底(extend 可能是字符串里的字符串),失败不抛(留 '')
- [ ] **Step 3:** 手动/Playwright 验证(无 node 测试,因 content.js 非 module):用 Playwright 注入新版 apiMsgToChat + 真实 m 样本,断言 messageId/sendTime/translation 正确
- [ ] **Step 4:** commit `feat(content): apiMsgToChat emits messageId/sendTime(ISO)/translation`
---
## Task 8: content.js — fetchChatByApi 扩展(提取 gmtCreate/country)
**Files:** `content.js`(fetchChatByApi ~451)
- [ ] **Step 1:** fetchChatByApi 在调 querySummary 后,额外提取 `d.gmtCreate`/`d.gmtModified`(毫秒→ISO)存入返回 meta;`d.contact?.countryCode` 或 `d.cid` 派生也行(spec 用 contact.countryCode)。返回 `{ messages, meta: { gmtCreate, gmtModified } }`
- [ ] **Step 2:** 调用方(processCurrentDetailPage / scrapeDetailData)把 meta 挂到 detail:`detail.createTimeFromApi = meta.gmtCreate; detail.updateTimeFromApi = meta.gmtModified`
- [ ] **Step 3:** commit `feat(content): fetchChatByApi extracts gmtCreate/gmtModified for payload`
---
## Task 9: content.js — processCurrentDetailPage 加 enqueue
**Files:** `content.js`(processCurrentDetailPage ~347,storeDetail 之后)
- [ ] **Step 1:** 在 storeDetail(本地导出)**之后**加 enqueue(推送):
```js
// 推送到后端(Plan 2):发 record + messages 给 SW 入队
try {
chrome.runtime.sendMessage({
type: 'enqueue',
record: merged, // {item, detail} 合并
messages: detail.chatMessages || [], // apiMsgToChat 产出数组
}, () => {}); // fire-and-forget,失败不影响本地导出
} catch (e) {}
```
> `detail.chatMessages` 需在 scrapeDetailData 产出(现有 chatText 是序列化串,加一个 `chatMessages` 数组字段)。确保 merged 含 `createTimeFromApi/updateTimeFromApi`(Task 8)。
- [ ] **Step 2:** commit `feat(content): enqueue record+messages to SW after detail scrape`
---
## Task 10: popup 配置页 + permissions.request
**Files:** `popup.html`, `popup.js`
- [ ] **Step 1:** popup.html 加「后端设置」区:服务器 URL 输入、Token 输入、启用推送 checkbox、「授权域名」按钮、「保存」按钮
- [ ] **Step 2:** popup.js:加载时 `chrome.runtime.sendMessage({type:'getPushConfig'})` 回填(或直接 `chrome.storage.local.get`);保存时 setPushConfig
- [ ] **Step 3:** 「授权域名」按钮 → `chrome.permissions.request({origins:[serverUrl + '/*']})`(用户手势触发);成功后提示
- [ ] **Step 4:** commit `feat(popup): backend config UI + runtime origin permission`
---
## Task 11: popup 推送状态显示
**Files:** `popup.html`, `popup.js`
- [ ] **Step 1:** popup 加状态条:「已推 N / 待推 M / 失败 K」
- [ ] **Step 2:** popup 打开时 + 每 2s `chrome.runtime.sendMessage({type:'getPushStats'})` 刷新
- [ ] **Step 3:** 失败 K > 0 时红字 + 可点击查看 lastError(简化:弹 alert 或展开列表)
- [ ] **Step 4:** commit `feat(popup): push stats display (pending/failed/sent)`
---
## Task 12: 端到端验证
**Files:** `scripts/e2e-plugin.md`(验证步骤文档)
- [ ] **Step 1:** 起 Plan 1 后端:`cd d:/alibb-server/backend && docker compose up -d --build`,等 health ok
- [ ] **Step 2:** Chrome `chrome://extensions` 开发者模式 → 加载 `d:\alibb`(本 plan 改后的插件)
- [ ] **Step 3:** 点插件图标 → 后端设置填 `http://localhost:8080` + token `dev-ingest-token-1234` + 启用 → 授权域名 → 保存
- [ ] **Step 4:** 打开 `message.alibaba.com` → 抓一个询盘(用插件现有抓取流程,或 Playwright 注入触发)
- [ ] **Step 5:** 验证:`curl -H "Authorization: Bearer <jwt>" http://localhost:8080/api/inquiries` 能看到刚抓的询盘;`docker compose exec mysql mysql -uroot -proot inquiry -e "SELECT inquiry_id, messages_count FROM inquiries"` 数据在
- [ ] **Step 6:** popup 状态条显示「已推 1」
- [ ] **Step 7:** 验证幂等:再抓同一询盘 → 后端 messages_count 不变(早返回)
- [ ] **Step 8:** commit `test: e2e plugin→backend validation doc`
---
## 完成标准(DoD)
- [ ] manifest:`type:module` + alarms/unlimitedStorage + optional_host_permissions
- [ ] `push/queue.js` / `payload.js` / `worker.js` 三个 module + vitest 测试全绿
- [ ] `background.js`:alarms(1 分钟)消费队列,POST `/api/ingest`,失败指数退避(≤8 次);保留本地 storeDetail/getBgData
- [ ] `content.js`:apiMsgToChat 产 messageId/sendTime(ISO)/translation;fetchChatByApi 提取 gmtCreate/gmtModified;processCurrentDetailPage enqueue
- [ ] popup:后端设置(URL/token/启用)+ permissions.request 授权 + 状态显示(已推/待推/失败)
- [ ] 端到端:抓一条询盘 → Plan 1 后端能看到数据;再抓同一条幂等(messages_count 不变)
- [ ] 断网/后端停 → 数据进队列不丢;后端恢复 → 自动推完
## 后续(本 plan 不含)
- Plan 3:管理后台前端(查看/筛选/导出/统计)
- Plan 4:正式部署(域名 + HTTPS + 备份定时 + 监控)
- 增量推送(插件记录 last sendTime,只推新增消息)
- 失败项重试 UI(popup 查看失败 + 手动重推)