docs(deploy): runbook with smoke script and emergency channel
P5-prep Task 2(docs/plans/2026-09-21-P5-准备件.md): - infra/deploy/RUNBOOK.md:发布前检查单(备份/水位快照(outbox_events deliveredAt、sync_state appliedVersion)/镜像 tag/env diff/磁盘);发布步骤 (迁移先行 → ov 后 cn 逐容器替换 → 四点冒烟 → 观察);回滚步骤(镜像回旧 tag、迁移不回滚的向前兼容原则、outbox/inbox/水位账本硬性保留、冻结 sync 的故障处置);紧急停用通道(ov emergency HMAC 调用样例,openssl dgst 可 复制片段,签名公式与 internal-sync.ts 实现交叉验证一致);故障速查三类 (同步积压/计数差异/CDN 缓存与隐藏照片失效窗口)。 - scripts/prod-smoke.sh:四点冒烟(两地域 health → 登录 → 时间戳相册 + 内嵌 800x600 JPEG 真实 presign PUT → complete → cn/ov 公开读轮询 ≤60s), PASS/FAIL 摘要 + 软删照片清理;--dry-run 打印调用序列、--skip-ov/--ov 跳过 ov 段、--wait-seconds 可调;仅依赖 bash+curl+openssl+base64。 验证:bash -n 通过;--dry-run / --skip-ov / 用法错误三路径本地实测输出正确 (含密码 JSON 转义);真实栈未起(无凭证,符合准备件边界),首次真实验证 点在 README-1Panel.md §7 演练步骤。 Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,186 @@
|
|||||||
|
# RUNBOOK —— 发布 / 迁移 / 回滚 / 紧急通道运行手册(P5-prep Task 2)
|
||||||
|
|
||||||
|
> 适用对象:photo-live 双地域生产栈(编排见 `docker-compose.cn.yml` /
|
||||||
|
> `docker-compose-ov.yml`,环境说明见 `README-1Panel.md`)。
|
||||||
|
> 原则:**一切发布向前兼容**(迁移只前滚不回滚;回滚=旧镜像跑新 schema);
|
||||||
|
> **同步账本(outbox / sync_inbox / sync_state 水位 / outbox_stat_events)任何
|
||||||
|
> 情况下不清理不重置**——幂等与补漏对账都建立在水位连续之上。
|
||||||
|
|
||||||
|
## 0. 角色与容器速查
|
||||||
|
|
||||||
|
| 地域 | 容器 | 角色 | 端口(容器内) |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| cn | api-cn | 写路径 API + cn 公开读 + user-sync | 8080(/v1/health) |
|
||||||
|
| cn | worker | sharp 变体流水线 + 发布事务 + outbox | 无 HTTP |
|
||||||
|
| cn | sync | outbox→ov 推送 + 对账 + 统计回传(双库直连) | 无 HTTP |
|
||||||
|
| cn | web | nginx:viewer + admin + /v1 反代 | 宿主机 127.0.0.1:8088 |
|
||||||
|
| ov | api-ov | ov 公开读 + 互动 + WS + inbox + emergency | 8081(/v1/health) |
|
||||||
|
| ov | web-ov | nginx 网关 | 宿主机 127.0.0.1:8089 |
|
||||||
|
|
||||||
|
## 1. 发布前检查单(每次发布,逐项打勾)
|
||||||
|
|
||||||
|
- [ ] **变更范围确认**:本次代码是否含 Prisma 迁移(`packages/db-*/prisma/migrations`
|
||||||
|
新增目录)?含则必须先走 §2 迁移步骤,且确认迁移向前兼容(新增列可空/带默认、
|
||||||
|
不删旧列不收窄类型——旧镜像需能跑在新 schema 上)。
|
||||||
|
- [ ] **数据库备份**(两地域各自):
|
||||||
|
`docker exec <mysql容器> mysqldump -uroot -p<密码> --single-transaction photo_cn | gzip > photo_cn_$(date +%Y%m%d%H%M).sql.gz`
|
||||||
|
(ov 用 photo_ov;备份文件留存 ≥ 2 份历史)。
|
||||||
|
- [ ] **水位与账本快照登记**(回滚后比对用,只记不删):
|
||||||
|
cn:`SELECT count(*) FROM outbox_events WHERE deliveredAt IS NULL;`(未投递账本)
|
||||||
|
ov:`SELECT entityType, entityId, appliedVersion FROM sync_state;`(行内水位)
|
||||||
|
+ sync 容器最近 20 行日志截图/粘贴。
|
||||||
|
- [ ] **镜像 tag 登记**:新 tag = `git短sha-日期`(server 与 web 同 tag 配套),登记到
|
||||||
|
发布记录;两台服务器均已导入该 tag(`docker images | grep photo-live`)。
|
||||||
|
- [ ] **env 变更 diff**:`.env.production-*` 与仓库 example 对比,确认无缺键、
|
||||||
|
`SYNC_INTERNAL_KEY` 两端一致、无真实密钥入 git。
|
||||||
|
- [ ] **磁盘与负载水位**:1Panel 概览确认磁盘 < 80%、MySQL/Redis 内存正常;
|
||||||
|
worker 队列无积压(api-cn 日志无 process job 超时)。
|
||||||
|
- [ ] **冷备窗口**:选择业务低峰;ov 端确认无正在进行的展示位活动(嵌入轮播依赖
|
||||||
|
ov 公开读可用性)。
|
||||||
|
|
||||||
|
## 2. 发布步骤(cn 与 ov 各自执行;顺序:迁移 → 服务 → 冒烟)
|
||||||
|
|
||||||
|
1. **迁移先行**(若本次含迁移;无迁移跳过):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# cn 服务器
|
||||||
|
docker run --rm --env-file /opt/photo-live/.env.production-cn --network 1panel-network \
|
||||||
|
photo-live-server:<新tag> migrate-deploy cn
|
||||||
|
# ov 服务器
|
||||||
|
docker run --rm --env-file /opt/photo-live/.env.production-ov --network 1panel-network \
|
||||||
|
photo-live-server:<新tag> migrate-deploy ov
|
||||||
|
```
|
||||||
|
|
||||||
|
Prisma `migrate deploy` 只应用未执行过的迁移,可安全重跑。
|
||||||
|
2. **切镜像起服务**(编排目录内;`SERVER_IMAGE`/`WEB_IMAGE` 指新 tag):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# ov 先(cn sync 依赖 ov 可达)
|
||||||
|
SERVER_IMAGE=photo-live-server:<新tag> WEB_IMAGE=photo-live-web:<新tag> \
|
||||||
|
docker compose -f docker-compose-ov.yml up -d
|
||||||
|
docker compose -f docker-compose-ov.yml ps # 等 api-ov healthy
|
||||||
|
# cn 后,逐容器替换(单实例"滚动":起稳一个再下一个)
|
||||||
|
SERVER_IMAGE=photo-live-server:<新tag> WEB_IMAGE=photo-live-web:<新tag> \
|
||||||
|
docker compose -f docker-compose.cn.yml up -d api-cn
|
||||||
|
docker compose -f docker-compose.cn.yml ps # api-cn healthy 后
|
||||||
|
SERVER_IMAGE=photo-live-server:<新tag> docker compose -f docker-compose.cn.yml up -d worker sync web
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **四点冒烟**(发布完成的判定口径):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/prod-smoke.sh https://photo.example.cn https://photo.example.com <账号> <密码>
|
||||||
|
```
|
||||||
|
|
||||||
|
覆盖:①两地域 /v1/health ②登录 ③建册→presign 上传→complete→公开可见
|
||||||
|
(worker 处理 + cn 公开读,轮询 ≤ 60s)④ov 公开读确认。脚本输出 PASS/FAIL
|
||||||
|
摘要并软删清理测试照片。任一 FAIL → 停在原地排查,必要时按 §3 回滚。
|
||||||
|
4. **观察 10 分钟**:sync 日志有 `outbox batch pushed`、无连续 `push failed`;
|
||||||
|
api 无 5xx 洪峰;1Panel 容器监控正常。之后发布记录归档(tag、时间、检查单)。
|
||||||
|
|
||||||
|
## 3. 回滚步骤(镜像回旧 tag;数据与同步状态保留)
|
||||||
|
|
||||||
|
1. **决策口径**:新版本故障且冒烟/观察发现 → 回滚镜像。**迁移不回滚**(发布前
|
||||||
|
检查单已保证旧代码兼容新 schema;反向迁移会破坏账本与水位)。
|
||||||
|
2. 执行(与发布 §2.2 相同的命令,tag 换回上一个登记过的稳定 tag,顺序 ov→cn、
|
||||||
|
api→worker/sync→web):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SERVER_IMAGE=photo-live-server:<旧tag> WEB_IMAGE=photo-live-web:<旧tag> \
|
||||||
|
docker compose -f docker-compose-ov.yml up -d
|
||||||
|
SERVER_IMAGE=photo-live-server:<旧tag> WEB_IMAGE=photo-live-web:<旧tag> \
|
||||||
|
docker compose -f docker-compose.cn.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **数据与同步状态保留原则**(硬性):
|
||||||
|
- 不清 `sync_outbox` / `sync_inbox` / `sync_state` / `outbox_stat_events`;
|
||||||
|
未确认事件在新一轮推送中按 eventId 幂等重放,水位断层才需要人工补漏
|
||||||
|
(对账循环 5min 自动按水位拉 reconcile,无需干预)。
|
||||||
|
- 不 `migrate resolve`/手工改 `_prisma_migrations`;异常迁移记录单独记录并
|
||||||
|
上报,由开发者出修正迁移。
|
||||||
|
- 回滚后必须重跑 §2.3 冒烟 + 比对 §1 水位快照(outbox 计数只允许持平或下降
|
||||||
|
到「已确认」值,不允许突降为 0)。
|
||||||
|
4. 若故障来自**数据本身**(坏行导致 sync/apI 循环报错):先 `docker compose stop
|
||||||
|
sync`(冻结推送,cn 本地功能不受影响),保留现场日志与 outbox 行,联系开发
|
||||||
|
处理;恢复后 `up -d sync` 从冻结水位继续。
|
||||||
|
|
||||||
|
## 4. 紧急停用通道(ov 侧 emergency,独立于同步链路)
|
||||||
|
|
||||||
|
用途:跨境同步中断/内容事故时,**直接调用 ov 的 emergency 端点**下架单册内容;
|
||||||
|
语义:仅允许更严屏蔽(disable),不修改 cn 权威数据,不因普通同步事件自动解除,
|
||||||
|
恢复需显式 enable(技术方案 §2A;鉴权=与 sync 同一把 `SYNC_INTERNAL_KEY` 的
|
||||||
|
HMAC-SHA256 原始请求体签名 + ±300s 时间戳)。
|
||||||
|
|
||||||
|
**可复制样例**(bash + openssl + curl;OV_BASE=ov 域名,KEY=SYNC_INTERNAL_KEY):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
OV_BASE="https://photo.example.com"
|
||||||
|
KEY="你的SYNC_INTERNAL_KEY"
|
||||||
|
ALBUM_TOKEN="目标相册publicToken" # 与 albumId 恰好提供其一
|
||||||
|
REASON="2026-09-21 事故下架:演示"
|
||||||
|
|
||||||
|
BODY=$(printf '{"albumToken":"%s","reason":"%s"}' "$ALBUM_TOKEN" "$REASON")
|
||||||
|
TS=$(date +%s)
|
||||||
|
# 签名 = hex(HMAC-SHA256(原始请求体字节, KEY));curl -d 发送的必须是同一字符串
|
||||||
|
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$KEY" | awk '{print $NF}')
|
||||||
|
|
||||||
|
curl -sS -X POST "$OV_BASE/v1/internal/emergency/disable" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "x-sync-timestamp: $TS" \
|
||||||
|
-H "x-sync-signature: $SIG" \
|
||||||
|
-d "$BODY"
|
||||||
|
# 恢复:body 换 {"albumToken":"..."}(无 reason),端点换 /enable,签名同样重算
|
||||||
|
```
|
||||||
|
|
||||||
|
注意:
|
||||||
|
- 401 = 签名/时间戳问题(两端正则:`X-Sync-Signature = hex(HMAC-SHA256(rawBody, KEY))`,
|
||||||
|
`X-Sync-Timestamp` Unix 秒且与 ov 服务器时差 < 300s——先 `date +%s` 对时)。
|
||||||
|
- 若 ov 整域不可达(DNS/断网),本通道同样不可达——这是它设计上依赖 ov API 存活
|
||||||
|
的边界;整域故障直接在 1Panel 停 web-ov/api-ov 容器(服务面下线,数据面无恙)。
|
||||||
|
- 调用成功后在 cn 管理端补记事件(audit 无自动跨域记录,凭 reason 字符串回溯)。
|
||||||
|
|
||||||
|
## 5. 故障速查(三类典型)
|
||||||
|
|
||||||
|
### 5.1 同步积压(cn 相册更新后 ov 长时间不出现)
|
||||||
|
|
||||||
|
1. `docker logs --tail 100 photo-live-cn-sync-1`:连续 `outbox push failed` →
|
||||||
|
看 err:`401/403` = SYNC_INTERNAL_KEY 两端不一致或时钟漂移;`timeout/ECONNREFUSED`
|
||||||
|
= cn 服务器到 OV_API_URL(公网域名)不通;`P1001` = 到 ov MySQL 3306 不通
|
||||||
|
(防火墙/白名单/SSL)。
|
||||||
|
2. 修复连通后无需人工干预:失败退避封顶 60s 自动重推;对账循环每 5min 按水位
|
||||||
|
补漏;满批(100/批)连续追推排空积压。
|
||||||
|
3. 单事件卡死(如 ov 库坏行):`docker compose stop sync` 冻结 → 定位
|
||||||
|
`sync_inbox` 对应 eventId 行与错误日志 → 上报开发 → 恢复 `up -d sync`。
|
||||||
|
|
||||||
|
### 5.2 计数差异(ov 显示的点赞/浏览与 cn 统计对不上)
|
||||||
|
|
||||||
|
1. 链路:ov `like_batches`/`visit_sessions` 聚合 → `outbox_stat_events` →
|
||||||
|
sync 统计回传循环(60s)→ cn `stat-snapshots` 版本替换。差异先看回传:
|
||||||
|
sync 日志 `stat snapshots returned` 是否持续推进。
|
||||||
|
2. `docker logs --tail 50 photo-live-cn-sync-1` 有 `stat return cycle failed` →
|
||||||
|
多为 cn 库连接闪断,幂等重试自愈;持续失败查 `outbox_stat_events` 未确认行
|
||||||
|
(ov 库)与 cn 库可达性。
|
||||||
|
3. 版本不前进:cn `stat-snapshots` 的 statVersion 单调,若 ov 侧聚合停滞
|
||||||
|
(互动未入账)→ 查 api-ov 日志 likes/visits 429 与 Redis ov 连接。
|
||||||
|
4. 窗口语义提醒:互动流水超窗口清理(like_batches 24h/visit 30min)只清流水,
|
||||||
|
累计总数永不清——对账以 `stat-snapshots` 权威,勿手工 UPDATE 副本计数。
|
||||||
|
|
||||||
|
### 5.3 CDN 缓存(隐藏照片仍可见 / 旧图不刷新)
|
||||||
|
|
||||||
|
1. 图片 URL 两类缓存语义:**variants 内容寻址键**(/variants/…按 photoId+规格,
|
||||||
|
长缓存 immutable)与**签名 URL 短缓存**(OSS 现签 GET,TTL=SIGNED_URL_TTL_
|
||||||
|
SECONDS=900s)。CDN 接入清单(P5-prep Task 3 `cdn-checklist.md`)落地前,
|
||||||
|
本节按「无 CDN」基线排查。
|
||||||
|
2. 隐藏/删除照片后旧链接仍出图:先确认语义——公开 API 立即 404(PHOTO_NOT_
|
||||||
|
PUBLIC),但**已签发的旧 URL 在 TTL 内仍可取到对象**(设计允许的失效窗口,
|
||||||
|
上限 900s;P4a 契约决策)。超过 900s 仍可取 → 查 OSS 桶是否误配公共读。
|
||||||
|
3. 缩略图/浏览图更新不刷新:variants 键含内容版本——同一 photoId 重新处理
|
||||||
|
会生成新键;旧键对象留存(禁止覆盖语义),前端拿到新签名 URL 即新键。若
|
||||||
|
页面仍显示旧图 → 浏览器本地缓存,强刷;CDN 接入后另见 cdn-checklist 的
|
||||||
|
失效 API 样例与隐藏照片失效窗口验证步骤(待凭证执行)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*本手册全部命令以 1Panel 目录约定 `/opt/photo-live/` 与网络 `1panel-network` 为例;
|
||||||
|
实际路径/网络名不同处按 README-1Panel.md §4 的说明替换。*
|
||||||
@@ -0,0 +1,186 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# scripts/prod-smoke.sh —— 生产四点冒烟(P5-prep Task 2;RUNBOOK.md §2.3 的判定口径)。
|
||||||
|
#
|
||||||
|
# 用法:
|
||||||
|
# scripts/prod-smoke.sh <BASE_CN> <BASE_OV> <ACCOUNT> <PASSWORD> [选项]
|
||||||
|
# BASE_CN/BASE_OV 两地域 API 基址(https://photo.example.cn 形态;末尾不带 /v1)
|
||||||
|
# --skip-ov ov 未部署时跳过 ov 段(--ov 为其别名,同义)
|
||||||
|
# --dry-run 只打印将执行的调用序列,不发任何请求(本地校验脚本逻辑用)
|
||||||
|
# --wait-seconds N 公开可见轮询上限秒(默认 60;worker 处理 + cn→ov 推送 2s
|
||||||
|
# 间隔通常几秒内完成,60s 覆盖慢盘/排队余量)
|
||||||
|
#
|
||||||
|
# 覆盖四点:①两地域 /v1/health ②登录(真实密码)③建时间戳相册 → presign 上传
|
||||||
|
# 真实 JPEG 字节(内置 800x600 测试图,刻意不用 1x1——避免变体流水线踩最小尺寸
|
||||||
|
# 边界)→ complete → ④公开读轮询至可见(cn)+ ov 侧同步后公开读确认。
|
||||||
|
# 结束打印 PASS/FAIL 摘要并清理(软删测试照片:公开端点即刻不可见;相册行保留
|
||||||
|
# 时间戳名供审计——相册无删除端点,照片软删后公开列表即空)。
|
||||||
|
# 依赖:bash + curl + openssl + base64(无 jq/node)。退出码:0=全过,1=有 FAIL,2=用法错。
|
||||||
|
set -u
|
||||||
|
|
||||||
|
# ── 参数 ──
|
||||||
|
BASE_CN="${1:-}"; BASE_OV="${2:-}"; ACCOUNT="${3:-}"; PASSWORD="${4:-}"
|
||||||
|
[ "$#" -ge 4 ] && shift 4
|
||||||
|
SKIP_OV=0; DRY_RUN=0; WAIT_SECONDS=60; PREV_ARG=""
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--skip-ov|--ov) SKIP_OV=1 ;;
|
||||||
|
--dry-run) DRY_RUN=1 ;;
|
||||||
|
--wait-seconds) : ;;
|
||||||
|
*) [ "$PREV_ARG" = "--wait-seconds" ] && WAIT_SECONDS="$arg" ;;
|
||||||
|
esac
|
||||||
|
PREV_ARG="$arg"
|
||||||
|
done
|
||||||
|
|
||||||
|
# 内置测试图:800x600 JPEG(quality 55,3118 字节)——base64 内嵌,跨机器无依赖。
|
||||||
|
JPEG_B64='/9j/2wBDAA4KCw0LCQ4NDA0QDw4RFiQXFhQUFiwgIRokNC43NjMuMjI6QVNGOj1OPjIySGJJTlZYXV5dOEVmbWVabFNbXVn/2wBDAQ8QEBYTFioXFypZOzI7WVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVn/wAARCAJYAyADASIAAhEBAxEB/8QAFQABAQAAAAAAAAAAAAAAAAAAAAX/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/8QAFQEBAQAAAAAAAAAAAAAAAAAAAAX/xAAUEQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIRAxEAPwCsAkKIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD//2Q=='
|
||||||
|
|
||||||
|
PASS=0; FAIL=0; FAILED_STEPS=""
|
||||||
|
TOKEN=""; ALBUM_ID=""; PUBLIC_TOKEN=""; PHOTO_ID=""; UPLOAD_ID=""; CRED_URL=""
|
||||||
|
HTTP_CODE="000"; BODY=""
|
||||||
|
JPG_FILE="$(mktemp "${TMPDIR:-/tmp}/photo-smoke-XXXXXX.jpg")"
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
step_pass() { PASS=$((PASS + 1)); echo " [PASS] $1"; }
|
||||||
|
step_fail() { FAIL=$((FAIL + 1)); FAILED_STEPS="${FAILED_STEPS}${1}; "; echo " [FAIL] $1"; }
|
||||||
|
|
||||||
|
# json_escape <s>:处理反斜杠/双引号(密码与标题的常见特殊字符;控制字符不入参)。
|
||||||
|
json_escape() { printf '%s' "$1" | sed -e 's/\\/\\\\/g' -e 's/"/\\"/g'; }
|
||||||
|
|
||||||
|
# extract_str <json> <key>:取扁平响应里 "key":"value" 的 value(本脚本响应均此形态)。
|
||||||
|
extract_str() { printf '%s' "$1" | sed -n "s/.*\"$2\":\"\([^\"]*\)\".*/\1/p" | head -n 1; }
|
||||||
|
|
||||||
|
# http <method> <url> <expected-codes(逗号分隔)> [json-body]
|
||||||
|
# 成功时置 HTTP_CODE/BODY 并返回 0。dry-run 下只打印调用并返回 0。
|
||||||
|
http() {
|
||||||
|
local method="$1" url="$2" expect="$3" body="${4:-}"
|
||||||
|
if [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
echo " curl -sS -X $method '$url'${body:+ -d '$body'}${TOKEN:+ -H 'Authorization: Bearer <token>'}"
|
||||||
|
HTTP_CODE="dry"; BODY=""
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local tmp; tmp="$(mktemp "${TMPDIR:-/tmp}/photo-smoke-body-XXXXXX")"
|
||||||
|
local -a args=(-sS -X "$method" "$url" -H 'Content-Type: application/json' -o "$tmp" -w '%{http_code}')
|
||||||
|
[ -n "$TOKEN" ] && args+=(-H "Authorization: Bearer $TOKEN")
|
||||||
|
if [ -n "$body" ]; then
|
||||||
|
HTTP_CODE="$(curl "${args[@]}" -d "$body" 2>/dev/null || echo 000)"
|
||||||
|
else
|
||||||
|
HTTP_CODE="$(curl "${args[@]}" 2>/dev/null || echo 000)"
|
||||||
|
fi
|
||||||
|
BODY="$(cat "$tmp" 2>/dev/null || true)"
|
||||||
|
rm -f "$tmp"
|
||||||
|
case ",$expect," in
|
||||||
|
*",$HTTP_CODE,"*) return 0 ;;
|
||||||
|
*) return 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
if [ "$DRY_RUN" -eq 0 ] && [ -n "$PHOTO_ID" ] && [ -n "$TOKEN" ] && [ "$TOKEN" != "dry" ]; then
|
||||||
|
echo "-- 清理:软删测试照片 $PHOTO_ID(公开列表即刻为空)"
|
||||||
|
if http DELETE "$BASE_CN/v1/photos/$PHOTO_ID" 204; then step_pass "cleanup 软删照片"; else step_fail "cleanup 软删照片 (HTTP=$HTTP_CODE)"; fi
|
||||||
|
fi
|
||||||
|
rm -f "$JPG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "== photo-live 生产冒烟 =="
|
||||||
|
echo " cn=$BASE_CN ov=$BASE_OV account=$ACCOUNT skip_ov=$SKIP_OV dry_run=$DRY_RUN"
|
||||||
|
if [ -z "$BASE_CN" ] || [ -z "$ACCOUNT" ] || [ -z "$PASSWORD" ] || { [ "$SKIP_OV" -eq 0 ] && [ -z "$BASE_OV" ]; }; then
|
||||||
|
echo "用法: $0 <BASE_CN> <BASE_OV> <ACCOUNT> <PASSWORD> [--skip-ov|--ov] [--dry-run] [--wait-seconds N]" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
printf '%s' "$JPEG_B64" | base64 -d >"$JPG_FILE" 2>/dev/null || { echo "base64 解码失败(需 GNU base64)" >&2; exit 2; }
|
||||||
|
JPEG_BYTES=$(wc -c <"$JPG_FILE" | tr -d ' ')
|
||||||
|
TS=$(date +%Y%m%d%H%M%S)
|
||||||
|
|
||||||
|
echo "-- ① 健康检查(两地域)"
|
||||||
|
if http GET "$BASE_CN/v1/health" 200; then step_pass "cn /v1/health"; else step_fail "cn /v1/health (HTTP=$HTTP_CODE)"; fi
|
||||||
|
if [ "$SKIP_OV" -eq 0 ]; then
|
||||||
|
if http GET "$BASE_OV/v1/health" 200; then step_pass "ov /v1/health"; else step_fail "ov /v1/health (HTTP=$HTTP_CODE)"; fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "-- ② 登录(真实密码,会话走 Bearer)"
|
||||||
|
if [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
http POST "$BASE_CN/v1/auth/password" 200 \
|
||||||
|
"{\"account\":\"$(json_escape "$ACCOUNT")\",\"password\":\"$(json_escape "$PASSWORD")\"}"
|
||||||
|
TOKEN="<dry-run-token>"
|
||||||
|
else
|
||||||
|
if http POST "$BASE_CN/v1/auth/password" 200 \
|
||||||
|
"{\"account\":\"$(json_escape "$ACCOUNT")\",\"password\":\"$(json_escape "$PASSWORD")\"}"; then
|
||||||
|
TOKEN=$(extract_str "$BODY" sessionToken)
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [ -n "$TOKEN" ]; then
|
||||||
|
step_pass "登录(sessionToken 获取)"
|
||||||
|
else
|
||||||
|
step_fail "登录 (HTTP=$HTTP_CODE BODY=$BODY)"
|
||||||
|
echo "== 摘要:PASS=$PASS FAIL=$FAIL 失败项: $FAILED_STEPS =="; echo "RESULT: FAIL"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "-- ③ 建时间戳相册 → presign 上传 → complete"
|
||||||
|
if http POST "$BASE_CN/v1/albums" 201 \
|
||||||
|
"{\"translations\":[{\"locale\":\"zh-CN\",\"title\":\"smoke-$TS\",\"description\":\"prod-smoke 自动冒烟\"}],\"defaultLocale\":\"zh-CN\",\"startAt\":\"$(date -u +%Y-%m-%dT%H:%M:%S.000Z)\"}"; then
|
||||||
|
ALBUM_ID=$(extract_str "$BODY" id); PUBLIC_TOKEN=$(extract_str "$BODY" publicToken)
|
||||||
|
fi
|
||||||
|
if [ -n "$ALBUM_ID" ] && [ -n "$PUBLIC_TOKEN" ]; then
|
||||||
|
step_pass "建相册(publicToken=$PUBLIC_TOKEN)"
|
||||||
|
elif [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
ALBUM_ID="<albumId>"; PUBLIC_TOKEN="<publicToken>"; step_pass "建相册(dry-run 占位)"
|
||||||
|
else
|
||||||
|
step_fail "建相册 (HTTP=$HTTP_CODE BODY=$BODY)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if http POST "$BASE_CN/v1/uploads" 201 \
|
||||||
|
"{\"albumId\":\"$ALBUM_ID\",\"clientTaskId\":\"smoke-$TS\",\"filename\":\"smoke.jpg\",\"bytes\":$JPEG_BYTES,\"contentType\":\"image/jpeg\"}"; then
|
||||||
|
UPLOAD_ID=$(extract_str "$BODY" uploadId); PHOTO_ID=$(extract_str "$BODY" photoId); CRED_URL=$(extract_str "$BODY" url)
|
||||||
|
fi
|
||||||
|
if [ -n "$UPLOAD_ID" ] && [ -n "$CRED_URL" ]; then
|
||||||
|
step_pass "presign 凭证(uploadId=$UPLOAD_ID)"
|
||||||
|
elif [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
UPLOAD_ID="<uploadId>"; PHOTO_ID="<photoId>"; CRED_URL="<presigned-url>"; step_pass "presign 凭证(dry-run 占位)"
|
||||||
|
else
|
||||||
|
step_fail "presign (HTTP=$HTTP_CODE BODY=$BODY)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
echo " curl -sS -X PUT '$CRED_URL' -H 'Content-Type: image/jpeg' --data-binary @$JPG_FILE ($JPEG_BYTES 字节)"
|
||||||
|
echo " (complete → cn/ov 公开读轮询逻辑与真实流程一致,见脚本 --dry-run 外执行)"
|
||||||
|
else
|
||||||
|
PUT_CODE="$(curl -sS -X PUT "$CRED_URL" -H 'Content-Type: image/jpeg' --data-binary @"$JPG_FILE" -o /dev/null -w '%{http_code}' 2>/dev/null || echo 000)"
|
||||||
|
case "$PUT_CODE" in
|
||||||
|
2*) step_pass "OSS presign PUT($JPEG_BYTES 字节真实上传)" ;;
|
||||||
|
*) step_fail "OSS PUT (HTTP=$PUT_CODE)" ;;
|
||||||
|
esac
|
||||||
|
if http POST "$BASE_CN/v1/uploads/$UPLOAD_ID/complete" 200; then step_pass "complete(进入 worker 处理)"; else step_fail "complete (HTTP=$HTTP_CODE BODY=$BODY)"; fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "-- ④ 公开读轮询至可见(≤${WAIT_SECONDS}s;worker 变体流水线 + 发布事务为异步)"
|
||||||
|
if [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
echo " 轮询 GET $BASE_CN/v1/public/albums/$PUBLIC_TOKEN/photos(间隔 2s)直到响应含 \"id\":\"$PHOTO_ID\""
|
||||||
|
else
|
||||||
|
SEEN_CN=0; WAITED=0
|
||||||
|
while [ "$WAITED" -le "$WAIT_SECONDS" ]; do
|
||||||
|
http GET "$BASE_CN/v1/public/albums/$PUBLIC_TOKEN/photos" 200 || true
|
||||||
|
if [ -n "$PHOTO_ID" ] && printf '%s' "$BODY" | grep -q "\"id\":\"$PHOTO_ID\""; then SEEN_CN=1; break; fi
|
||||||
|
sleep 2; WAITED=$((WAITED + 2))
|
||||||
|
done
|
||||||
|
if [ "$SEEN_CN" -eq 1 ]; then step_pass "cn 公开可见(${WAITED}s)"; else step_fail "cn 公开可见超时 ${WAIT_SECONDS}s(查 worker 日志)"; fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$SKIP_OV" -eq 0 ]; then
|
||||||
|
echo "-- ④' ov 侧公开读确认(cn→ov 同步:推送 2s 间隔 + 对账 5min 兜底)"
|
||||||
|
if [ "$DRY_RUN" -eq 1 ]; then
|
||||||
|
echo " 轮询 GET $BASE_OV/v1/public/albums/$PUBLIC_TOKEN/photos(间隔 2s)直到响应含 \"id\":\"$PHOTO_ID\""
|
||||||
|
else
|
||||||
|
SEEN_OV=0; WAITED=0
|
||||||
|
while [ "$WAITED" -le "$WAIT_SECONDS" ]; do
|
||||||
|
http GET "$BASE_OV/v1/public/albums/$PUBLIC_TOKEN/photos" 200 || true
|
||||||
|
if [ -n "$PHOTO_ID" ] && printf '%s' "$BODY" | grep -q "\"id\":\"$PHOTO_ID\""; then SEEN_OV=1; break; fi
|
||||||
|
sleep 2; WAITED=$((WAITED + 2))
|
||||||
|
done
|
||||||
|
if [ "$SEEN_OV" -eq 1 ]; then step_pass "ov 公开可见(同步链路端到端,${WAITED}s)"; else step_fail "ov 公开可见超时 ${WAIT_SECONDS}s(查 sync 日志 / RUNBOOK §5.1)"; fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "== 摘要:PASS=$PASS FAIL=$FAIL ${FAILED_STEPS:+失败项: $FAILED_STEPS}=="
|
||||||
|
if [ "$FAIL" -eq 0 ]; then echo "RESULT: PASS"; else echo "RESULT: FAIL"; exit 1; fi
|
||||||
Reference in New Issue
Block a user