docs: M1 计划四维审查修复(30+ 处)

- 后端: py-ip2region 正确包名与API/会话缓存键/并发F表达式/重放聚合排除/parser与Django5时区500/类型与体积深度白名单/限流测试夹具/字段截断
- SDK: localStorage内存降级/异步路径try-catch/批次context补首触字段/page_stay同源path/4xx不重试/测试技巧
- 安全: 生产fail-fast/请求体双层上限/deny统一防枚举/拒绝回ACAO/文件缓存共享限流/前端禁v-html/JWT取舍明示
- 交付: .env.production/xdb volume挂载/compose全量与MySQL三路径/.dockerignore/大陆镜像源/隐私告知前置/跨天验证指引

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
浅念
2026-09-15 11:02:50 +08:00
parent 63f711d915
commit 8af480e4a4
@@ -38,7 +38,7 @@ cms-cj/
│ │ ├── management/commands/refresh_stats.py │ │ ├── management/commands/refresh_stats.py
│ │ └── tests/ # pytest 测试 │ │ └── tests/ # pytest 测试
│ ├── Dockerfile │ ├── Dockerfile
│ └── data/ip2region.xdb # 离线 IP 库(下载,不进 git) │ └── tracker/data/ip2region.xdb # 离线 IP 库(下载,不进 git,部署时 volume 挂载)
├── sdk/ # JS SDK ├── sdk/ # JS SDK
│ ├── package.json vitest.config.ts build.mjs │ ├── package.json vitest.config.ts build.mjs
│ ├── src/{index,config,identity,context,transport,events,util}.ts │ ├── src/{index,config,identity,context,transport,events,util}.ts
@@ -57,6 +57,7 @@ cms-cj/
- **响应包**:所有 `/api/admin/*` 返回 `{code: 200, data: ..., message: "ok"}`(适配 Art Design Pro 拦截器默认约定);错误 `code` 为业务码(401 未登录/403 无权/4xx 参数/500 内部)。 - **响应包**:所有 `/api/admin/*` 返回 `{code: 200, data: ..., message: "ok"}`(适配 Art Design Pro 拦截器默认约定);错误 `code` 为业务码(401 未登录/403 无权/4xx 参数/500 内部)。
- **时间**:全库 UTC 存储(`USE_TZ=True`),展示层按 `settings.TIME_ZONE='Asia/Shanghai'`。 - **时间**:全库 UTC 存储(`USE_TZ=True`),展示层按 `settings.TIME_ZONE='Asia/Shanghai'`。
- **采集跨域**:SDK 用 `Content-Type: text/plain` 发 JSON(简单请求免预检),服务端用 `PlainJSONParser` 解析;`django-cors-headers` + 动态白名单(DB 中 Site.domain_whitelist,60s 缓存)回 `Access-Control-Allow-Origin`。 - **采集跨域**:SDK 用 `Content-Type: text/plain` 发 JSON(简单请求免预检),服务端用 `PlainJSONParser` 解析;`django-cors-headers` + 动态白名单(DB 中 Site.domain_whitelist,60s 缓存)回 `Access-Control-Allow-Origin`。
- **前端安全红线**:全项目禁用 `v-html`;访客来源数据(path/title/referrer/utm)不得直接绑定 `href/src`——如需渲染成链接,协议白名单仅 http/https(Vue 不拦截 `javascript:` URI)。
- **测试**:后端 `pytest`(SQLite 内存库,`pytest.ini` 指定 `--ds=config.settings_test`);SDK `vitest` + happy-dom + fake timers。 - **测试**:后端 `pytest`(SQLite 内存库,`pytest.ini` 指定 `--ds=config.settings_test`);SDK `vitest` + happy-dom + fake timers。
- **提交**:每任务一提交,`feat:/test:/chore:` 前缀。 - **提交**:每任务一提交,`feat:/test:/chore:` 前缀。
- **M1 明确不做**(防执行者自作主张加码):SPA 路由、点击/滚动/表单/意图事件、客户(contact)体系、Redis、事件表分区迁移、npm 发布。 - **M1 明确不做**(防执行者自作主张加码):SPA 路由、点击/滚动/表单/意图事件、客户(contact)体系、Redis、事件表分区迁移、npm 发布。
@@ -79,7 +80,7 @@ djangorestframework>=3.15
djangorestframework-simplejwt>=5.3 djangorestframework-simplejwt>=5.3
django-cors-headers>=4.4 django-cors-headers>=4.4
PyMySQL>=1.1 PyMySQL>=1.1
ip2region>=2.0 py-ip2region>=3.0
gunicorn>=22.0 gunicorn>=22.0
``` ```
@@ -109,8 +110,11 @@ import os
BASE_DIR = Path(__file__).resolve().parent.parent BASE_DIR = Path(__file__).resolve().parent.parent
SECRET_KEY = os.environ.get("DJANGO_SECRET_KEY", "dev-insecure") SECRET_KEY = os.environ.get("DJANGO_SECRET_KEY", "dev-insecure")
DEBUG = os.environ.get("DJANGO_DEBUG", "1") == "1" DEBUG = os.environ.get("DJANGO_DEBUG", "0" if os.environ.get("DJANGO_ENV") == "prod" else "1") == "1"
ALLOWED_HOSTS = os.environ.get("DJANGO_ALLOWED_HOSTS", "*").split(",") ALLOWED_HOSTS = os.environ.get("DJANGO_ALLOWED_HOSTS", "*").split(",")
if os.environ.get("DJANGO_ENV") == "prod": # 生产 fail-fast:漏配即启动崩溃,不带病上线
assert SECRET_KEY != "dev-insecure", "生产必须配置 DJANGO_SECRET_KEY"
assert os.environ.get("DJANGO_ALLOWED_HOSTS"), "生产必须显式配置 DJANGO_ALLOWED_HOSTS"
INSTALLED_APPS = [ INSTALLED_APPS = [
"django.contrib.contenttypes", "django.contrib.contenttypes",
@@ -154,6 +158,9 @@ REST_FRAMEWORK = {
} }
SIMPLE_JWT = {"ACCESS_TOKEN_LIFETIME": timedelta(hours=12)} SIMPLE_JWT = {"ACCESS_TOKEN_LIFETIME": timedelta(hours=12)}
CORS_ALLOW_ALL_ORIGINS = False CORS_ALLOW_ALL_ORIGINS = False
CACHES = {"default": {"BACKEND": "django.core.cache.backends.filebased.FileBasedCache",
"LOCATION": str(BASE_DIR / ".cache")}} # 限流计数跨 gunicorn worker 共享(不共享则限额×3)
DATA_UPLOAD_MAX_MEMORY_SIZE = 262144 # 256KB 应用层请求体兜底
TIME_ZONE = "Asia/Shanghai" TIME_ZONE = "Asia/Shanghai"
USE_TZ = True USE_TZ = True
LANGUAGE_CODE = "zh-hans" LANGUAGE_CODE = "zh-hans"
@@ -186,6 +193,13 @@ SENSITIVE_KEY_PATTERNS = ("password", "passwd", "token", "secret", "card", "veri
URL_PARAM_DROP_PATTERNS = ("email", "phone", "mobile", "token", "key", "password", "verif") URL_PARAM_DROP_PATTERNS = ("email", "phone", "mobile", "token", "key", "password", "verif")
SESSION_IDLE_MS = 30 * 60 * 1000 SESSION_IDLE_MS = 30 * 60 * 1000
BOT_UA_KEYWORDS = ("bot", "spider", "crawl", "slurp", "bingpreview", "headless", "lighthouse") BOT_UA_KEYWORDS = ("bot", "spider", "crawl", "slurp", "bingpreview", "headless", "lighthouse")
EVENT_PROP_WHITELIST = { # spec §9.1:未知字段拒绝。免费文本/任意属性不得入库
"page_view": {"referrer", "referrer_domain", "utm"},
"page_stay": {"active_ms", "path"},
"sdk_loaded": {"sdk_version"},
}
MAX_PROP_DEPTH = 2
MAX_PROP_JSON_BYTES = 4096
``` ```
- [ ] **Step 3: 建 Django 工程与 app 骨架** - [ ] **Step 3: 建 Django 工程与 app 骨架**
@@ -466,6 +480,11 @@ def test_sensitive_property_rejected():
r = validate_event(_ev(properties={"user_password": "1"})) r = validate_event(_ev(properties={"user_password": "1"}))
assert "sensitive" in r assert "sensitive" in r
def test_shape_type_and_whitelist():
assert validate_event(["x"]) == "event_invalid_shape"
assert validate_event(_ev(event_id=123)) == "invalid_event_id"
assert validate_event(_ev(properties={"foo": "bar"})) == "property_not_allowed"
def test_too_many_props_and_long_string(): def test_too_many_props_and_long_string():
r = validate_event(_ev(properties={f"k{i}": "v" for i in range(40)})) r = validate_event(_ev(properties={f"k{i}": "v" for i in range(40)}))
assert "properties" in r assert "properties" in r
@@ -478,8 +497,9 @@ def test_future_and_stale_time():
assert validate_event(_ev(occurred_at=NOW - timedelta(hours=2))) is None # 窗内迟到可收 assert validate_event(_ev(occurred_at=NOW - timedelta(hours=2))) is None # 窗内迟到可收
def test_sanitize_url_drops_sensitive_params(): def test_sanitize_url_drops_sensitive_params():
u = sanitize_url("/p?email=a@b.com&token=x&id=3&utm_source=google") u = sanitize_url("/p?email=a@b.com&token=x&id=3&utm_source=google&monkey=1")
assert "email=" not in u and "token=" not in u and "utm_source=google" in u assert "email=" not in u and "token=" not in u and "utm_source=google" in u
assert "monkey=1" in u # fullmatch:含 key 子串的正常参数不误伤
``` ```
- [ ] **Step 2: 跑测试确认失败** → ModuleNotFoundError。 - [ ] **Step 2: 跑测试确认失败** → ModuleNotFoundError。
@@ -487,23 +507,44 @@ def test_sanitize_url_drops_sensitive_params():
- [ ] **Step 3: 实现** - [ ] **Step 3: 实现**
```python ```python
import json
import re import re
from datetime import timedelta from datetime import datetime, timedelta
from urllib.parse import parse_qsl, urlencode, urlparse
from django.utils import timezone from django.utils import timezone
from tracker import constants as C from tracker import constants as C
_SENSITIVE = re.compile("|".join(C.SENSITIVE_KEY_PATTERNS), re.I) _SENSITIVE = re.compile("|".join(C.SENSITIVE_KEY_PATTERNS), re.I)
_URL_DROP = re.compile("&?(%s)=[^&]*" % "|".join(C.URL_PARAM_DROP_PATTERNS), re.I) _URL_DROP_KEY = re.compile("|".join(C.URL_PARAM_DROP_PATTERNS), re.I)
def sanitize_url(url: str) -> str: def sanitize_url(url: str) -> str:
return _URL_DROP.sub("", url).replace("?&", "?").rstrip("?") """按参数名整键删除(fullmatch 防止误伤 monkey 这类含 key 子串的参数)。"""
p = urlparse(url)
kept = [(k, v) for k, v in parse_qsl(p.query, keep_blank_values=True)
if not _URL_DROP_KEY.fullmatch(k)]
return p._replace(query=urlencode(kept)).geturl()
def validate_event(ev: dict) -> str | None: def _depth_ok(v, depth=0) -> bool:
"""返回 None 表示通过;否则返回拒绝原因码。""" if depth > C.MAX_PROP_DEPTH:
return False
if isinstance(v, dict):
return all(_depth_ok(x, depth + 1) for x in v.values())
if isinstance(v, list):
return all(_depth_ok(x, depth + 1) for x in v)
return True
def validate_event(ev) -> str | None:
"""返回 None 表示通过;否则返回拒绝原因码。类型/白名单/体积/深度全在此守门。"""
if not isinstance(ev, dict):
return "event_invalid_shape"
if ev.get("event_name") not in C.EVENT_NAMES: if ev.get("event_name") not in C.EVENT_NAMES:
return "event_name_not_allowed" return "event_name_not_allowed"
if not ev.get("event_id") or len(ev["event_id"]) > 64: eid = ev.get("event_id")
if not isinstance(eid, str) or not eid or len(eid) > 64:
return "invalid_event_id" return "invalid_event_id"
for k in ("path", "title"):
if ev.get(k) is not None and not isinstance(ev[k], str):
return "invalid_field_type"
occ = ev.get("occurred_at") occ = ev.get("occurred_at")
if not isinstance(occ, datetime): if not isinstance(occ, datetime):
return "invalid_occurred_at" return "invalid_occurred_at"
@@ -513,6 +554,12 @@ def validate_event(ev: dict) -> str | None:
props = ev.get("properties") or {} props = ev.get("properties") or {}
if not isinstance(props, dict) or len(props) > C.MAX_PROP_KEYS: if not isinstance(props, dict) or len(props) > C.MAX_PROP_KEYS:
return "properties_invalid" return "properties_invalid"
if not set(props) <= C.EVENT_PROP_WHITELIST.get(ev["event_name"], set()):
return "property_not_allowed"
if not _depth_ok(props):
return "properties_depth"
if len(json.dumps(props, ensure_ascii=False, default=str)) > C.MAX_PROP_JSON_BYTES:
return "properties_too_large"
for k, v in props.items(): for k, v in props.items():
if _SENSITIVE.search(str(k)): if _SENSITIVE.search(str(k)):
return "sensitive_field" return "sensitive_field"
@@ -559,13 +606,17 @@ import functools, ipaddress
from django.conf import settings from django.conf import settings
class GeoResolver: class GeoResolver:
"""进程级单例:xdb 一次读入内存(官方 buffer 模式声明线程安全)。
pip 包名 py-ip2region(模块名 ip2region);包缺失或 xdb 缺失时优雅降级返回 None。"""
def __init__(self): def __init__(self):
self._searcher = None self._searcher = None
try: try:
from ip2region import XdbSearcher # pip ip2region;失败则降级 from ip2region import searcher as ip2region_searcher
self._searcher_cls = XdbSearcher with open(settings.IP2REGION_XDB, "rb") as f:
self._searcher = ip2region_searcher.new_with_buffer(f.read())
except Exception: except Exception:
self._searcher_cls = None self._searcher = None
@functools.lru_cache(maxsize=10000) @functools.lru_cache(maxsize=10000)
def resolve(self, ip: str): def resolve(self, ip: str):
@@ -576,17 +627,16 @@ class GeoResolver:
return None return None
except ValueError: except ValueError:
return None return None
if not self._searcher_cls: if not self._searcher:
return None return None
try: try:
with self._searcher_cls(contentBuff=open(settings.IP2REGION_XDB, "rb").read()) as s: raw = self._searcher.search(ip) # "国家|区域|省|市|ISP"
raw = s.search(ip) # "国家|区域|省|市|ISP"
parts = (raw or "").split("|") parts = (raw or "").split("|")
return {"country": parts[0], "province": parts[2], "city": parts[3]} return {"country": parts[0], "province": parts[2], "city": parts[3]}
except Exception: except Exception:
return None return None
``` ```
注:若 `ip2region` pip 包 API 不同,以官方 README 为准适配 `XdbSearcher` 的调用方式;解析结果格式 `国家|区域|省|市|ISP`。中文区返回如 `中国|0|广东省|深圳市|电信`,海外如 `美国|0|加利福尼亚|旧金山|0`。 注:py-ip2region 的实际调用以其 README 为准(模块 `ip2region`,入口 `searcher.new_with_buffer(buff)` 后 `search(ip)`;**不支持 with 上下文管理器**——用错会被 except 静默吞掉、地理功能永久失效且测试照样绿,务必按此实现)。解析结果格式 `国家|区域|省|市|ISP`,如 `中国|0|广东省|深圳市|电信`、`美国|0|加利福尼亚|旧金山|0`。
- [ ] **Step 3: 跑测试通过**(`server/.gitignore` 加 `tracker/data/*.xdb`) - [ ] **Step 3: 跑测试通过**(`server/.gitignore` 加 `tracker/data/*.xdb`)
@@ -662,6 +712,7 @@ def test_rejected_events_reported_with_reason():
```python ```python
from datetime import timedelta from datetime import timedelta
from django.db import transaction from django.db import transaction
from django.db.models import F
from django.utils import timezone from django.utils import timezone
from tracker import constants as C from tracker import constants as C
from tracker.models import Visitor, VisitSession, Event from tracker.models import Visitor, VisitSession, Event
@@ -670,27 +721,34 @@ from tracker.services.geo import GeoResolver
_geo = GeoResolver() _geo = GeoResolver()
_FIELD_LEN = {"device_type": 16, "os": 64, "browser": 64, "language": 32, "screen": 32}
def _clean_utm(d):
"""utm 键白名单(utm_* 前缀)+ 截断——任意字符串不得经 utm 入库渲染。"""
return {k[:50]: str(v)[:100] for k, v in (d or {}).items()
if isinstance(k, str) and k.startswith("utm_")}
def _touch_visitor(site, vid, ctx, ip): def _touch_visitor(site, vid, ctx, ip):
env = {k: str(ctx.get(k, "") or "")[:n] for k, n in _FIELD_LEN.items()}
v, created = Visitor.objects.get_or_create( v, created = Visitor.objects.get_or_create(
site=site, visitor_id=vid, site=site, visitor_id=vid, defaults={**env, "last_ip": ip})
defaults={"last_ip": ip, **{k: str(ctx.get(k, "") or "")[:64]
for k in ("device_type", "os", "browser", "language", "screen")}},
)
if created:
v.first_referrer_domain = (ctx.get("referrer_domain") or "")[:200]
v.first_utm = {k: str(val)[:100] for k, val in (ctx.get("utm") or {}).items()}
v.first_landing_path = (ctx.get("path") or "")[:500]
now = timezone.now() now = timezone.now()
v.last_seen_at = now upd = {"last_seen_at": now} # 聚合字段一律 filter().update(),避免并发下全量 save() 用旧值覆盖赢家
if created:
upd.update(first_referrer_domain=(ctx.get("referrer_domain") or "")[:200],
first_utm=_clean_utm(ctx.get("utm")),
first_landing_path=(ctx.get("path") or "")[:500],
last_active_date=now.date(), active_days=1)
elif not Visitor.objects.filter(pk=v.pk, last_active_date=now.date()).exists():
upd.update(last_active_date=now.date(), active_days=F("active_days") + 1)
if ip: if ip:
v.last_ip = ip upd["last_ip"] = ip
g = _geo.resolve(ip) g = _geo.resolve(ip)
if g: if g:
v.country, v.province, v.city = g["country"], g["province"], g["city"] upd.update(country=g["country"][:64], province=g["province"][:64],
if v.last_active_date != now.date(): city=g["city"][:64])
v.active_days += 1 Visitor.objects.filter(pk=v.pk).update(**upd)
v.last_active_date = now.date() v.refresh_from_db()
v.save()
return v return v
def _get_session(site, visitor, sid, ctx, occurred_at): def _get_session(site, visitor, sid, ctx, occurred_at):
@@ -703,14 +761,19 @@ def _get_session(site, visitor, sid, ctx, occurred_at):
sid = f"{sid}#{suffix}" sid = f"{sid}#{suffix}"
sess = None sess = None
if not sess: if not sess:
try:
sess = VisitSession.objects.create( sess = VisitSession.objects.create(
site=site, visitor=visitor, session_id=sid, site=site, visitor=visitor, session_id=sid,
started_at=occurred_at, ended_at=occurred_at, started_at=occurred_at, ended_at=occurred_at,
referrer_domain=(ctx.get("referrer_domain") or "")[:200], referrer_domain=(ctx.get("referrer_domain") or "")[:200],
utm={k: str(val)[:100] for k, val in (ctx.get("utm") or {}).items()}, utm=_clean_utm(ctx.get("utm")),
landing_path=(ctx.get("path") or "")[:500]) landing_path=(ctx.get("path") or "")[:500])
Visitor.objects.filter(pk=visitor.pk).update(visit_count=visitor.visit_count + 1) except IntegrityError: # 并发同 sid 同时切分出相同后缀撞唯一键——重查复用
visitor.visit_count += 1 from django.db import IntegrityError as _IE # 顶部统一 import IntegrityError
sess = VisitSession.objects.filter(site=site, session_id=sid).first()
if not sess:
raise
Visitor.objects.filter(pk=visitor.pk).update(visit_count=F("visit_count") + 1)
return sess return sess
@transaction.atomic @transaction.atomic
@@ -739,7 +802,8 @@ def ingest_batch(site, payload: dict, ip: str | None, ua: str = "") -> dict:
sess = session_cache.get(sid) sess = session_cache.get(sid)
if not sess: if not sess:
sess = _get_session(site, visitor, sid, ctx, occ) sess = _get_session(site, visitor, sid, ctx, occ)
session_cache[sess.session_id] = sess session_cache[sid] = sess # 必须按原始 sid 存:切分兜底后 sess.session_id 是 "s1#2",
# 若按它存,同批后续事件按 "s1" 查会 miss → 每条再切一个新会话
ready.append(Event( ready.append(Event(
site=site, visitor=visitor, session=sess, site=site, visitor=visitor, session=sess,
event_id=ev["event_id"], event_name=ev["event_name"], event_id=ev["event_id"], event_name=ev["event_name"],
@@ -776,8 +840,9 @@ def ingest_batch(site, payload: dict, ip: str | None, ua: str = "") -> dict:
```python ```python
from django.db.models import F, Greatest from django.db.models import F, Greatest
for sess in {e.session for e in ready}: new_events = [e for e in ready if e.event_id not in existed] # 重放命中的不参与聚合!
se = [e for e in ready if e.session.pk == sess.pk] for sess in {e.session for e in new_events}:
se = [e for e in new_events if e.session.pk == sess.pk]
new_end = max(x.occurred_at for x in se) new_end = max(x.occurred_at for x in se)
span_ms = int((new_end - min(x.occurred_at for x in se)).total_seconds() * 1000) span_ms = int((new_end - min(x.occurred_at for x in se)).total_seconds() * 1000)
stay_add = sum(int(e.properties.get("active_ms", 0) or 0) stay_add = sum(int(e.properties.get("active_ms", 0) or 0)
@@ -788,6 +853,7 @@ for sess in {e.session for e in ready}:
duration_ms=F("duration_ms") + span_ms, duration_ms=F("duration_ms") + span_ms,
stay_ms=F("stay_ms") + stay_add) # 有效停留单独累计,供访客"总停留"取数 stay_ms=F("stay_ms") + stay_add) # 有效停留单独累计,供访客"总停留"取数
``` ```
- **重放批不得抬高会话聚合**:event_count/stay_ms/duration_ms 只用 `new_events`(否则重试/重放被幂等去重后仍把会话计数刷高——`test_replay_duplicated` 需补断言 `VisitSession.event_count == 1`)。
- 同批多会话各自按上式计算。 - 同批多会话各自按上式计算。
- `duplicated` 计数允许并发近似(唯一约束保证绝不重复入库,仅统计口径可能差 1-2,M1 接受)。 - `duplicated` 计数允许并发近似(唯一约束保证绝不重复入库,仅统计口径可能差 1-2,M1 接受)。
@@ -803,12 +869,17 @@ for sess in {e.session for e in ready}:
- [ ] **Step 1: 失败测试**(要点:白名单域放行/非白名单 403/限流触发 429/textplain 解析/响应最小化/幂等重放) - [ ] **Step 1: 失败测试**(要点:白名单域放行/非白名单 403/限流触发 429/textplain 解析/响应最小化/幂等重放)
```python ```python
import json import json
from datetime import timezone as tz import pytest
from django.core.cache import cache
from django.utils import timezone from django.utils import timezone
from tracker.models import Site from tracker.models import Site
NOW = timezone.now() NOW = timezone.now()
@pytest.fixture(autouse=True)
def _clear_cache():
cache.clear() # locmem 限流历史跨测试残留,会让 429 用例假红
def _post(client, site, body, origin="https://www.a.com", extra=None): def _post(client, site, body, origin="https://www.a.com", extra=None):
return client.post("/api/collect/batch", data=json.dumps(body), return client.post("/api/collect/batch", data=json.dumps(body),
content_type="text/plain", HTTP_ORIGIN=origin, **(extra or {})) content_type="text/plain", HTTP_ORIGIN=origin, **(extra or {}))
@@ -860,8 +931,7 @@ def test_throttle_429(db, client, settings):
`server/tracker/api/collect.py`: `server/tracker/api/collect.py`:
```python ```python
import json import json
from datetime import datetime from datetime import datetime, timezone as dt_tz
from django.utils import timezone
from rest_framework.views import APIView from rest_framework.views import APIView
from rest_framework.response import Response from rest_framework.response import Response
from rest_framework.permissions import AllowAny from rest_framework.permissions import AllowAny
@@ -873,7 +943,11 @@ from tracker.services.ingest import ingest_batch
class PlainJSONParser(BaseParser): class PlainJSONParser(BaseParser):
media_type = "text/plain" media_type = "text/plain"
def parse(self, stream, *args, **kwargs): def parse(self, stream, *args, **kwargs):
try:
return json.loads(stream.read().decode("utf-8") or "{}") return json.loads(stream.read().decode("utf-8") or "{}")
except (ValueError, UnicodeDecodeError):
from rest_framework.exceptions import ParseError
raise ParseError("invalid_json") # DRF 对 parser 异常是 re-raise 不是包装,不自包就是 500
class CollectThrottle(AnonRateThrottle): class CollectThrottle(AnonRateThrottle):
scope = "collect" scope = "collect"
@@ -882,7 +956,7 @@ def _iso(dt):
if isinstance(dt, str): if isinstance(dt, str):
s = dt.replace("Z", "+00:00") s = dt.replace("Z", "+00:00")
d = datetime.fromisoformat(s) d = datetime.fromisoformat(s)
return d if d.tzinfo else d.replace(tzinfo=timezone.utc) return d if d.tzinfo else d.replace(tzinfo=dt_tz.utc) # Django≥5 已删除 django.utils.timezone.utc
return dt return dt
class CollectBatchView(APIView): class CollectBatchView(APIView):
@@ -893,18 +967,20 @@ class CollectBatchView(APIView):
def post(self, request): def post(self, request):
origin = request.headers.get("Origin") or request.headers.get("Referer") or "" origin = request.headers.get("Origin") or request.headers.get("Referer") or ""
if int(request.headers.get("Content-Length") or 0) > 262144:
return self._deny(400, "payload_too_large") # 应用层兜底(nginx 另设 client_max_body_size)
site = self._match_site(request) site = self._match_site(request)
if site is None or not site.is_active: if site is None or not site.is_active:
return self._deny(403, "site_not_allowed") return self._deny(403, "not_allowed") # 三种拒绝统一 reason,防 site_id 枚举探测
host = (origin.split("//")[-1].split("/")[0].split(":")[0].lower() host = (origin.split("//")[-1].split("/")[0].split(":")[0].lower()
if origin else "") if origin else "")
if host and host not in [d.lower() for d in site.domains]: if host and host not in [d.lower() for d in site.domains]:
return self._deny(403, "origin_not_allowed") return self._deny(403, "not_allowed")
if not host and not site.allow_no_origin: if not host and not site.allow_no_origin:
return self._deny(403, "origin_required") return self._deny(403, "not_allowed")
payload = request.data or {} payload = request.data or {}
for ev in payload.get("events") or []: for ev in payload.get("events") or []:
if "occurred_at" in ev: if isinstance(ev, dict) and "occurred_at" in ev:
try: try:
ev["occurred_at"] = _iso(ev["occurred_at"]) ev["occurred_at"] = _iso(ev["occurred_at"])
except ValueError: except ValueError:
@@ -931,7 +1007,11 @@ class CollectBatchView(APIView):
return request.META.get("REMOTE_ADDR") return request.META.get("REMOTE_ADDR")
def _deny(self, code, reason): def _deny(self, code, reason):
return Response({"code": code, "data": None, "message": reason}, status=code) resp = Response({"code": code, "data": None, "message": reason}, status=code)
origin = self.request.headers.get("Origin")
if origin: # 拒绝也要回 ACAO,否则浏览器读不到错误体、SDK 误判为网络错误走重试
resp["Access-Control-Allow-Origin"] = origin
return resp
``` ```
`server/config/urls.py`: `server/config/urls.py`:
```python ```python
@@ -968,16 +1048,19 @@ return resp
- [ ] **Step 1: 失败测试**:造 2 天数据(访客 v1 两天访问、v2 一天)→ `refresh(site, date)` 后 DailyStat 的 pv/uv/sessions/new_visitors 正确;uv 按访客去重、new_visitors 按 first_seen_at 当日计数。 - [ ] **Step 1: 失败测试**:造 2 天数据(访客 v1 两天访问、v2 一天)→ `refresh(site, date)` 后 DailyStat 的 pv/uv/sessions/new_visitors 正确;uv 按访客去重、new_visitors 按 first_seen_at 当日计数。
- [ ] **Step 2: 实现** `stats.py`: - [ ] **Step 2: 实现** `stats.py`:
```python ```python
from datetime import datetime, time, timedelta
from django.utils import timezone
from tracker.models import DailyStat, Event, VisitSession
def refresh_daily(site, date) -> DailyStat: def refresh_daily(site, date) -> DailyStat:
from tracker.models import Event, VisitSession, Visitor, DailyStat
from django.db.models import Count
day_start = timezone.make_aware(datetime.combine(date, time.min)) day_start = timezone.make_aware(datetime.combine(date, time.min))
day_end = day_start + timedelta(days=1) day_end = day_start + timedelta(days=1)
pv = Event.objects.filter(site=site, is_bot=False, pv = Event.objects.filter(site=site, is_bot=False,
occurred_at__gte=day_start, occurred_at__lt=day_end).count() occurred_at__gte=day_start, occurred_at__lt=day_end).count()
uv = Event.objects.filter(site=site, is_bot=False, occurred_at__gte=day_start, uv = Event.objects.filter(site=site, is_bot=False, occurred_at__gte=day_start,
occurred_at__lt=day_end).values("visitor").distinct().count() occurred_at__lt=day_end).values("visitor").distinct().count()
sessions = VisitSession.objects.filter(site=site, started_at__gte=day_start, sessions = VisitSession.objects.filter(site=site, visitor__is_bot=False,
started_at__gte=day_start,
started_at__lt=day_end).count() started_at__lt=day_end).count()
new_v = Visitor.objects.filter(site=site, first_seen_at__gte=day_start, new_v = Visitor.objects.filter(site=site, first_seen_at__gte=day_start,
first_seen_at__lt=day_end).count() first_seen_at__lt=day_end).count()
@@ -995,7 +1078,7 @@ def refresh_daily(site, date) -> DailyStat:
- Create: `server/tracker/api/auth.py`, `server/tracker/tests/test_auth.py`, Modify: `config/urls.py` - Create: `server/tracker/api/auth.py`, `server/tracker/tests/test_auth.py`, Modify: `config/urls.py`
- [ ] **Step 1: 失败测试**:`/api/admin/auth/token` 用户名密码 → access/refresh;错密码 401;`/api/admin/auth/me` 带 token 返回 username/is_admin(is_staff);无 token 401;`/api/admin/users` 仅 is_staff 可 POST 建普通成员。 - [ ] **Step 1: 失败测试**:`/api/admin/auth/token` 用户名密码 → access/refresh;错密码 401;`/api/admin/auth/me` 带 token 返回 username/is_admin(is_staff);无 token 401;`/api/admin/users` 仅 is_staff 可 POST 建普通成员。
- [ ] **Step 2: 实现**:`TokenObtainPairView` 子类(返回 `{code:200,data:{access,refresh},message:"ok"}` 包一层),`me` 返回 `{username, is_admin: user.is_staff}`;`UserCreateView`(IsAdminUser)`username/password` 创建 `is_staff=False` 用户。登录接口挂 `ScopedRateThrottle` scope=`login`(10/min/IP,防爆破;settings 已含 rate)。 - [ ] **Step 2: 实现**:`TokenObtainPairView` 子类(返回 `{code:200,data:{access,refresh},message:"ok"}` 包一层),`me` 返回 `{username, is_admin: user.is_staff}`;`UserCreateView`(IsAdminUser)`username/password` 创建 `is_staff=False` 用户。登录接口挂 `ScopedRateThrottle` scope=`login`(10/min/IP,防爆破;settings 已含 rate)。**JWT 取舍(M1 决策,README-deploy 写明)**:access 12h、refresh 显式配置 24h、无黑名单/轮换——token 泄露的唯一应急是更换 SECRET_KEY(全体登出)。
- [ ] **Step 3: 测试通过,Commit** `feat: JWT 登录与用户接口` - [ ] **Step 3: 测试通过,Commit** `feat: JWT 登录与用户接口`
### Task 10: 站点管理 API ### Task 10: 站点管理 API
@@ -1043,7 +1126,7 @@ GET /api/admin/visitors/<visitor_id>/sessions/<session_id> 单会话页面路
- 列表字段齐全:visitor_id/country/province/city/device_type/os/browser/first_seen_at/last_seen_at/visit_count/active_days/session_count/total_stay_ms/first_source/last_source/online;筛选 country/device_type/online=true/search=访客ID或IP前缀;按 last_seen_at 倒序分页(page_size 20)。 - 列表字段齐全:visitor_id/country/province/city/device_type/os/browser/first_seen_at/last_seen_at/visit_count/active_days/session_count/total_stay_ms/first_source/last_source/online;筛选 country/device_type/online=true/search=访客ID或IP前缀;按 last_seen_at 倒序分页(page_size 20)。
- timeline 结构:`[{date:"2026-09-15", sessions:[{session_id, started_at, events:[{time,event_name,path,title,active_ms}]}]}]`,多天数据按日倒序、日内时间正序;默认近 30 天,`?days=` 可调,单次上限 1000 事件。 - timeline 结构:`[{date:"2026-09-15", sessions:[{session_id, started_at, events:[{time,event_name,path,title,active_ms}]}]}]`,多天数据按日倒序、日内时间正序;默认近 30 天,`?days=` 可调,单次上限 1000 事件。
- 会话详情:事件按 occurred_at 正序,含 active_ms(page_stay 的 properties.active_ms 就地并入相邻 page_view 或单列,见实现说明)。 - 会话详情:事件按 occurred_at 正序,含 active_ms(page_stay 的 properties.active_ms 就地并入相邻 page_view 或单列,见实现说明)。
- [ ] **Step 2: 实现**:ORM 查询 + Python 分组;`total_stay_ms` = `Sum("sessions__stay_ms")` 注解(**有效停留口径,不用 duration_ms 跨度**);`first_source` = `first_referrer_domain or first_utm.utm_source or "直接访问"`;`last_source` 取最近会话同逻辑;`online` = last_seen_at > now-5min;列表默认过滤 `is_bot=False`(`?show_bot=1` 可查爬虫)。**page_view 与 page_stay 合并展示**:时间线返回时若 page_stay 的 path 与前一条 page_view 相同,则把 `active_ms` 合入该 page_view(服务端做,前端省事)。 - [ ] **Step 2: 实现**:ORM 查询 + Python 分组;`total_stay_ms` = `Sum("sessions__stay_ms")` 注解(**有效停留口径,不用 duration_ms 跨度**);`first_source` = `first_referrer_domain or first_utm.utm_source or "直接访问"`;`last_source` 取最近会话同逻辑;`online` = last_seen_at > now-5min;列表默认过滤 `is_bot=False`(`?show_bot=1` 可查爬虫)。**page_view 与 page_stay 合并展示**:时间线返回时若 page_stay 的 path 与前一条 page_view 相同,则把 `active_ms` 合入该 page_view(服务端做,前端省事);同 path 连续多段 page_stay(多次隐藏/恢复)的 active_ms **累加**后合入。会话详情展示"时长"统一用 `stay_ms` 口径(`duration_ms` 是批内跨度累加,仅内部字段不展示)。
- [ ] **Step 3: 测试通过,Commit** `feat: 访客列表/详情/时间线/会话接口` - [ ] **Step 3: 测试通过,Commit** `feat: 访客列表/详情/时间线/会话接口`
--- ---
@@ -1084,16 +1167,27 @@ context 测试:`parseUtm("?utm_source=google&utm_medium=cpc&other=1")` 只留
import { uuid } from "./util"; import { uuid } from "./util";
const K = { vid: "ctk_vid", sid: "ctk_sid", act: "ctk_act" }; const K = { vid: "ctk_vid", sid: "ctk_sid", act: "ctk_act" };
const IDLE = 30 * 60_000; const IDLE = 30 * 60_000;
// localStorage 可能抛异常(隐私模式/禁存储,Safari ITP):统一走带内存降级的 store,
// 任何路径都不得让异常冒泡到宿主页(spec §4.5 红线)
const mem: Record<string, string> = {};
const store = {
get(k: string): string | null {
try { return localStorage.getItem(k); } catch { return mem[k] ?? null; }
},
set(k: string, v: string) {
try { localStorage.setItem(k, v); } catch { mem[k] = v; }
},
};
export function getVisitorId(): string { export function getVisitorId(): string {
let v = localStorage.getItem(K.vid); let v = store.get(K.vid);
if (!v) { v = uuid(); localStorage.setItem(K.vid, v); } if (!v) { v = uuid(); store.set(K.vid, v); }
return v; return v;
} }
export function touchActivity() { localStorage.setItem(K.act, String(Date.now())); } export function touchActivity() { store.set(K.act, String(Date.now())); }
export function getSessionId(): string { export function getSessionId(): string {
const last = +(localStorage.getItem(K.act) || 0); const last = +(store.get(K.act) || 0);
let s = localStorage.getItem(K.sid); let s = store.get(K.sid);
if (!s || Date.now() - last > IDLE) { s = uuid(); localStorage.setItem(K.sid, s); } if (!s || Date.now() - last > IDLE) { s = uuid(); store.set(K.sid, s); }
touchActivity(); touchActivity();
return s; return s;
} }
@@ -1109,9 +1203,10 @@ export function getSessionId(): string {
- [ ] **Step 1: 失败测试**(vi.stubGlobal fetch;fake timers): - [ ] **Step 1: 失败测试**(vi.stubGlobal fetch;fake timers):
- 满 10 条即发;15 秒定时器到即发;payload 含 batch_id/site_id/visitor_id/session_id/events。 - 满 10 条即发;15 秒定时器到即发;payload 含 batch_id/site_id/visitor_id/session_id/events。
- 500 失败→退避重试(500ms×2^n 抖动)最多 3 次→丢弃计数;400 不重试直接丢。 - 500 失败→退避重试(500ms×2^n 抖动)最多 3 次→丢弃计数;400 不重试直接丢。
- 429 → 重试。 - 429 → 重试;403/400 → 不重试直接丢。
- `flushBeacon()` 调 `navigator.sendBeacon`(spy 断言 URL 与 body)。 - `flushBeacon()` 调 `navigator.sendBeacon`(spy 断言 URL 与 body)。**happy-dom 没有 sendBeacon,用 `vi.stubGlobal("navigator", { sendBeacon: spy })`,测完 restore**。
- 队列上限 200,超限丢最旧并计数。 - 队列上限 200(用 `batchSize > queueMax` 的自定义 cfg 测,默认 batchSize=10 到不了 200),超限丢最旧并计数。
- **测试技巧**:异步断言用 `await vi.advanceTimersByTimeAsync()`(flush/send 是 async);fake timers 需含 Date(vitest 默认 toFake 即含);`document.hidden` 是 getter,用 `Object.defineProperty(document, "hidden", { value: true, configurable: true })` + `dispatchEvent(new Event("visibilitychange"))` 模拟;每个用例后 `vi.useRealTimers()`。
- [ ] **Step 2: 实现** - [ ] **Step 2: 实现**
```ts ```ts
import { uuid } from "./util"; import { uuid } from "./util";
@@ -1134,6 +1229,7 @@ export class Transport {
flush(beacon = false) { flush(beacon = false) {
if (this.timer) { clearTimeout(this.timer); this.timer = null; } if (this.timer) { clearTimeout(this.timer); this.timer = null; }
if (!this.q.length) return; if (!this.q.length) return;
try {
const events = this.q; this.q = []; const events = this.q; this.q = [];
const body = JSON.stringify({ batch_id: uuid(), site_id: this.cfg.siteId, const body = JSON.stringify({ batch_id: uuid(), site_id: this.cfg.siteId,
visitor_id: getVisitorId(), session_id: getSessionId(), visitor_id: getVisitorId(), session_id: getSessionId(),
@@ -1143,20 +1239,23 @@ export class Transport {
navigator.sendBeacon(this.cfg.endpoint, new Blob([body], {type: "text/plain"})); navigator.sendBeacon(this.cfg.endpoint, new Blob([body], {type: "text/plain"}));
return; return;
} }
this.send(body); this.send(body, events.length);
} catch { /* 定时器/事件回调内的任何异常不得冒泡到宿主页 */ }
} }
private async send(body: string) { private async send(body: string, count: number) {
try { try {
const res = await fetch(this.cfg.endpoint, { method: "POST", const res = await fetch(this.cfg.endpoint, { method: "POST",
keepalive: true, headers: { "Content-Type": "text/plain" }, body }); keepalive: true, headers: { "Content-Type": "text/plain" }, body });
if (res.status === 400) { this.retry = 0; return; } // 参数错不重试 if (res.status >= 400 && res.status < 500 && res.status !== 429) {
this.retry = 0; return; // 4xx 参数/权限错不重试(403 重试只会掩盖配置错误)
}
if (!res.ok) throw new Error(String(res.status)); if (!res.ok) throw new Error(String(res.status));
this.retry = 0; this.retry = 0;
} catch { } catch {
if (this.retry++ < this.cfg.maxRetry) { if (this.retry++ < this.cfg.maxRetry) {
const delay = 500 * 2 ** this.retry * (0.8 + Math.random() * 0.4); const delay = 500 * 2 ** this.retry * (0.8 + Math.random() * 0.4);
setTimeout(() => this.send(body), delay); setTimeout(() => this.send(body, count).catch(() => {}), delay);
} else { this.retry = 0; this.dropped += 1; } } else { this.retry = 0; this.dropped += count; } // 按事件数计丢弃
} }
} }
} }
@@ -1170,8 +1269,11 @@ export class Transport {
- Create: `sdk/src/events.ts`, `sdk/src/index.ts`, `sdk/test/events.test.ts`, `sdk/demo/index.html`, Modify: `.gitignore`(`sdk/node_modules`, `sdk/dist`) - Create: `sdk/src/events.ts`, `sdk/src/index.ts`, `sdk/test/events.test.ts`, `sdk/demo/index.html`, Modify: `.gitignore`(`sdk/node_modules`, `sdk/dist`)
- [ ] **Step 1: 失败测试**: - [ ] **Step 1: 失败测试**:
- init 后 `sdk_loaded` 事件入队;`page_view` 含 path/title/referrer_domain/utm(utm 仅首页带)。 - init 后 `sdk_loaded` 事件入队;`page_view` 含 path/title/referrer_domain/utm(utm 随每页上报,服务端只写首个,可接受冗余)。
- 批次 context 含设备环境 + referrer_domain/utm/path(首触归因链路,断言 `__ctk_ctx` 字段齐全)。
- 有效停留:fake timers 下 `visible` 且有 pointerdown → 3 秒后 active_ms≈3000;`document.hidden=true` 后不再累计;隐藏时自动 flush beacon(page_stay 已入队)。 - 有效停留:fake timers 下 `visible` 且有 pointerdown → 3 秒后 active_ms≈3000;`document.hidden=true` 后不再累计;隐藏时自动 flush beacon(page_stay 已入队)。
- **localStorage 抛异常时 15 秒定时 flush 不产生未捕获异常**(stub localStorage 全方法抛错 + `await vi.advanceTimersByTimeAsync(15000)`,配 `vi.onUnhandledError` 断言;且身份降级到内存后事件照常发出)。
- 隐藏→恢复→再隐藏产生两段 page_stay,active_ms 各自正确(第二段从 0 重新累计)。
- [ ] **Step 2: 实现** - [ ] **Step 2: 实现**
```ts ```ts
// events.ts // events.ts
@@ -1184,7 +1286,7 @@ export function startPageTracking(t: Transport) {
const act = () => { lastAct = Date.now(); }; const act = () => { lastAct = Date.now(); };
["pointerdown", "keydown", "wheel", "touchstart"].forEach(e => ["pointerdown", "keydown", "wheel", "touchstart"].forEach(e =>
document.addEventListener(e, act, { passive: true, capture: true })); document.addEventListener(e, act, { passive: true, capture: true }));
setInterval(() => { if (!document.hidden && Date.now() - lastAct < 30_000) activeMs += 1000; }, 1000); setInterval(() => { try { if (!document.hidden && Date.now() - lastAct < 30_000) activeMs += 1000; } catch { /* 静默 */ } }, 1000);
const emit = (event_name: string, properties: any = {}) => { const emit = (event_name: string, properties: any = {}) => {
const pc = pageContext(); const pc = pageContext();
t.push({ event_id: uuid(), event_name, occurred_at: new Date().toISOString(), t.push({ event_id: uuid(), event_name, occurred_at: new Date().toISOString(),
@@ -1193,8 +1295,12 @@ export function startPageTracking(t: Transport) {
emit("page_view", { referrer: document.referrer, emit("page_view", { referrer: document.referrer,
referrer_domain: pageContext().referrer_domain, utm: pageContext().utm }); referrer_domain: pageContext().referrer_domain, utm: pageContext().utm });
const close = () => { const close = () => {
if (activeMs > 0) emit("page_stay", { active_ms: activeMs, path: location.pathname }); try {
// path 必须与 page_view 同源同清洗(pageContext().path 含 query),
// 否则带 UTM 进落地页时服务端"page_stay 合并进相邻 page_view"匹配失败
if (activeMs > 0) emit("page_stay", { active_ms: activeMs, path: pageContext().path });
activeMs = 0; t.flush(true); activeMs = 0; t.flush(true);
} catch { /* 静默 */ }
}; };
document.addEventListener("visibilitychange", () => { if (document.hidden) close(); }); document.addEventListener("visibilitychange", () => { if (document.hidden) close(); });
window.addEventListener("pagehide", close); window.addEventListener("pagehide", close);
@@ -1205,7 +1311,7 @@ export function startPageTracking(t: Transport) {
import { mergeConfig } from "./config"; import { mergeConfig } from "./config";
import { Transport } from "./transport"; import { Transport } from "./transport";
import { startPageTracking } from "./events"; import { startPageTracking } from "./events";
import { buildEnvContext } from "./context"; import { buildEnvContext, pageContext } from "./context";
const g = globalThis as any; const g = globalThis as any;
const q: any[][] = g.ctk?.q || []; const q: any[][] = g.ctk?.q || [];
@@ -1214,7 +1320,11 @@ let transport: Transport | null = null;
function dispatch(cmd: string, arg?: any) { function dispatch(cmd: string, arg?: any) {
if (cmd === "init" && !transport) { if (cmd === "init" && !transport) {
const cfg = mergeConfig(arg); const cfg = mergeConfig(arg);
(globalThis as any).__ctk_ctx = buildEnvContext(); // 首触归因修复:批次 context 除设备环境外必须带 referrer_domain/utm/path——
// 服务端 _touch_visitor/_get_session 只从 context 读首触,缺了则首触/会话来源恒空
const pc = pageContext();
(globalThis as any).__ctk_ctx = { ...buildEnvContext(),
referrer_domain: pc.referrer_domain, utm: pc.utm, path: pc.path };
transport = new Transport(cfg); transport = new Transport(cfg);
transport.push({ event_id: uuid(), event_name: "sdk_loaded", transport.push({ event_id: uuid(), event_name: "sdk_loaded",
occurred_at: new Date().toISOString(), occurred_at: new Date().toISOString(),
@@ -1248,7 +1358,7 @@ gzip -c dist/v1.js | wc -c # 必须 ≤ 15360(15KB),超标则精简(
### Task 16: SDK ↔ 后端本地集成验证(手动脚本) ### Task 16: SDK ↔ 后端本地集成验证(手动脚本)
- [ ] **Step 1**: 后端起服务——本地无 MySQL 用开发模式:`DB_ENGINE=sqlite python manage.py runserver`(生成 db.sqlite3);有 MySQL 则按 `.env` 连接。`createsuperuser` 建 admin,`shell` 里建 Site(domains 含 127.0.0.1,`allow_no_origin=True`)。 - [ ] **Step 1**: 后端起服务(本地开发):`DB_ENGINE=sqlite python manage.py migrate && DJANGO_DEBUG=1 DB_ENGINE=sqlite python manage.py runserver`(生成 db.sqlite3,后续 Task 19-21 验证复用同一环境)。`createsuperuser` 建 admin。**demo 站点必须写死 site_id**(demo HTML 硬编码):`python manage.py shell` 执行 `Site.objects.create(name="demo", site_id="site_x", domains=["127.0.0.1"], allow_no_origin=True)`。
- [ ] **Step 2**: `cd sdk && python -m http.server 8080`,浏览器开 `http://127.0.0.1:8080/demo/`,点击、停留、切后台、跳 page2。 - [ ] **Step 2**: `cd sdk && python -m http.server 8080`,浏览器开 `http://127.0.0.1:8080/demo/`,点击、停留、切后台、跳 page2。
- [ ] **Step 3**: 断言:`Event` 表出现 page_view/page_stay/sdk_loaded;Visitor.visit_count/active_days 正确;重复刷新不产生重复 event_id;后台 Django shell 查询验证(截图/记录到 PR 描述)。 - [ ] **Step 3**: 断言:`Event` 表出现 page_view/page_stay/sdk_loaded;Visitor.visit_count/active_days 正确;重复刷新不产生重复 event_id;后台 Django shell 查询验证(截图/记录到 PR 描述)。
- [ ] **Step 4: Commit**(如有修复)`fix: SDK↔后端联调修复` - [ ] **Step 4: Commit**(如有修复)`fix: SDK↔后端联调修复`
@@ -1260,15 +1370,15 @@ gzip -c dist/v1.js | wc -c # 必须 ≤ 15360(15KB),超标则精简(
### Task 17: 模板落地与请求层适配 ### Task 17: 模板落地与请求层适配
**Files:** **Files:**
- Create: `admin/`(模板克隆),Modify: `admin/.env.development`, `admin/vite.config.ts`, `admin/src/utils/request/*`(按模板实际结构定位) - Create: `admin/`(模板克隆),Modify: `admin/.env.development`, `admin/.env.production`, `admin/vite.config.ts`, `admin/src/utils/request/*`(按模板实际结构定位)
- [ ] **Step 1**: 克隆模板 - [ ] **Step 1**: 克隆模板(大陆网络备用:gitee 镜像 `https://gitee.com/lingchen163/art-design-pro.git`;npm 慢加 `--registry=https://registry.npmmirror.com`)
```bash ```bash
git clone --depth 1 https://github.com/Daymychen/art-design-pro.git admin git clone --depth 1 https://github.com/Daymychen/art-design-pro.git admin
cd admin && rm -rf .git && pnpm install && pnpm dev # 确认模板可跑 cd admin && rm -rf .git && pnpm install && pnpm dev # 确认模板可跑
pnpm clean:dev # 一键精简演示页(按提示输入 yes) pnpm clean:dev # 一键精简演示页(按提示输入 yes)
``` ```
- [ ] **Step 2**: `.env.development` 设 `VITE_API_URL=/api`;`vite.config.ts` 加 proxy `"/api": { target: "http://127.0.0.1:8000", changeOrigin: true }`。 - [ ] **Step 2**: `.env.development` 设 `VITE_API_URL=/api`;**`.env.production` 同样设 `VITE_API_URL=/api`(模板默认指向 Apifox Mock 域名,漏改则 `pnpm build` 的产物请求 mock,部署必挂)**;`vite.config.ts` 加 proxy `"/api": { target: "http://127.0.0.1:8000", changeOrigin: true }`。
- [ ] **Step 3**: 改造模板 `src/utils/request`(或等价拦截器文件):响应成功判定改为 `res.data.code === 200` 时 `return res.data.data`;401 跳登录页携带 redirect;token 从登录响应存入(沿用模板的 token 管理,替换其 mock 登录为真实 API)。 - [ ] **Step 3**: 改造模板 `src/utils/request`(或等价拦截器文件):响应成功判定改为 `res.data.code === 200` 时 `return res.data.data`;401 跳登录页携带 redirect;token 从登录响应存入(沿用模板的 token 管理,替换其 mock 登录为真实 API)。
- [ ] **Step 4**: 登录页接 `/api/admin/auth/token`(用户名/密码 → access/refresh),登录成功进布局页。手动验证:错误密码报错、正确 token 进入首页。 - [ ] **Step 4**: 登录页接 `/api/admin/auth/token`(用户名/密码 → access/refresh),登录成功进布局页。手动验证:错误密码报错、正确 token 进入首页。
- [ ] **Step 5: Commit** `feat(admin): 模板精简+请求层适配 Django JWT` - [ ] **Step 5: Commit** `feat(admin): 模板精简+请求层适配 Django JWT`
@@ -1303,7 +1413,7 @@ export const getSession = (vid: string, sid: string) =>
- Create: `admin/src/views/tracker/site/index.vue`, `admin/src/views/tracker/dashboard/index.vue` - Create: `admin/src/views/tracker/site/index.vue`, `admin/src/views/tracker/dashboard/index.vue`
- [ ] **Step 1**: 站点页:Element Plus 表格(名称/site_id 复制按钮/域名列表 tag/状态开关)+ 新建编辑 Dialog(域名校验转小写)+ 停用开关调 PATCH。手动验证增删改。 - [ ] **Step 1**: 站点页:Element Plus 表格(名称/site_id 复制按钮/域名列表 tag/状态开关)+ 新建编辑 Dialog(域名校验转小写)+ 停用开关调 PATCH。手动验证增删改。
- [ ] **Step 2**: Dashboard 页:5 张统计卡(今日 PV/UV/会话/新访客/在线)、ECharts 30 天趋势折线(pv/uv/sessions 三线)、来源饼图 top8、热门页面表格。用模板自带图表组件或直接 ECharts。手动对拍后端返回。 - [ ] **Step 2**: Dashboard 页:5 张统计卡(今日 PV/UV/会话/新访客/在线)、ECharts 30 天趋势折线(pv/uv/sessions 三线)、来源饼图 top8、热门页面表格。用模板自带图表组件或直接 ECharts。手动对拍后端返回(后端以 `DJANGO_DEBUG=1 DB_ENGINE=sqlite python manage.py runserver` 起,复用 Task 16 的 db.sqlite3)。
- [ ] **Step 3: Commit** `feat(admin): 站点管理与总览页` - [ ] **Step 3: Commit** `feat(admin): 站点管理与总览页`
### Task 20: 访客列表页 ### Task 20: 访客列表页
@@ -1322,7 +1432,7 @@ export const getSession = (vid: string, sid: string) =>
- [ ] **Step 1**: 布局:顶部档案卡(访客ID/国家·城市/IP/设备环境 grid/首触来源 utm 链/统计 4 数字)+ 时间线主体(`el-timeline`,日期分组倒序;组内会话块:起止时间+来源+时长,事件行:`10:24 · 看了 /pricing · 停留 2分18秒`,事件 icon 按类型区分 page_view/page_stay 合并行/sdk_loaded)+ 顶部"会话"Tab 切换会话列表(含页面路径视图:路径序列 + 各页停留)。 - [ ] **Step 1**: 布局:顶部档案卡(访客ID/国家·城市/IP/设备环境 grid/首触来源 utm 链/统计 4 数字)+ 时间线主体(`el-timeline`,日期分组倒序;组内会话块:起止时间+来源+时长,事件行:`10:24 · 看了 /pricing · 停留 2分18秒`,事件 icon 按类型区分 page_view/page_stay 合并行/sdk_loaded)+ 顶部"会话"Tab 切换会话列表(含页面路径视图:路径序列 + 各页停留)。
- [ ] **Step 2**: 停留格式化工具(`formatStay(ms)`);空态/加载态;days 切换(7/30/90)。 - [ ] **Step 2**: 停留格式化工具(`formatStay(ms)`);空态/加载态;days 切换(7/30/90)。
- [ ] **Step 3**: 手动验证(真实数据跨天回看:Day1 与 Day2 各一组)。**这是 M1 验收页面。** - [ ] **Step 3**: 手动验证:当天数据即可验收交互;跨天分组用 Django shell 直插一条昨日事件验证分组渲染(`Event.objects.create(..., occurred_at=昨天时间)` 绕过 ingest 时间窗),真实跨天效果并入 Task 23 次日复验。**这是 M1 验收页面。**
- [ ] **Step 4: Commit** `feat(admin): 访客详情跨天时间线` - [ ] **Step 4: Commit** `feat(admin): 访客详情跨天时间线`
--- ---
@@ -1341,35 +1451,60 @@ WORKDIR /app
COPY requirements.txt . COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple RUN pip install --no-cache-dir -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
COPY . . COPY . .
RUN python manage.py collectstatic --noinput 2>/dev/null || true
EXPOSE 8000 EXPOSE 8000
CMD ["sh", "-c", "python manage.py migrate && gunicorn config.wsgi:application -b 0.0.0.0:8000 -w 3 --forwarded-allow-ips=*"] CMD ["sh", "-c", "python manage.py migrate && gunicorn config.wsgi:application -b 0.0.0.0:8000 -w 3 --forwarded-allow-ips=*"]
``` ```
- [ ] **Step 2**: `deploy/nginx.conf` 关键段(compose 服务名统一为 `web`):`location /api/ { proxy_pass http://web:8000; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }`(配合 DRF `NUM_PROXIES=1` 取真实 IP);`location /sdk/ { alias /var/www/sdk/; }`(**静态别名**,放 `sdk/dist/v1.js`,不走代理);`location / { root /var/www/admin; try_files $uri $uri/ /index.html; }`(SPA fallback)。 - [ ] **Step 2**: `deploy/nginx.conf` 关键段(compose 服务名统一为 `web`):`location /api/ { client_max_body_size 256k; proxy_pass http://web:8000; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }`(体积上限第一道 + 配合 DRF `NUM_PROXIES=1` 取真实 IP);`location /sdk/ { alias /var/www/sdk/; }`(**静态别名**,放 `sdk/dist/v1.js`,不走代理);`location / { root /var/www/admin; try_files $uri $uri/ /index.html; }`(SPA fallback)。
- [ ] **Step 3**: `docker-compose.yml`:`web`(build server, env_file .env, 依赖外部 MySQL——1Panel 已有容器,compose 里不建 MySQL,`DB_HOST` 填宿主内网地址)、`nginx`(挂 admin/dist 与 sdk/dist、80/443)。`.env.example` 列全环境变量。 - [ ] **Step 3**: `deploy/docker-compose.yml`(全量写出)+ `server/.dockerignore`(内容:`.venv/`、`db.sqlite3`、`__pycache__/`、`.cache/`——否则 Windows 的 .venv(数百 MB)会被 `COPY . .` 打进镜像):
- [ ] **Step 4**: `README-deploy.md`:**前置构建**(`cd admin && pnpm build`、`cd sdk && pnpm build` 产出 dist 供挂载)→ 1Panel 步骤(建 MySQL 库 → Compose 部署 → 进容器 `createsuperuser` → 上传 xdb → 1Panel 网站功能给 API 域名挂 HTTPS 证书 → 建站点拿 site_id → 目标网站贴接入代码)→ 目标网站接入片段(同 sdk/demo,endpoint 换正式域名)→ **若目标站套 CDN/WAF,X-Forwarded-For 层数变化需同步调整 `_client_ip` 与 `NUM_PROXIES`**。
```yaml
services:
web:
build: ../server
env_file: .env # 必含 DJANGO_ENV=prod DJANGO_DEBUG=0 DJANGO_SECRET_KEY DJANGO_ALLOWED_HOSTS DB_*
volumes:
- ./data/ip2region.xdb:/app/tracker/data/ip2region.xdb:ro # xdb 挂载进容器(不是 docker cp,重建不丢)
extra_hosts:
- "host.docker.internal:host-gateway"
nginx:
image: nginx:alpine
ports: ["80:80", "443:443"]
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
- ../admin/dist:/var/www/admin:ro
- ../sdk/dist:/var/www/sdk:ro
depends_on: [web]
```
**连 1Panel 已有 MySQL 的三条路**(任选其一,README 写清排查步骤):
1. 上面已配的 `extra_hosts: host-gateway` + `DB_HOST=host.docker.internal`——前提:MySQL 端口发布在 `0.0.0.0` 且账号 host 为 `%`;
2. **推荐**:加入 1Panel 网络——compose 末尾加 `networks: { default: { external: true, name: 1panel-network } }`,`DB_HOST=<MySQL 容器名>`;
3. `DB_HOST=172.17.0.1`(docker0 网关,MySQL 需监听可达)。
- [ ] **Step 4**: `README-deploy.md`:**前置构建**(`cd admin && pnpm build`、`cd sdk && pnpm build` 产出 dist 供挂载)→ 1Panel 步骤(建 MySQL 库 → Compose 部署 → 进容器 `createsuperuser` → xdb 放宿主 `deploy/data/` 由 volume 挂载 → 1Panel 网站功能给 API 域名挂 HTTPS 证书 → 建站点拿 site_id → 目标网站贴接入代码)→ 目标网站接入片段(同 sdk/demo,endpoint 换正式域名)→ **生产 .env 必填项清单**(`DJANGO_ENV=prod`、随机 `SECRET_KEY`:`python -c "import secrets; print(secrets.token_urlsafe(48))"`、实际域名的 `ALLOWED_HOSTS`——漏配启动即崩,这是设计好的 fail-fast)→ **JWT 应急预案**(token 泄露唯一吊销手段 = 换 SECRET_KEY,全体登出)→ **大陆网络备用源**(模板 gitee 镜像、xdb gitee 下载、`pip -i 清华源`、`pnpm --registry=https://registry.npmmirror.com`)→ **若目标站套 CDN/WAF,X-Forwarded-For 层数变化需同步调整 `_client_ip` 与 `NUM_PROXIES`**。
- [ ] **Step 5: Commit** `chore: Docker/Nginx 部署产物与 1Panel 指南` - [ ] **Step 5: Commit** `chore: Docker/Nginx 部署产物与 1Panel 指南`
### Task 23: M1 端到端验收(真实网站) ### Task 23: M1 端到端验收(真实网站)
- [ ] **Step 1**: 选一个真实网站(用户的 WordPress 站或 demo 站)贴接入代码,白名单填其域名。 - [ ] **Step 1**: 选一个真实网站贴接入代码(白名单填其域名)。**前置(spec 治理底线,不装不验收):目标站先上线隐私告知**——一段话说明"本站使用第一方分析存储收集访问行为与设备信息,用于改进服务",附联系方式(文案模板取 spec §9.4)。
- [ ] **Step 2**: 按验收清单逐项过(写入 PR/交付记录): - [ ] **Step 2**: 按验收清单逐项过(写入 PR/交付记录):
- [ ] 新访客首次访问 → 后台访客列表出现,环境/来源/地理正确 - [ ] 新访客首次访问 → 后台访客列表出现,环境/来源/地理正确
- [ ] 浏览 3 个页面各停留可感知时长 → 时间线 3 条 page_view 且 active_ms 合理 - [ ] 浏览 3 个页面各停留可感知时长 → 时间线 3 条 page_view 且 active_ms 合理
- [ ] 隔天再访(**需真实次日复验**——服务端拒绝 >24h 时间偏移事件,无法伪造跨天数据,这是防篡改设计;当天先验证:等 30 分钟后新访问使 visit_count+1、时间线新增会话组)→ 同 visitor_id、visit_count+1、时间线新增日期组 - [ ] 隔天再访(**需真实次日复验**——服务端拒绝 >24h 时间偏移事件,无法伪造跨天数据,这是防篡改设计;当天先验证:等 30 分钟后新访问使 visit_count+1、时间线新增会话组)→ 同 visitor_id、visit_count+1、时间线新增日期组
- [ ] 首触来源 Google/带 UTM 链接进入 → first_referrer_domain/first_utm 正确 - [ ] 首触来源 Google/带 UTM 链接进入 → first_referrer_domain/first_utm 正确(无 Google 环境时:无痕窗口 + `?utm_source=google&utm_medium=test` 链接造新访客验证)
- [ ] 直接刷新/断网重试/重复上报 → 无重复事件(event_id 幂等) - [ ] 直接刷新/断网重试/重复上报 → 无重复事件(event_id 幂等)(断网用 DevTools Network→Offline 再恢复)
- [ ] 非白名单域 curl 伪造上报 → 403;超频 → 429 - [ ] curl 伪造非白名单 Origin 上报 → 403;超频 → 429(注意:白名单 Origin 本身可被伪造,这是归属控制不是认证——spec §11 定位)
- [ ] Dashboard 数字与访客明细对得上;总停留=各会话之和 - [ ] Dashboard 数字与访客明细对得上;总停留=各会话之和
- [ ] SDK 对页面加载性能无可感知影响(DevTools Network 确认 async + ≤15KB) - [ ] SDK 对页面加载性能无可感知影响(DevTools Network 确认 async + ≤15KB)
- [ ] **Step 3**: 全部通过 → `git tag m1-beta`,Commit 验收记录 `docs: M1 验收通过` - [ ] **Step 3**: 全部通过 → `git tag m1-beta`;验收记录(含截图与数据对拍)写入 `docs/验收记录-m1.md`,Commit `docs: M1 验收通过`。**同时记录采集开始日期:M2 的 IP 截断任务必须在其 180 天内上线,否则先跑截断脚本再继续采集。**
--- ---
## 执行注意 ## 执行注意
- Task 6 ingest 的 `duration_ms` 以"计划内正确版本"实现(F 表达式),不要照抄占位伪码。 - Task 6 ingest 的 `duration_ms` 以"计划内正确版本"实现(F 表达式 + new_events 排除重放),不要照抄占位伪码。
- Art Design Pro 模板内部组件 API 以克隆下来的实际代码为准(Task 17 后先读 `src/views` 现存示例再写页面);本计划页面代码是基线,允许按模板约定适配,**不允许**砍功能。 - geo.py 是进程级单例(xdb 一次读入 buffer);dropped 统一按事件数计。
- MySQL 的 `INSERT IGNORE`(ignore_conflicts)会连字段截断类错误一起吞——`_FIELD_LEN` 已按模型长度截断,属双保险;导入/迁移后抽查字段长度。
- Art Design Pro 模板内部组件 API 以克隆下来的实际代码为准(Task 17 后先读 `src/views` 现存示例再写页面,注意模板 request 封装签名可能是 `request.get({url, params})`);本计划页面代码是基线,允许按模板约定适配,**不允许**砍功能。
- ip2region pip 包若与计划 API 不符,参照其 README 调整 `geo.py`,测试断言不变。 - ip2region pip 包若与计划 API 不符,参照其 README 调整 `geo.py`,测试断言不变。
- 所有中文文案直接用于界面;代码注释保持中文、少量、说明意图。 - 所有中文文案直接用于界面;代码注释保持中文、少量、说明意图。
- 每任务完成即提交;测试红→绿→提交,禁止跳测试。 - 每任务完成即提交;测试红→绿→提交,禁止跳测试。