docs: M1 计划四维审查修复(30+ 处)
- 后端: py-ip2region 正确包名与API/会话缓存键/并发F表达式/重放聚合排除/parser与Django5时区500/类型与体积深度白名单/限流测试夹具/字段截断 - SDK: localStorage内存降级/异步路径try-catch/批次context补首触字段/page_stay同源path/4xx不重试/测试技巧 - 安全: 生产fail-fast/请求体双层上限/deny统一防枚举/拒绝回ACAO/文件缓存共享限流/前端禁v-html/JWT取舍明示 - 交付: .env.production/xdb volume挂载/compose全量与MySQL三路径/.dockerignore/大陆镜像源/隐私告知前置/跨天验证指引 Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
@@ -38,7 +38,7 @@ cms-cj/
|
||||
│ │ ├── management/commands/refresh_stats.py
|
||||
│ │ └── tests/ # pytest 测试
|
||||
│ ├── Dockerfile
|
||||
│ └── data/ip2region.xdb # 离线 IP 库(下载,不进 git)
|
||||
│ └── tracker/data/ip2region.xdb # 离线 IP 库(下载,不进 git,部署时 volume 挂载)
|
||||
├── sdk/ # JS SDK
|
||||
│ ├── package.json vitest.config.ts build.mjs
|
||||
│ ├── src/{index,config,identity,context,transport,events,util}.ts
|
||||
@@ -57,6 +57,7 @@ cms-cj/
|
||||
- **响应包**:所有 `/api/admin/*` 返回 `{code: 200, data: ..., message: "ok"}`(适配 Art Design Pro 拦截器默认约定);错误 `code` 为业务码(401 未登录/403 无权/4xx 参数/500 内部)。
|
||||
- **时间**:全库 UTC 存储(`USE_TZ=True`),展示层按 `settings.TIME_ZONE='Asia/Shanghai'`。
|
||||
- **采集跨域**:SDK 用 `Content-Type: text/plain` 发 JSON(简单请求免预检),服务端用 `PlainJSONParser` 解析;`django-cors-headers` + 动态白名单(DB 中 Site.domain_whitelist,60s 缓存)回 `Access-Control-Allow-Origin`。
|
||||
- **前端安全红线**:全项目禁用 `v-html`;访客来源数据(path/title/referrer/utm)不得直接绑定 `href/src`——如需渲染成链接,协议白名单仅 http/https(Vue 不拦截 `javascript:` URI)。
|
||||
- **测试**:后端 `pytest`(SQLite 内存库,`pytest.ini` 指定 `--ds=config.settings_test`);SDK `vitest` + happy-dom + fake timers。
|
||||
- **提交**:每任务一提交,`feat:/test:/chore:` 前缀。
|
||||
- **M1 明确不做**(防执行者自作主张加码):SPA 路由、点击/滚动/表单/意图事件、客户(contact)体系、Redis、事件表分区迁移、npm 发布。
|
||||
@@ -79,7 +80,7 @@ djangorestframework>=3.15
|
||||
djangorestframework-simplejwt>=5.3
|
||||
django-cors-headers>=4.4
|
||||
PyMySQL>=1.1
|
||||
ip2region>=2.0
|
||||
py-ip2region>=3.0
|
||||
gunicorn>=22.0
|
||||
```
|
||||
|
||||
@@ -109,8 +110,11 @@ import os
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent.parent
|
||||
SECRET_KEY = os.environ.get("DJANGO_SECRET_KEY", "dev-insecure")
|
||||
DEBUG = os.environ.get("DJANGO_DEBUG", "1") == "1"
|
||||
DEBUG = os.environ.get("DJANGO_DEBUG", "0" if os.environ.get("DJANGO_ENV") == "prod" else "1") == "1"
|
||||
ALLOWED_HOSTS = os.environ.get("DJANGO_ALLOWED_HOSTS", "*").split(",")
|
||||
if os.environ.get("DJANGO_ENV") == "prod": # 生产 fail-fast:漏配即启动崩溃,不带病上线
|
||||
assert SECRET_KEY != "dev-insecure", "生产必须配置 DJANGO_SECRET_KEY"
|
||||
assert os.environ.get("DJANGO_ALLOWED_HOSTS"), "生产必须显式配置 DJANGO_ALLOWED_HOSTS"
|
||||
|
||||
INSTALLED_APPS = [
|
||||
"django.contrib.contenttypes",
|
||||
@@ -154,6 +158,9 @@ REST_FRAMEWORK = {
|
||||
}
|
||||
SIMPLE_JWT = {"ACCESS_TOKEN_LIFETIME": timedelta(hours=12)}
|
||||
CORS_ALLOW_ALL_ORIGINS = False
|
||||
CACHES = {"default": {"BACKEND": "django.core.cache.backends.filebased.FileBasedCache",
|
||||
"LOCATION": str(BASE_DIR / ".cache")}} # 限流计数跨 gunicorn worker 共享(不共享则限额×3)
|
||||
DATA_UPLOAD_MAX_MEMORY_SIZE = 262144 # 256KB 应用层请求体兜底
|
||||
TIME_ZONE = "Asia/Shanghai"
|
||||
USE_TZ = True
|
||||
LANGUAGE_CODE = "zh-hans"
|
||||
@@ -186,6 +193,13 @@ SENSITIVE_KEY_PATTERNS = ("password", "passwd", "token", "secret", "card", "veri
|
||||
URL_PARAM_DROP_PATTERNS = ("email", "phone", "mobile", "token", "key", "password", "verif")
|
||||
SESSION_IDLE_MS = 30 * 60 * 1000
|
||||
BOT_UA_KEYWORDS = ("bot", "spider", "crawl", "slurp", "bingpreview", "headless", "lighthouse")
|
||||
EVENT_PROP_WHITELIST = { # spec §9.1:未知字段拒绝。免费文本/任意属性不得入库
|
||||
"page_view": {"referrer", "referrer_domain", "utm"},
|
||||
"page_stay": {"active_ms", "path"},
|
||||
"sdk_loaded": {"sdk_version"},
|
||||
}
|
||||
MAX_PROP_DEPTH = 2
|
||||
MAX_PROP_JSON_BYTES = 4096
|
||||
```
|
||||
|
||||
- [ ] **Step 3: 建 Django 工程与 app 骨架**
|
||||
@@ -466,6 +480,11 @@ def test_sensitive_property_rejected():
|
||||
r = validate_event(_ev(properties={"user_password": "1"}))
|
||||
assert "sensitive" in r
|
||||
|
||||
def test_shape_type_and_whitelist():
|
||||
assert validate_event(["x"]) == "event_invalid_shape"
|
||||
assert validate_event(_ev(event_id=123)) == "invalid_event_id"
|
||||
assert validate_event(_ev(properties={"foo": "bar"})) == "property_not_allowed"
|
||||
|
||||
def test_too_many_props_and_long_string():
|
||||
r = validate_event(_ev(properties={f"k{i}": "v" for i in range(40)}))
|
||||
assert "properties" in r
|
||||
@@ -478,8 +497,9 @@ def test_future_and_stale_time():
|
||||
assert validate_event(_ev(occurred_at=NOW - timedelta(hours=2))) is None # 窗内迟到可收
|
||||
|
||||
def test_sanitize_url_drops_sensitive_params():
|
||||
u = sanitize_url("/p?email=a@b.com&token=x&id=3&utm_source=google")
|
||||
u = sanitize_url("/p?email=a@b.com&token=x&id=3&utm_source=google&monkey=1")
|
||||
assert "email=" not in u and "token=" not in u and "utm_source=google" in u
|
||||
assert "monkey=1" in u # fullmatch:含 key 子串的正常参数不误伤
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 跑测试确认失败** → ModuleNotFoundError。
|
||||
@@ -487,23 +507,44 @@ def test_sanitize_url_drops_sensitive_params():
|
||||
- [ ] **Step 3: 实现**
|
||||
|
||||
```python
|
||||
import json
|
||||
import re
|
||||
from datetime import timedelta
|
||||
from datetime import datetime, timedelta
|
||||
from urllib.parse import parse_qsl, urlencode, urlparse
|
||||
from django.utils import timezone
|
||||
from tracker import constants as C
|
||||
|
||||
_SENSITIVE = re.compile("|".join(C.SENSITIVE_KEY_PATTERNS), re.I)
|
||||
_URL_DROP = re.compile("&?(%s)=[^&]*" % "|".join(C.URL_PARAM_DROP_PATTERNS), re.I)
|
||||
_URL_DROP_KEY = re.compile("|".join(C.URL_PARAM_DROP_PATTERNS), re.I)
|
||||
|
||||
def sanitize_url(url: str) -> str:
|
||||
return _URL_DROP.sub("", url).replace("?&", "?").rstrip("?")
|
||||
"""按参数名整键删除(fullmatch 防止误伤 monkey 这类含 key 子串的参数)。"""
|
||||
p = urlparse(url)
|
||||
kept = [(k, v) for k, v in parse_qsl(p.query, keep_blank_values=True)
|
||||
if not _URL_DROP_KEY.fullmatch(k)]
|
||||
return p._replace(query=urlencode(kept)).geturl()
|
||||
|
||||
def validate_event(ev: dict) -> str | None:
|
||||
"""返回 None 表示通过;否则返回拒绝原因码。"""
|
||||
def _depth_ok(v, depth=0) -> bool:
|
||||
if depth > C.MAX_PROP_DEPTH:
|
||||
return False
|
||||
if isinstance(v, dict):
|
||||
return all(_depth_ok(x, depth + 1) for x in v.values())
|
||||
if isinstance(v, list):
|
||||
return all(_depth_ok(x, depth + 1) for x in v)
|
||||
return True
|
||||
|
||||
def validate_event(ev) -> str | None:
|
||||
"""返回 None 表示通过;否则返回拒绝原因码。类型/白名单/体积/深度全在此守门。"""
|
||||
if not isinstance(ev, dict):
|
||||
return "event_invalid_shape"
|
||||
if ev.get("event_name") not in C.EVENT_NAMES:
|
||||
return "event_name_not_allowed"
|
||||
if not ev.get("event_id") or len(ev["event_id"]) > 64:
|
||||
eid = ev.get("event_id")
|
||||
if not isinstance(eid, str) or not eid or len(eid) > 64:
|
||||
return "invalid_event_id"
|
||||
for k in ("path", "title"):
|
||||
if ev.get(k) is not None and not isinstance(ev[k], str):
|
||||
return "invalid_field_type"
|
||||
occ = ev.get("occurred_at")
|
||||
if not isinstance(occ, datetime):
|
||||
return "invalid_occurred_at"
|
||||
@@ -513,6 +554,12 @@ def validate_event(ev: dict) -> str | None:
|
||||
props = ev.get("properties") or {}
|
||||
if not isinstance(props, dict) or len(props) > C.MAX_PROP_KEYS:
|
||||
return "properties_invalid"
|
||||
if not set(props) <= C.EVENT_PROP_WHITELIST.get(ev["event_name"], set()):
|
||||
return "property_not_allowed"
|
||||
if not _depth_ok(props):
|
||||
return "properties_depth"
|
||||
if len(json.dumps(props, ensure_ascii=False, default=str)) > C.MAX_PROP_JSON_BYTES:
|
||||
return "properties_too_large"
|
||||
for k, v in props.items():
|
||||
if _SENSITIVE.search(str(k)):
|
||||
return "sensitive_field"
|
||||
@@ -559,13 +606,17 @@ import functools, ipaddress
|
||||
from django.conf import settings
|
||||
|
||||
class GeoResolver:
|
||||
"""进程级单例:xdb 一次读入内存(官方 buffer 模式声明线程安全)。
|
||||
pip 包名 py-ip2region(模块名 ip2region);包缺失或 xdb 缺失时优雅降级返回 None。"""
|
||||
|
||||
def __init__(self):
|
||||
self._searcher = None
|
||||
try:
|
||||
from ip2region import XdbSearcher # pip ip2region;失败则降级
|
||||
self._searcher_cls = XdbSearcher
|
||||
from ip2region import searcher as ip2region_searcher
|
||||
with open(settings.IP2REGION_XDB, "rb") as f:
|
||||
self._searcher = ip2region_searcher.new_with_buffer(f.read())
|
||||
except Exception:
|
||||
self._searcher_cls = None
|
||||
self._searcher = None
|
||||
|
||||
@functools.lru_cache(maxsize=10000)
|
||||
def resolve(self, ip: str):
|
||||
@@ -576,17 +627,16 @@ class GeoResolver:
|
||||
return None
|
||||
except ValueError:
|
||||
return None
|
||||
if not self._searcher_cls:
|
||||
if not self._searcher:
|
||||
return None
|
||||
try:
|
||||
with self._searcher_cls(contentBuff=open(settings.IP2REGION_XDB, "rb").read()) as s:
|
||||
raw = s.search(ip) # "国家|区域|省|市|ISP"
|
||||
raw = self._searcher.search(ip) # "国家|区域|省|市|ISP"
|
||||
parts = (raw or "").split("|")
|
||||
return {"country": parts[0], "province": parts[2], "city": parts[3]}
|
||||
except Exception:
|
||||
return None
|
||||
```
|
||||
注:若 `ip2region` pip 包 API 不同,以官方 README 为准适配 `XdbSearcher` 的调用方式;解析结果格式 `国家|区域|省|市|ISP`。中文区返回如 `中国|0|广东省|深圳市|电信`,海外如 `美国|0|加利福尼亚|旧金山|0`。
|
||||
注:py-ip2region 的实际调用以其 README 为准(模块 `ip2region`,入口 `searcher.new_with_buffer(buff)` 后 `search(ip)`;**不支持 with 上下文管理器**——用错会被 except 静默吞掉、地理功能永久失效且测试照样绿,务必按此实现)。解析结果格式 `国家|区域|省|市|ISP`,如 `中国|0|广东省|深圳市|电信`、`美国|0|加利福尼亚|旧金山|0`。
|
||||
|
||||
- [ ] **Step 3: 跑测试通过**(`server/.gitignore` 加 `tracker/data/*.xdb`)
|
||||
|
||||
@@ -662,6 +712,7 @@ def test_rejected_events_reported_with_reason():
|
||||
```python
|
||||
from datetime import timedelta
|
||||
from django.db import transaction
|
||||
from django.db.models import F
|
||||
from django.utils import timezone
|
||||
from tracker import constants as C
|
||||
from tracker.models import Visitor, VisitSession, Event
|
||||
@@ -670,27 +721,34 @@ from tracker.services.geo import GeoResolver
|
||||
|
||||
_geo = GeoResolver()
|
||||
|
||||
_FIELD_LEN = {"device_type": 16, "os": 64, "browser": 64, "language": 32, "screen": 32}
|
||||
|
||||
def _clean_utm(d):
|
||||
"""utm 键白名单(utm_* 前缀)+ 截断——任意字符串不得经 utm 入库渲染。"""
|
||||
return {k[:50]: str(v)[:100] for k, v in (d or {}).items()
|
||||
if isinstance(k, str) and k.startswith("utm_")}
|
||||
|
||||
def _touch_visitor(site, vid, ctx, ip):
|
||||
env = {k: str(ctx.get(k, "") or "")[:n] for k, n in _FIELD_LEN.items()}
|
||||
v, created = Visitor.objects.get_or_create(
|
||||
site=site, visitor_id=vid,
|
||||
defaults={"last_ip": ip, **{k: str(ctx.get(k, "") or "")[:64]
|
||||
for k in ("device_type", "os", "browser", "language", "screen")}},
|
||||
)
|
||||
if created:
|
||||
v.first_referrer_domain = (ctx.get("referrer_domain") or "")[:200]
|
||||
v.first_utm = {k: str(val)[:100] for k, val in (ctx.get("utm") or {}).items()}
|
||||
v.first_landing_path = (ctx.get("path") or "")[:500]
|
||||
site=site, visitor_id=vid, defaults={**env, "last_ip": ip})
|
||||
now = timezone.now()
|
||||
v.last_seen_at = now
|
||||
upd = {"last_seen_at": now} # 聚合字段一律 filter().update(),避免并发下全量 save() 用旧值覆盖赢家
|
||||
if created:
|
||||
upd.update(first_referrer_domain=(ctx.get("referrer_domain") or "")[:200],
|
||||
first_utm=_clean_utm(ctx.get("utm")),
|
||||
first_landing_path=(ctx.get("path") or "")[:500],
|
||||
last_active_date=now.date(), active_days=1)
|
||||
elif not Visitor.objects.filter(pk=v.pk, last_active_date=now.date()).exists():
|
||||
upd.update(last_active_date=now.date(), active_days=F("active_days") + 1)
|
||||
if ip:
|
||||
v.last_ip = ip
|
||||
upd["last_ip"] = ip
|
||||
g = _geo.resolve(ip)
|
||||
if g:
|
||||
v.country, v.province, v.city = g["country"], g["province"], g["city"]
|
||||
if v.last_active_date != now.date():
|
||||
v.active_days += 1
|
||||
v.last_active_date = now.date()
|
||||
v.save()
|
||||
upd.update(country=g["country"][:64], province=g["province"][:64],
|
||||
city=g["city"][:64])
|
||||
Visitor.objects.filter(pk=v.pk).update(**upd)
|
||||
v.refresh_from_db()
|
||||
return v
|
||||
|
||||
def _get_session(site, visitor, sid, ctx, occurred_at):
|
||||
@@ -703,14 +761,19 @@ def _get_session(site, visitor, sid, ctx, occurred_at):
|
||||
sid = f"{sid}#{suffix}"
|
||||
sess = None
|
||||
if not sess:
|
||||
sess = VisitSession.objects.create(
|
||||
site=site, visitor=visitor, session_id=sid,
|
||||
started_at=occurred_at, ended_at=occurred_at,
|
||||
referrer_domain=(ctx.get("referrer_domain") or "")[:200],
|
||||
utm={k: str(val)[:100] for k, val in (ctx.get("utm") or {}).items()},
|
||||
landing_path=(ctx.get("path") or "")[:500])
|
||||
Visitor.objects.filter(pk=visitor.pk).update(visit_count=visitor.visit_count + 1)
|
||||
visitor.visit_count += 1
|
||||
try:
|
||||
sess = VisitSession.objects.create(
|
||||
site=site, visitor=visitor, session_id=sid,
|
||||
started_at=occurred_at, ended_at=occurred_at,
|
||||
referrer_domain=(ctx.get("referrer_domain") or "")[:200],
|
||||
utm=_clean_utm(ctx.get("utm")),
|
||||
landing_path=(ctx.get("path") or "")[:500])
|
||||
except IntegrityError: # 并发同 sid 同时切分出相同后缀撞唯一键——重查复用
|
||||
from django.db import IntegrityError as _IE # 顶部统一 import IntegrityError
|
||||
sess = VisitSession.objects.filter(site=site, session_id=sid).first()
|
||||
if not sess:
|
||||
raise
|
||||
Visitor.objects.filter(pk=visitor.pk).update(visit_count=F("visit_count") + 1)
|
||||
return sess
|
||||
|
||||
@transaction.atomic
|
||||
@@ -739,7 +802,8 @@ def ingest_batch(site, payload: dict, ip: str | None, ua: str = "") -> dict:
|
||||
sess = session_cache.get(sid)
|
||||
if not sess:
|
||||
sess = _get_session(site, visitor, sid, ctx, occ)
|
||||
session_cache[sess.session_id] = sess
|
||||
session_cache[sid] = sess # 必须按原始 sid 存:切分兜底后 sess.session_id 是 "s1#2",
|
||||
# 若按它存,同批后续事件按 "s1" 查会 miss → 每条再切一个新会话
|
||||
ready.append(Event(
|
||||
site=site, visitor=visitor, session=sess,
|
||||
event_id=ev["event_id"], event_name=ev["event_name"],
|
||||
@@ -776,8 +840,9 @@ def ingest_batch(site, payload: dict, ip: str | None, ua: str = "") -> dict:
|
||||
```python
|
||||
from django.db.models import F, Greatest
|
||||
|
||||
for sess in {e.session for e in ready}:
|
||||
se = [e for e in ready if e.session.pk == sess.pk]
|
||||
new_events = [e for e in ready if e.event_id not in existed] # 重放命中的不参与聚合!
|
||||
for sess in {e.session for e in new_events}:
|
||||
se = [e for e in new_events if e.session.pk == sess.pk]
|
||||
new_end = max(x.occurred_at for x in se)
|
||||
span_ms = int((new_end - min(x.occurred_at for x in se)).total_seconds() * 1000)
|
||||
stay_add = sum(int(e.properties.get("active_ms", 0) or 0)
|
||||
@@ -788,6 +853,7 @@ for sess in {e.session for e in ready}:
|
||||
duration_ms=F("duration_ms") + span_ms,
|
||||
stay_ms=F("stay_ms") + stay_add) # 有效停留单独累计,供访客"总停留"取数
|
||||
```
|
||||
- **重放批不得抬高会话聚合**:event_count/stay_ms/duration_ms 只用 `new_events`(否则重试/重放被幂等去重后仍把会话计数刷高——`test_replay_duplicated` 需补断言 `VisitSession.event_count == 1`)。
|
||||
- 同批多会话各自按上式计算。
|
||||
- `duplicated` 计数允许并发近似(唯一约束保证绝不重复入库,仅统计口径可能差 1-2,M1 接受)。
|
||||
|
||||
@@ -803,12 +869,17 @@ for sess in {e.session for e in ready}:
|
||||
- [ ] **Step 1: 失败测试**(要点:白名单域放行/非白名单 403/限流触发 429/textplain 解析/响应最小化/幂等重放)
|
||||
```python
|
||||
import json
|
||||
from datetime import timezone as tz
|
||||
import pytest
|
||||
from django.core.cache import cache
|
||||
from django.utils import timezone
|
||||
from tracker.models import Site
|
||||
|
||||
NOW = timezone.now()
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clear_cache():
|
||||
cache.clear() # locmem 限流历史跨测试残留,会让 429 用例假红
|
||||
|
||||
def _post(client, site, body, origin="https://www.a.com", extra=None):
|
||||
return client.post("/api/collect/batch", data=json.dumps(body),
|
||||
content_type="text/plain", HTTP_ORIGIN=origin, **(extra or {}))
|
||||
@@ -860,8 +931,7 @@ def test_throttle_429(db, client, settings):
|
||||
`server/tracker/api/collect.py`:
|
||||
```python
|
||||
import json
|
||||
from datetime import datetime
|
||||
from django.utils import timezone
|
||||
from datetime import datetime, timezone as dt_tz
|
||||
from rest_framework.views import APIView
|
||||
from rest_framework.response import Response
|
||||
from rest_framework.permissions import AllowAny
|
||||
@@ -873,7 +943,11 @@ from tracker.services.ingest import ingest_batch
|
||||
class PlainJSONParser(BaseParser):
|
||||
media_type = "text/plain"
|
||||
def parse(self, stream, *args, **kwargs):
|
||||
return json.loads(stream.read().decode("utf-8") or "{}")
|
||||
try:
|
||||
return json.loads(stream.read().decode("utf-8") or "{}")
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
from rest_framework.exceptions import ParseError
|
||||
raise ParseError("invalid_json") # DRF 对 parser 异常是 re-raise 不是包装,不自包就是 500
|
||||
|
||||
class CollectThrottle(AnonRateThrottle):
|
||||
scope = "collect"
|
||||
@@ -882,7 +956,7 @@ def _iso(dt):
|
||||
if isinstance(dt, str):
|
||||
s = dt.replace("Z", "+00:00")
|
||||
d = datetime.fromisoformat(s)
|
||||
return d if d.tzinfo else d.replace(tzinfo=timezone.utc)
|
||||
return d if d.tzinfo else d.replace(tzinfo=dt_tz.utc) # Django≥5 已删除 django.utils.timezone.utc
|
||||
return dt
|
||||
|
||||
class CollectBatchView(APIView):
|
||||
@@ -893,18 +967,20 @@ class CollectBatchView(APIView):
|
||||
|
||||
def post(self, request):
|
||||
origin = request.headers.get("Origin") or request.headers.get("Referer") or ""
|
||||
if int(request.headers.get("Content-Length") or 0) > 262144:
|
||||
return self._deny(400, "payload_too_large") # 应用层兜底(nginx 另设 client_max_body_size)
|
||||
site = self._match_site(request)
|
||||
if site is None or not site.is_active:
|
||||
return self._deny(403, "site_not_allowed")
|
||||
return self._deny(403, "not_allowed") # 三种拒绝统一 reason,防 site_id 枚举探测
|
||||
host = (origin.split("//")[-1].split("/")[0].split(":")[0].lower()
|
||||
if origin else "")
|
||||
if host and host not in [d.lower() for d in site.domains]:
|
||||
return self._deny(403, "origin_not_allowed")
|
||||
return self._deny(403, "not_allowed")
|
||||
if not host and not site.allow_no_origin:
|
||||
return self._deny(403, "origin_required")
|
||||
return self._deny(403, "not_allowed")
|
||||
payload = request.data or {}
|
||||
for ev in payload.get("events") or []:
|
||||
if "occurred_at" in ev:
|
||||
if isinstance(ev, dict) and "occurred_at" in ev:
|
||||
try:
|
||||
ev["occurred_at"] = _iso(ev["occurred_at"])
|
||||
except ValueError:
|
||||
@@ -931,7 +1007,11 @@ class CollectBatchView(APIView):
|
||||
return request.META.get("REMOTE_ADDR")
|
||||
|
||||
def _deny(self, code, reason):
|
||||
return Response({"code": code, "data": None, "message": reason}, status=code)
|
||||
resp = Response({"code": code, "data": None, "message": reason}, status=code)
|
||||
origin = self.request.headers.get("Origin")
|
||||
if origin: # 拒绝也要回 ACAO,否则浏览器读不到错误体、SDK 误判为网络错误走重试
|
||||
resp["Access-Control-Allow-Origin"] = origin
|
||||
return resp
|
||||
```
|
||||
`server/config/urls.py`:
|
||||
```python
|
||||
@@ -968,16 +1048,19 @@ return resp
|
||||
- [ ] **Step 1: 失败测试**:造 2 天数据(访客 v1 两天访问、v2 一天)→ `refresh(site, date)` 后 DailyStat 的 pv/uv/sessions/new_visitors 正确;uv 按访客去重、new_visitors 按 first_seen_at 当日计数。
|
||||
- [ ] **Step 2: 实现** `stats.py`:
|
||||
```python
|
||||
from datetime import datetime, time, timedelta
|
||||
from django.utils import timezone
|
||||
from tracker.models import DailyStat, Event, VisitSession
|
||||
|
||||
def refresh_daily(site, date) -> DailyStat:
|
||||
from tracker.models import Event, VisitSession, Visitor, DailyStat
|
||||
from django.db.models import Count
|
||||
day_start = timezone.make_aware(datetime.combine(date, time.min))
|
||||
day_end = day_start + timedelta(days=1)
|
||||
pv = Event.objects.filter(site=site, is_bot=False,
|
||||
occurred_at__gte=day_start, occurred_at__lt=day_end).count()
|
||||
uv = Event.objects.filter(site=site, is_bot=False, occurred_at__gte=day_start,
|
||||
occurred_at__lt=day_end).values("visitor").distinct().count()
|
||||
sessions = VisitSession.objects.filter(site=site, started_at__gte=day_start,
|
||||
sessions = VisitSession.objects.filter(site=site, visitor__is_bot=False,
|
||||
started_at__gte=day_start,
|
||||
started_at__lt=day_end).count()
|
||||
new_v = Visitor.objects.filter(site=site, first_seen_at__gte=day_start,
|
||||
first_seen_at__lt=day_end).count()
|
||||
@@ -995,7 +1078,7 @@ def refresh_daily(site, date) -> DailyStat:
|
||||
- Create: `server/tracker/api/auth.py`, `server/tracker/tests/test_auth.py`, Modify: `config/urls.py`
|
||||
|
||||
- [ ] **Step 1: 失败测试**:`/api/admin/auth/token` 用户名密码 → access/refresh;错密码 401;`/api/admin/auth/me` 带 token 返回 username/is_admin(is_staff);无 token 401;`/api/admin/users` 仅 is_staff 可 POST 建普通成员。
|
||||
- [ ] **Step 2: 实现**:`TokenObtainPairView` 子类(返回 `{code:200,data:{access,refresh},message:"ok"}` 包一层),`me` 返回 `{username, is_admin: user.is_staff}`;`UserCreateView`(IsAdminUser)`username/password` 创建 `is_staff=False` 用户。登录接口挂 `ScopedRateThrottle` scope=`login`(10/min/IP,防爆破;settings 已含 rate)。
|
||||
- [ ] **Step 2: 实现**:`TokenObtainPairView` 子类(返回 `{code:200,data:{access,refresh},message:"ok"}` 包一层),`me` 返回 `{username, is_admin: user.is_staff}`;`UserCreateView`(IsAdminUser)`username/password` 创建 `is_staff=False` 用户。登录接口挂 `ScopedRateThrottle` scope=`login`(10/min/IP,防爆破;settings 已含 rate)。**JWT 取舍(M1 决策,README-deploy 写明)**:access 12h、refresh 显式配置 24h、无黑名单/轮换——token 泄露的唯一应急是更换 SECRET_KEY(全体登出)。
|
||||
- [ ] **Step 3: 测试通过,Commit** `feat: JWT 登录与用户接口`
|
||||
|
||||
### Task 10: 站点管理 API
|
||||
@@ -1043,7 +1126,7 @@ GET /api/admin/visitors/<visitor_id>/sessions/<session_id> 单会话页面路
|
||||
- 列表字段齐全:visitor_id/country/province/city/device_type/os/browser/first_seen_at/last_seen_at/visit_count/active_days/session_count/total_stay_ms/first_source/last_source/online;筛选 country/device_type/online=true/search=访客ID或IP前缀;按 last_seen_at 倒序分页(page_size 20)。
|
||||
- timeline 结构:`[{date:"2026-09-15", sessions:[{session_id, started_at, events:[{time,event_name,path,title,active_ms}]}]}]`,多天数据按日倒序、日内时间正序;默认近 30 天,`?days=` 可调,单次上限 1000 事件。
|
||||
- 会话详情:事件按 occurred_at 正序,含 active_ms(page_stay 的 properties.active_ms 就地并入相邻 page_view 或单列,见实现说明)。
|
||||
- [ ] **Step 2: 实现**:ORM 查询 + Python 分组;`total_stay_ms` = `Sum("sessions__stay_ms")` 注解(**有效停留口径,不用 duration_ms 跨度**);`first_source` = `first_referrer_domain or first_utm.utm_source or "直接访问"`;`last_source` 取最近会话同逻辑;`online` = last_seen_at > now-5min;列表默认过滤 `is_bot=False`(`?show_bot=1` 可查爬虫)。**page_view 与 page_stay 合并展示**:时间线返回时若 page_stay 的 path 与前一条 page_view 相同,则把 `active_ms` 合入该 page_view(服务端做,前端省事)。
|
||||
- [ ] **Step 2: 实现**:ORM 查询 + Python 分组;`total_stay_ms` = `Sum("sessions__stay_ms")` 注解(**有效停留口径,不用 duration_ms 跨度**);`first_source` = `first_referrer_domain or first_utm.utm_source or "直接访问"`;`last_source` 取最近会话同逻辑;`online` = last_seen_at > now-5min;列表默认过滤 `is_bot=False`(`?show_bot=1` 可查爬虫)。**page_view 与 page_stay 合并展示**:时间线返回时若 page_stay 的 path 与前一条 page_view 相同,则把 `active_ms` 合入该 page_view(服务端做,前端省事);同 path 连续多段 page_stay(多次隐藏/恢复)的 active_ms **累加**后合入。会话详情展示"时长"统一用 `stay_ms` 口径(`duration_ms` 是批内跨度累加,仅内部字段不展示)。
|
||||
- [ ] **Step 3: 测试通过,Commit** `feat: 访客列表/详情/时间线/会话接口`
|
||||
|
||||
---
|
||||
@@ -1084,16 +1167,27 @@ context 测试:`parseUtm("?utm_source=google&utm_medium=cpc&other=1")` 只留
|
||||
import { uuid } from "./util";
|
||||
const K = { vid: "ctk_vid", sid: "ctk_sid", act: "ctk_act" };
|
||||
const IDLE = 30 * 60_000;
|
||||
// localStorage 可能抛异常(隐私模式/禁存储,Safari ITP):统一走带内存降级的 store,
|
||||
// 任何路径都不得让异常冒泡到宿主页(spec §4.5 红线)
|
||||
const mem: Record<string, string> = {};
|
||||
const store = {
|
||||
get(k: string): string | null {
|
||||
try { return localStorage.getItem(k); } catch { return mem[k] ?? null; }
|
||||
},
|
||||
set(k: string, v: string) {
|
||||
try { localStorage.setItem(k, v); } catch { mem[k] = v; }
|
||||
},
|
||||
};
|
||||
export function getVisitorId(): string {
|
||||
let v = localStorage.getItem(K.vid);
|
||||
if (!v) { v = uuid(); localStorage.setItem(K.vid, v); }
|
||||
let v = store.get(K.vid);
|
||||
if (!v) { v = uuid(); store.set(K.vid, v); }
|
||||
return v;
|
||||
}
|
||||
export function touchActivity() { localStorage.setItem(K.act, String(Date.now())); }
|
||||
export function touchActivity() { store.set(K.act, String(Date.now())); }
|
||||
export function getSessionId(): string {
|
||||
const last = +(localStorage.getItem(K.act) || 0);
|
||||
let s = localStorage.getItem(K.sid);
|
||||
if (!s || Date.now() - last > IDLE) { s = uuid(); localStorage.setItem(K.sid, s); }
|
||||
const last = +(store.get(K.act) || 0);
|
||||
let s = store.get(K.sid);
|
||||
if (!s || Date.now() - last > IDLE) { s = uuid(); store.set(K.sid, s); }
|
||||
touchActivity();
|
||||
return s;
|
||||
}
|
||||
@@ -1109,9 +1203,10 @@ export function getSessionId(): string {
|
||||
- [ ] **Step 1: 失败测试**(vi.stubGlobal fetch;fake timers):
|
||||
- 满 10 条即发;15 秒定时器到即发;payload 含 batch_id/site_id/visitor_id/session_id/events。
|
||||
- 500 失败→退避重试(500ms×2^n 抖动)最多 3 次→丢弃计数;400 不重试直接丢。
|
||||
- 429 → 重试。
|
||||
- `flushBeacon()` 调 `navigator.sendBeacon`(spy 断言 URL 与 body)。
|
||||
- 队列上限 200,超限丢最旧并计数。
|
||||
- 429 → 重试;403/400 → 不重试直接丢。
|
||||
- `flushBeacon()` 调 `navigator.sendBeacon`(spy 断言 URL 与 body)。**happy-dom 没有 sendBeacon,用 `vi.stubGlobal("navigator", { sendBeacon: spy })`,测完 restore**。
|
||||
- 队列上限 200(用 `batchSize > queueMax` 的自定义 cfg 测,默认 batchSize=10 到不了 200),超限丢最旧并计数。
|
||||
- **测试技巧**:异步断言用 `await vi.advanceTimersByTimeAsync()`(flush/send 是 async);fake timers 需含 Date(vitest 默认 toFake 即含);`document.hidden` 是 getter,用 `Object.defineProperty(document, "hidden", { value: true, configurable: true })` + `dispatchEvent(new Event("visibilitychange"))` 模拟;每个用例后 `vi.useRealTimers()`。
|
||||
- [ ] **Step 2: 实现**
|
||||
```ts
|
||||
import { uuid } from "./util";
|
||||
@@ -1134,29 +1229,33 @@ export class Transport {
|
||||
flush(beacon = false) {
|
||||
if (this.timer) { clearTimeout(this.timer); this.timer = null; }
|
||||
if (!this.q.length) return;
|
||||
const events = this.q; this.q = [];
|
||||
const body = JSON.stringify({ batch_id: uuid(), site_id: this.cfg.siteId,
|
||||
visitor_id: getVisitorId(), session_id: getSessionId(),
|
||||
sent_at: new Date().toISOString(), context: (globalThis as any).__ctk_ctx,
|
||||
events });
|
||||
if (beacon && navigator.sendBeacon) {
|
||||
navigator.sendBeacon(this.cfg.endpoint, new Blob([body], {type: "text/plain"}));
|
||||
return;
|
||||
}
|
||||
this.send(body);
|
||||
try {
|
||||
const events = this.q; this.q = [];
|
||||
const body = JSON.stringify({ batch_id: uuid(), site_id: this.cfg.siteId,
|
||||
visitor_id: getVisitorId(), session_id: getSessionId(),
|
||||
sent_at: new Date().toISOString(), context: (globalThis as any).__ctk_ctx,
|
||||
events });
|
||||
if (beacon && navigator.sendBeacon) {
|
||||
navigator.sendBeacon(this.cfg.endpoint, new Blob([body], {type: "text/plain"}));
|
||||
return;
|
||||
}
|
||||
this.send(body, events.length);
|
||||
} catch { /* 定时器/事件回调内的任何异常不得冒泡到宿主页 */ }
|
||||
}
|
||||
private async send(body: string) {
|
||||
private async send(body: string, count: number) {
|
||||
try {
|
||||
const res = await fetch(this.cfg.endpoint, { method: "POST",
|
||||
keepalive: true, headers: { "Content-Type": "text/plain" }, body });
|
||||
if (res.status === 400) { this.retry = 0; return; } // 参数错不重试
|
||||
if (res.status >= 400 && res.status < 500 && res.status !== 429) {
|
||||
this.retry = 0; return; // 4xx 参数/权限错不重试(403 重试只会掩盖配置错误)
|
||||
}
|
||||
if (!res.ok) throw new Error(String(res.status));
|
||||
this.retry = 0;
|
||||
} catch {
|
||||
if (this.retry++ < this.cfg.maxRetry) {
|
||||
const delay = 500 * 2 ** this.retry * (0.8 + Math.random() * 0.4);
|
||||
setTimeout(() => this.send(body), delay);
|
||||
} else { this.retry = 0; this.dropped += 1; }
|
||||
setTimeout(() => this.send(body, count).catch(() => {}), delay);
|
||||
} else { this.retry = 0; this.dropped += count; } // 按事件数计丢弃
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1170,8 +1269,11 @@ export class Transport {
|
||||
- Create: `sdk/src/events.ts`, `sdk/src/index.ts`, `sdk/test/events.test.ts`, `sdk/demo/index.html`, Modify: `.gitignore`(`sdk/node_modules`, `sdk/dist`)
|
||||
|
||||
- [ ] **Step 1: 失败测试**:
|
||||
- init 后 `sdk_loaded` 事件入队;`page_view` 含 path/title/referrer_domain/utm(utm 仅首页带)。
|
||||
- init 后 `sdk_loaded` 事件入队;`page_view` 含 path/title/referrer_domain/utm(utm 随每页上报,服务端只写首个,可接受冗余)。
|
||||
- 批次 context 含设备环境 + referrer_domain/utm/path(首触归因链路,断言 `__ctk_ctx` 字段齐全)。
|
||||
- 有效停留:fake timers 下 `visible` 且有 pointerdown → 3 秒后 active_ms≈3000;`document.hidden=true` 后不再累计;隐藏时自动 flush beacon(page_stay 已入队)。
|
||||
- **localStorage 抛异常时 15 秒定时 flush 不产生未捕获异常**(stub localStorage 全方法抛错 + `await vi.advanceTimersByTimeAsync(15000)`,配 `vi.onUnhandledError` 断言;且身份降级到内存后事件照常发出)。
|
||||
- 隐藏→恢复→再隐藏产生两段 page_stay,active_ms 各自正确(第二段从 0 重新累计)。
|
||||
- [ ] **Step 2: 实现**
|
||||
```ts
|
||||
// events.ts
|
||||
@@ -1184,7 +1286,7 @@ export function startPageTracking(t: Transport) {
|
||||
const act = () => { lastAct = Date.now(); };
|
||||
["pointerdown", "keydown", "wheel", "touchstart"].forEach(e =>
|
||||
document.addEventListener(e, act, { passive: true, capture: true }));
|
||||
setInterval(() => { if (!document.hidden && Date.now() - lastAct < 30_000) activeMs += 1000; }, 1000);
|
||||
setInterval(() => { try { if (!document.hidden && Date.now() - lastAct < 30_000) activeMs += 1000; } catch { /* 静默 */ } }, 1000);
|
||||
const emit = (event_name: string, properties: any = {}) => {
|
||||
const pc = pageContext();
|
||||
t.push({ event_id: uuid(), event_name, occurred_at: new Date().toISOString(),
|
||||
@@ -1193,8 +1295,12 @@ export function startPageTracking(t: Transport) {
|
||||
emit("page_view", { referrer: document.referrer,
|
||||
referrer_domain: pageContext().referrer_domain, utm: pageContext().utm });
|
||||
const close = () => {
|
||||
if (activeMs > 0) emit("page_stay", { active_ms: activeMs, path: location.pathname });
|
||||
activeMs = 0; t.flush(true);
|
||||
try {
|
||||
// path 必须与 page_view 同源同清洗(pageContext().path 含 query),
|
||||
// 否则带 UTM 进落地页时服务端"page_stay 合并进相邻 page_view"匹配失败
|
||||
if (activeMs > 0) emit("page_stay", { active_ms: activeMs, path: pageContext().path });
|
||||
activeMs = 0; t.flush(true);
|
||||
} catch { /* 静默 */ }
|
||||
};
|
||||
document.addEventListener("visibilitychange", () => { if (document.hidden) close(); });
|
||||
window.addEventListener("pagehide", close);
|
||||
@@ -1205,7 +1311,7 @@ export function startPageTracking(t: Transport) {
|
||||
import { mergeConfig } from "./config";
|
||||
import { Transport } from "./transport";
|
||||
import { startPageTracking } from "./events";
|
||||
import { buildEnvContext } from "./context";
|
||||
import { buildEnvContext, pageContext } from "./context";
|
||||
|
||||
const g = globalThis as any;
|
||||
const q: any[][] = g.ctk?.q || [];
|
||||
@@ -1214,7 +1320,11 @@ let transport: Transport | null = null;
|
||||
function dispatch(cmd: string, arg?: any) {
|
||||
if (cmd === "init" && !transport) {
|
||||
const cfg = mergeConfig(arg);
|
||||
(globalThis as any).__ctk_ctx = buildEnvContext();
|
||||
// 首触归因修复:批次 context 除设备环境外必须带 referrer_domain/utm/path——
|
||||
// 服务端 _touch_visitor/_get_session 只从 context 读首触,缺了则首触/会话来源恒空
|
||||
const pc = pageContext();
|
||||
(globalThis as any).__ctk_ctx = { ...buildEnvContext(),
|
||||
referrer_domain: pc.referrer_domain, utm: pc.utm, path: pc.path };
|
||||
transport = new Transport(cfg);
|
||||
transport.push({ event_id: uuid(), event_name: "sdk_loaded",
|
||||
occurred_at: new Date().toISOString(),
|
||||
@@ -1248,7 +1358,7 @@ gzip -c dist/v1.js | wc -c # 必须 ≤ 15360(15KB),超标则精简(
|
||||
|
||||
### Task 16: SDK ↔ 后端本地集成验证(手动脚本)
|
||||
|
||||
- [ ] **Step 1**: 后端起服务——本地无 MySQL 用开发模式:`DB_ENGINE=sqlite python manage.py runserver`(生成 db.sqlite3);有 MySQL 则按 `.env` 连接。`createsuperuser` 建 admin,`shell` 里建 Site(domains 含 127.0.0.1,`allow_no_origin=True`)。
|
||||
- [ ] **Step 1**: 后端起服务(本地开发):`DB_ENGINE=sqlite python manage.py migrate && DJANGO_DEBUG=1 DB_ENGINE=sqlite python manage.py runserver`(生成 db.sqlite3,后续 Task 19-21 验证复用同一环境)。`createsuperuser` 建 admin。**demo 站点必须写死 site_id**(demo HTML 硬编码):`python manage.py shell` 执行 `Site.objects.create(name="demo", site_id="site_x", domains=["127.0.0.1"], allow_no_origin=True)`。
|
||||
- [ ] **Step 2**: `cd sdk && python -m http.server 8080`,浏览器开 `http://127.0.0.1:8080/demo/`,点击、停留、切后台、跳 page2。
|
||||
- [ ] **Step 3**: 断言:`Event` 表出现 page_view/page_stay/sdk_loaded;Visitor.visit_count/active_days 正确;重复刷新不产生重复 event_id;后台 Django shell 查询验证(截图/记录到 PR 描述)。
|
||||
- [ ] **Step 4: Commit**(如有修复)`fix: SDK↔后端联调修复`
|
||||
@@ -1260,15 +1370,15 @@ gzip -c dist/v1.js | wc -c # 必须 ≤ 15360(15KB),超标则精简(
|
||||
### Task 17: 模板落地与请求层适配
|
||||
|
||||
**Files:**
|
||||
- Create: `admin/`(模板克隆),Modify: `admin/.env.development`, `admin/vite.config.ts`, `admin/src/utils/request/*`(按模板实际结构定位)
|
||||
- Create: `admin/`(模板克隆),Modify: `admin/.env.development`, `admin/.env.production`, `admin/vite.config.ts`, `admin/src/utils/request/*`(按模板实际结构定位)
|
||||
|
||||
- [ ] **Step 1**: 克隆模板
|
||||
- [ ] **Step 1**: 克隆模板(大陆网络备用:gitee 镜像 `https://gitee.com/lingchen163/art-design-pro.git`;npm 慢加 `--registry=https://registry.npmmirror.com`)
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/Daymychen/art-design-pro.git admin
|
||||
cd admin && rm -rf .git && pnpm install && pnpm dev # 确认模板可跑
|
||||
pnpm clean:dev # 一键精简演示页(按提示输入 yes)
|
||||
```
|
||||
- [ ] **Step 2**: `.env.development` 设 `VITE_API_URL=/api`;`vite.config.ts` 加 proxy `"/api": { target: "http://127.0.0.1:8000", changeOrigin: true }`。
|
||||
- [ ] **Step 2**: `.env.development` 设 `VITE_API_URL=/api`;**`.env.production` 同样设 `VITE_API_URL=/api`(模板默认指向 Apifox Mock 域名,漏改则 `pnpm build` 的产物请求 mock,部署必挂)**;`vite.config.ts` 加 proxy `"/api": { target: "http://127.0.0.1:8000", changeOrigin: true }`。
|
||||
- [ ] **Step 3**: 改造模板 `src/utils/request`(或等价拦截器文件):响应成功判定改为 `res.data.code === 200` 时 `return res.data.data`;401 跳登录页携带 redirect;token 从登录响应存入(沿用模板的 token 管理,替换其 mock 登录为真实 API)。
|
||||
- [ ] **Step 4**: 登录页接 `/api/admin/auth/token`(用户名/密码 → access/refresh),登录成功进布局页。手动验证:错误密码报错、正确 token 进入首页。
|
||||
- [ ] **Step 5: Commit** `feat(admin): 模板精简+请求层适配 Django JWT`
|
||||
@@ -1303,7 +1413,7 @@ export const getSession = (vid: string, sid: string) =>
|
||||
- Create: `admin/src/views/tracker/site/index.vue`, `admin/src/views/tracker/dashboard/index.vue`
|
||||
|
||||
- [ ] **Step 1**: 站点页:Element Plus 表格(名称/site_id 复制按钮/域名列表 tag/状态开关)+ 新建编辑 Dialog(域名校验转小写)+ 停用开关调 PATCH。手动验证增删改。
|
||||
- [ ] **Step 2**: Dashboard 页:5 张统计卡(今日 PV/UV/会话/新访客/在线)、ECharts 30 天趋势折线(pv/uv/sessions 三线)、来源饼图 top8、热门页面表格。用模板自带图表组件或直接 ECharts。手动对拍后端返回。
|
||||
- [ ] **Step 2**: Dashboard 页:5 张统计卡(今日 PV/UV/会话/新访客/在线)、ECharts 30 天趋势折线(pv/uv/sessions 三线)、来源饼图 top8、热门页面表格。用模板自带图表组件或直接 ECharts。手动对拍后端返回(后端以 `DJANGO_DEBUG=1 DB_ENGINE=sqlite python manage.py runserver` 起,复用 Task 16 的 db.sqlite3)。
|
||||
- [ ] **Step 3: Commit** `feat(admin): 站点管理与总览页`
|
||||
|
||||
### Task 20: 访客列表页
|
||||
@@ -1322,7 +1432,7 @@ export const getSession = (vid: string, sid: string) =>
|
||||
|
||||
- [ ] **Step 1**: 布局:顶部档案卡(访客ID/国家·城市/IP/设备环境 grid/首触来源 utm 链/统计 4 数字)+ 时间线主体(`el-timeline`,日期分组倒序;组内会话块:起止时间+来源+时长,事件行:`10:24 · 看了 /pricing · 停留 2分18秒`,事件 icon 按类型区分 page_view/page_stay 合并行/sdk_loaded)+ 顶部"会话"Tab 切换会话列表(含页面路径视图:路径序列 + 各页停留)。
|
||||
- [ ] **Step 2**: 停留格式化工具(`formatStay(ms)`);空态/加载态;days 切换(7/30/90)。
|
||||
- [ ] **Step 3**: 手动验证(真实数据跨天回看:Day1 与 Day2 各一组)。**这是 M1 验收页面。**
|
||||
- [ ] **Step 3**: 手动验证:当天数据即可验收交互;跨天分组用 Django shell 直插一条昨日事件验证分组渲染(`Event.objects.create(..., occurred_at=昨天时间)` 绕过 ingest 时间窗),真实跨天效果并入 Task 23 次日复验。**这是 M1 验收页面。**
|
||||
- [ ] **Step 4: Commit** `feat(admin): 访客详情跨天时间线`
|
||||
|
||||
---
|
||||
@@ -1341,35 +1451,60 @@ WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
COPY . .
|
||||
RUN python manage.py collectstatic --noinput 2>/dev/null || true
|
||||
EXPOSE 8000
|
||||
CMD ["sh", "-c", "python manage.py migrate && gunicorn config.wsgi:application -b 0.0.0.0:8000 -w 3 --forwarded-allow-ips=*"]
|
||||
```
|
||||
- [ ] **Step 2**: `deploy/nginx.conf` 关键段(compose 服务名统一为 `web`):`location /api/ { proxy_pass http://web:8000; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }`(配合 DRF `NUM_PROXIES=1` 取真实 IP);`location /sdk/ { alias /var/www/sdk/; }`(**静态别名**,放 `sdk/dist/v1.js`,不走代理);`location / { root /var/www/admin; try_files $uri $uri/ /index.html; }`(SPA fallback)。
|
||||
- [ ] **Step 3**: `docker-compose.yml`:`web`(build server, env_file .env, 依赖外部 MySQL——1Panel 已有容器,compose 里不建 MySQL,`DB_HOST` 填宿主内网地址)、`nginx`(挂 admin/dist 与 sdk/dist、80/443)。`.env.example` 列全环境变量。
|
||||
- [ ] **Step 4**: `README-deploy.md`:**前置构建**(`cd admin && pnpm build`、`cd sdk && pnpm build` 产出 dist 供挂载)→ 1Panel 步骤(建 MySQL 库 → Compose 部署 → 进容器 `createsuperuser` → 上传 xdb → 1Panel 网站功能给 API 域名挂 HTTPS 证书 → 建站点拿 site_id → 目标网站贴接入代码)→ 目标网站接入片段(同 sdk/demo,endpoint 换正式域名)→ **若目标站套 CDN/WAF,X-Forwarded-For 层数变化需同步调整 `_client_ip` 与 `NUM_PROXIES`**。
|
||||
- [ ] **Step 2**: `deploy/nginx.conf` 关键段(compose 服务名统一为 `web`):`location /api/ { client_max_body_size 256k; proxy_pass http://web:8000; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }`(体积上限第一道 + 配合 DRF `NUM_PROXIES=1` 取真实 IP);`location /sdk/ { alias /var/www/sdk/; }`(**静态别名**,放 `sdk/dist/v1.js`,不走代理);`location / { root /var/www/admin; try_files $uri $uri/ /index.html; }`(SPA fallback)。
|
||||
- [ ] **Step 3**: `deploy/docker-compose.yml`(全量写出)+ `server/.dockerignore`(内容:`.venv/`、`db.sqlite3`、`__pycache__/`、`.cache/`——否则 Windows 的 .venv(数百 MB)会被 `COPY . .` 打进镜像):
|
||||
|
||||
```yaml
|
||||
services:
|
||||
web:
|
||||
build: ../server
|
||||
env_file: .env # 必含 DJANGO_ENV=prod DJANGO_DEBUG=0 DJANGO_SECRET_KEY DJANGO_ALLOWED_HOSTS DB_*
|
||||
volumes:
|
||||
- ./data/ip2region.xdb:/app/tracker/data/ip2region.xdb:ro # xdb 挂载进容器(不是 docker cp,重建不丢)
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports: ["80:80", "443:443"]
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
|
||||
- ../admin/dist:/var/www/admin:ro
|
||||
- ../sdk/dist:/var/www/sdk:ro
|
||||
depends_on: [web]
|
||||
```
|
||||
|
||||
**连 1Panel 已有 MySQL 的三条路**(任选其一,README 写清排查步骤):
|
||||
1. 上面已配的 `extra_hosts: host-gateway` + `DB_HOST=host.docker.internal`——前提:MySQL 端口发布在 `0.0.0.0` 且账号 host 为 `%`;
|
||||
2. **推荐**:加入 1Panel 网络——compose 末尾加 `networks: { default: { external: true, name: 1panel-network } }`,`DB_HOST=<MySQL 容器名>`;
|
||||
3. `DB_HOST=172.17.0.1`(docker0 网关,MySQL 需监听可达)。
|
||||
- [ ] **Step 4**: `README-deploy.md`:**前置构建**(`cd admin && pnpm build`、`cd sdk && pnpm build` 产出 dist 供挂载)→ 1Panel 步骤(建 MySQL 库 → Compose 部署 → 进容器 `createsuperuser` → xdb 放宿主 `deploy/data/` 由 volume 挂载 → 1Panel 网站功能给 API 域名挂 HTTPS 证书 → 建站点拿 site_id → 目标网站贴接入代码)→ 目标网站接入片段(同 sdk/demo,endpoint 换正式域名)→ **生产 .env 必填项清单**(`DJANGO_ENV=prod`、随机 `SECRET_KEY`:`python -c "import secrets; print(secrets.token_urlsafe(48))"`、实际域名的 `ALLOWED_HOSTS`——漏配启动即崩,这是设计好的 fail-fast)→ **JWT 应急预案**(token 泄露唯一吊销手段 = 换 SECRET_KEY,全体登出)→ **大陆网络备用源**(模板 gitee 镜像、xdb gitee 下载、`pip -i 清华源`、`pnpm --registry=https://registry.npmmirror.com`)→ **若目标站套 CDN/WAF,X-Forwarded-For 层数变化需同步调整 `_client_ip` 与 `NUM_PROXIES`**。
|
||||
- [ ] **Step 5: Commit** `chore: Docker/Nginx 部署产物与 1Panel 指南`
|
||||
|
||||
### Task 23: M1 端到端验收(真实网站)
|
||||
|
||||
- [ ] **Step 1**: 选一个真实网站(用户的 WordPress 站或 demo 站)贴接入代码,白名单填其域名。
|
||||
- [ ] **Step 1**: 选一个真实网站贴接入代码(白名单填其域名)。**前置(spec 治理底线,不装不验收):目标站先上线隐私告知**——一段话说明"本站使用第一方分析存储收集访问行为与设备信息,用于改进服务",附联系方式(文案模板取 spec §9.4)。
|
||||
- [ ] **Step 2**: 按验收清单逐项过(写入 PR/交付记录):
|
||||
- [ ] 新访客首次访问 → 后台访客列表出现,环境/来源/地理正确
|
||||
- [ ] 浏览 3 个页面各停留可感知时长 → 时间线 3 条 page_view 且 active_ms 合理
|
||||
- [ ] 隔天再访(**需真实次日复验**——服务端拒绝 >24h 时间偏移事件,无法伪造跨天数据,这是防篡改设计;当天先验证:等 30 分钟后新访问使 visit_count+1、时间线新增会话组)→ 同 visitor_id、visit_count+1、时间线新增日期组
|
||||
- [ ] 首触来源 Google/带 UTM 链接进入 → first_referrer_domain/first_utm 正确
|
||||
- [ ] 直接刷新/断网重试/重复上报 → 无重复事件(event_id 幂等)
|
||||
- [ ] 非白名单域 curl 伪造上报 → 403;超频 → 429
|
||||
- [ ] 首触来源 Google/带 UTM 链接进入 → first_referrer_domain/first_utm 正确(无 Google 环境时:无痕窗口 + `?utm_source=google&utm_medium=test` 链接造新访客验证)
|
||||
- [ ] 直接刷新/断网重试/重复上报 → 无重复事件(event_id 幂等)(断网用 DevTools Network→Offline 再恢复)
|
||||
- [ ] curl 伪造非白名单 Origin 上报 → 403;超频 → 429(注意:白名单 Origin 本身可被伪造,这是归属控制不是认证——spec §11 定位)
|
||||
- [ ] Dashboard 数字与访客明细对得上;总停留=各会话之和
|
||||
- [ ] SDK 对页面加载性能无可感知影响(DevTools Network 确认 async + ≤15KB)
|
||||
- [ ] **Step 3**: 全部通过 → `git tag m1-beta`,Commit 验收记录 `docs: M1 验收通过`
|
||||
- [ ] **Step 3**: 全部通过 → `git tag m1-beta`;验收记录(含截图与数据对拍)写入 `docs/验收记录-m1.md`,Commit `docs: M1 验收通过`。**同时记录采集开始日期:M2 的 IP 截断任务必须在其 180 天内上线,否则先跑截断脚本再继续采集。**
|
||||
|
||||
---
|
||||
|
||||
## 执行注意
|
||||
|
||||
- Task 6 ingest 的 `duration_ms` 以"计划内正确版本"实现(F 表达式),不要照抄占位伪码。
|
||||
- Art Design Pro 模板内部组件 API 以克隆下来的实际代码为准(Task 17 后先读 `src/views` 现存示例再写页面);本计划页面代码是基线,允许按模板约定适配,**不允许**砍功能。
|
||||
- Task 6 ingest 的 `duration_ms` 以"计划内正确版本"实现(F 表达式 + new_events 排除重放),不要照抄占位伪码。
|
||||
- geo.py 是进程级单例(xdb 一次读入 buffer);dropped 统一按事件数计。
|
||||
- MySQL 的 `INSERT IGNORE`(ignore_conflicts)会连字段截断类错误一起吞——`_FIELD_LEN` 已按模型长度截断,属双保险;导入/迁移后抽查字段长度。
|
||||
- Art Design Pro 模板内部组件 API 以克隆下来的实际代码为准(Task 17 后先读 `src/views` 现存示例再写页面,注意模板 request 封装签名可能是 `request.get({url, params})`);本计划页面代码是基线,允许按模板约定适配,**不允许**砍功能。
|
||||
- ip2region pip 包若与计划 API 不符,参照其 README 调整 `geo.py`,测试断言不变。
|
||||
- 所有中文文案直接用于界面;代码注释保持中文、少量、说明意图。
|
||||
- 每任务完成即提交;测试红→绿→提交,禁止跳测试。
|
||||
|
||||
Reference in New Issue
Block a user