feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)

Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):

- app/config.py: add 6 tunable settings with conservative defaults
  (request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
  risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
  method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
  for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
  headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
  headers (per-request overrides win); awaits the shared jittered limiter
  before sending; enforces a per-run request cap (RequestCapReached); and
  backs off exponentially on risk signals (HTTP 429 or 200 body carrying
  频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
  conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
  override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
  raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.

Suite: 171 -> 181 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 14:51:46 +08:00
parent 83b1aeb97c
commit 46e3880c0b
6 changed files with 603 additions and 16 deletions
+18
View File
@@ -67,3 +67,21 @@ def test_settings_constructable_without_env():
s = Settings()
assert isinstance(s.db_url, str)
assert s.encryption_key == ""
def test_stealth_defaults_are_conservative():
"""The anti-risk-control settings declare conservative defaults.
Asserted via Pydantic field-introspection rather than ``Settings()`` so it
is immune to the test-env throttle-zeroing in conftest (which intentionally
pins ``REQUEST_MIN_INTERVAL``/``REQUEST_JITTER`` to 0 for suite speed).
Production deployments that don't set these env vars get the slow,
jittered defaults declared here.
"""
f = Settings.model_fields
assert f["request_min_interval"].default == 1.0
assert f["request_jitter"].default == 0.8
assert f["max_requests_per_run"].default == 0
assert f["risk_backoff_base"].default == 2.0
assert f["risk_backoff_max"].default == 60.0
assert f["risk_backoff_attempts"].default == 4