feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -67,3 +67,21 @@ def test_settings_constructable_without_env():
|
||||
s = Settings()
|
||||
assert isinstance(s.db_url, str)
|
||||
assert s.encryption_key == ""
|
||||
|
||||
|
||||
def test_stealth_defaults_are_conservative():
|
||||
"""The anti-risk-control settings declare conservative defaults.
|
||||
|
||||
Asserted via Pydantic field-introspection rather than ``Settings()`` so it
|
||||
is immune to the test-env throttle-zeroing in conftest (which intentionally
|
||||
pins ``REQUEST_MIN_INTERVAL``/``REQUEST_JITTER`` to 0 for suite speed).
|
||||
Production deployments that don't set these env vars get the slow,
|
||||
jittered defaults declared here.
|
||||
"""
|
||||
f = Settings.model_fields
|
||||
assert f["request_min_interval"].default == 1.0
|
||||
assert f["request_jitter"].default == 0.8
|
||||
assert f["max_requests_per_run"].default == 0
|
||||
assert f["risk_backoff_base"].default == 2.0
|
||||
assert f["risk_backoff_max"].default == 60.0
|
||||
assert f["risk_backoff_attempts"].default == 4
|
||||
|
||||
Reference in New Issue
Block a user