feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -15,3 +15,22 @@ def reset_settings_cache():
|
||||
"""
|
||||
yield
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _disable_request_throttle(monkeypatch):
|
||||
"""Zero out the conservative request-pace settings for the test suite.
|
||||
|
||||
Production defaults (``request_min_interval=1.0``, ``request_jitter=0.8``)
|
||||
would make every mocked request sleep ~1-1.8s and balloon the suite's
|
||||
runtime. In tests we want the *behaviour* (header stealth, backoff, cap)
|
||||
without the wall-clock cost, so we pin the throttle env vars to 0. The
|
||||
defaults themselves are still asserted directly via ``Settings`` field
|
||||
introspection in ``test_config.py``. The cache is cleared on either side
|
||||
so the zeroed env is picked up the first time a test reads settings.
|
||||
"""
|
||||
get_settings.cache_clear()
|
||||
monkeypatch.setenv("REQUEST_MIN_INTERVAL", "0")
|
||||
monkeypatch.setenv("REQUEST_JITTER", "0")
|
||||
yield
|
||||
get_settings.cache_clear()
|
||||
|
||||
Reference in New Issue
Block a user