feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -28,6 +28,27 @@ class Settings(BaseSettings):
|
||||
# <token>`` or ``?token=<token>``. See app/api/auth.py.
|
||||
api_token: str = ""
|
||||
|
||||
# --- Anti-risk-control / stealth tuning ---------------------------------
|
||||
# These are the knobs that pace automated scraping so it resembles the
|
||||
# normal web app. Defaults are deliberately CONSERVATIVE (slow + jittered):
|
||||
# roughly one request every ~1.0..1.8s. Loosen these only once we have
|
||||
# evidence the CRM tolerates a faster pace. See app.session.yunke_client.
|
||||
# Lower bound on seconds between successive upstream requests.
|
||||
request_min_interval: float = 1.0
|
||||
# Random extra 0..jitter seconds added on top of ``request_min_interval``
|
||||
# per request, so the cadence is not a perfectly periodic signature.
|
||||
request_jitter: float = 0.8
|
||||
# Hard cap on the number of upstream requests a single client may send in
|
||||
# one run. 0 = unlimited; >0 = enforce, then raise RequestCapReached.
|
||||
max_requests_per_run: int = 0
|
||||
# Exponential backoff parameters applied when the CRM signals risk control
|
||||
# (HTTP 429 or a 200 body carrying a 风控/限流 marker). Delay for retry N
|
||||
# (0-indexed) is ``risk_backoff_base * 2**N`` clamped to ``risk_backoff_max``.
|
||||
risk_backoff_base: float = 2.0
|
||||
risk_backoff_max: float = 60.0
|
||||
# Number of retries attempted after the initial request before giving up.
|
||||
risk_backoff_attempts: int = 4
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_settings() -> Settings:
|
||||
|
||||
Reference in New Issue
Block a user