feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)

Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):

- app/config.py: add 6 tunable settings with conservative defaults
  (request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
  risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
  method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
  for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
  headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
  headers (per-request overrides win); awaits the shared jittered limiter
  before sending; enforces a per-run request cap (RequestCapReached); and
  backs off exponentially on risk signals (HTTP 429 or 200 body carrying
  频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
  conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
  override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
  raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.

Suite: 171 -> 181 passing. ruff clean.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
浅念
2026-08-01 14:51:46 +08:00
parent 83b1aeb97c
commit 46e3880c0b
6 changed files with 603 additions and 16 deletions
+40 -7
View File
@@ -34,11 +34,16 @@ job is just to advance the cursor and stop on ``has_more == False``.
from __future__ import annotations
import asyncio
import random
import time
from collections.abc import AsyncIterator, Awaitable, Callable
from typing import Any
from typing import TYPE_CHECKING, Any
from app.session.yunke_client import YunkeClient
if TYPE_CHECKING:
# ``YunkeClient`` is referenced only in the paginator signatures. Kept
# under TYPE_CHECKING so ``app.session.yunke_client`` may import
# :class:`RateLimiter` at runtime without forming an import cycle.
from app.session.yunke_client import YunkeClient
__all__ = ["RateLimiter", "paginate_by_cursor", "paginate_by_page"]
@@ -175,21 +180,49 @@ async def paginate_by_cursor(
class RateLimiter:
"""Minimal asyncio limiter enforcing a minimum interval between calls.
Call ``await limiter.acquire()`` before each upstream request. The first
call never waits. Successive calls wait just long enough that the gap
between the previous call's dispatch and this one is at least
``min_interval`` seconds.
Two pacing modes are supported:
* :meth:`acquire` — gap-based spacing. Call it before each upstream
request; the first call never waits and successive calls wait just long
enough that the gap between dispatches is at least ``min_interval``
seconds. Used by the list paginators (:func:`paginate_by_page` /
:func:`paginate_by_cursor`).
* :meth:`wait` — fixed per-call throttle. Sleeps ``min_interval`` plus a
uniform random jitter in ``[0, jitter)`` every time it is awaited. Used
by :class:`app.session.yunke_client.YunkeClient` to pace every upstream
request with a conservative, slightly randomized cadence (the jitter
prevents a perfectly periodic signature that risk control could finger).
This is deliberately tiny: no tokens, no concurrency cap, no skew handling
— just a floor on inter-call spacing, which is all the yunke scrapers need
to stay polite to the CRM. If you need a semaphore or token bucket, that
belongs in a future helper, not here.
``jitter`` defaults to ``0.0`` so existing callers that pass only
``min_interval`` keep their exact prior behaviour (gap-based ``acquire``
never consults ``jitter``).
"""
def __init__(self, min_interval: float = 0.0) -> None:
def __init__(self, min_interval: float = 0.0, jitter: float = 0.0) -> None:
self._min_interval = float(min_interval)
self._jitter = float(jitter)
self._last: float = 0.0
async def wait(self) -> None:
"""Sleep ``min_interval`` plus a random ``[0, jitter)`` every call.
Unlike :meth:`acquire` this is not gap-aware — it is a flat, slightly
randomized per-call delay, which is what the yunke client wants ahead
of each upstream request. With ``min_interval == 0`` and
``jitter == 0`` it is a no-op.
"""
delay = self._min_interval
if self._jitter > 0:
delay += random.uniform(0.0, self._jitter)
if delay > 0:
await asyncio.sleep(delay)
async def acquire(self) -> None:
"""Block until at least ``min_interval`` has elapsed since the last call."""
if self._min_interval <= 0: