feat: anti-risk-control stealth layer (browser headers, jittered throttle, backoff, volume cap)
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -34,11 +34,16 @@ job is just to advance the cursor and stop on ``has_more == False``.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import random
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.session.yunke_client import YunkeClient
|
||||
if TYPE_CHECKING:
|
||||
# ``YunkeClient`` is referenced only in the paginator signatures. Kept
|
||||
# under TYPE_CHECKING so ``app.session.yunke_client`` may import
|
||||
# :class:`RateLimiter` at runtime without forming an import cycle.
|
||||
from app.session.yunke_client import YunkeClient
|
||||
|
||||
__all__ = ["RateLimiter", "paginate_by_cursor", "paginate_by_page"]
|
||||
|
||||
@@ -175,21 +180,49 @@ async def paginate_by_cursor(
|
||||
class RateLimiter:
|
||||
"""Minimal asyncio limiter enforcing a minimum interval between calls.
|
||||
|
||||
Call ``await limiter.acquire()`` before each upstream request. The first
|
||||
call never waits. Successive calls wait just long enough that the gap
|
||||
between the previous call's dispatch and this one is at least
|
||||
``min_interval`` seconds.
|
||||
Two pacing modes are supported:
|
||||
|
||||
* :meth:`acquire` — gap-based spacing. Call it before each upstream
|
||||
request; the first call never waits and successive calls wait just long
|
||||
enough that the gap between dispatches is at least ``min_interval``
|
||||
seconds. Used by the list paginators (:func:`paginate_by_page` /
|
||||
:func:`paginate_by_cursor`).
|
||||
|
||||
* :meth:`wait` — fixed per-call throttle. Sleeps ``min_interval`` plus a
|
||||
uniform random jitter in ``[0, jitter)`` every time it is awaited. Used
|
||||
by :class:`app.session.yunke_client.YunkeClient` to pace every upstream
|
||||
request with a conservative, slightly randomized cadence (the jitter
|
||||
prevents a perfectly periodic signature that risk control could finger).
|
||||
|
||||
This is deliberately tiny: no tokens, no concurrency cap, no skew handling
|
||||
— just a floor on inter-call spacing, which is all the yunke scrapers need
|
||||
to stay polite to the CRM. If you need a semaphore or token bucket, that
|
||||
belongs in a future helper, not here.
|
||||
|
||||
``jitter`` defaults to ``0.0`` so existing callers that pass only
|
||||
``min_interval`` keep their exact prior behaviour (gap-based ``acquire``
|
||||
never consults ``jitter``).
|
||||
"""
|
||||
|
||||
def __init__(self, min_interval: float = 0.0) -> None:
|
||||
def __init__(self, min_interval: float = 0.0, jitter: float = 0.0) -> None:
|
||||
self._min_interval = float(min_interval)
|
||||
self._jitter = float(jitter)
|
||||
self._last: float = 0.0
|
||||
|
||||
async def wait(self) -> None:
|
||||
"""Sleep ``min_interval`` plus a random ``[0, jitter)`` every call.
|
||||
|
||||
Unlike :meth:`acquire` this is not gap-aware — it is a flat, slightly
|
||||
randomized per-call delay, which is what the yunke client wants ahead
|
||||
of each upstream request. With ``min_interval == 0`` and
|
||||
``jitter == 0`` it is a no-op.
|
||||
"""
|
||||
delay = self._min_interval
|
||||
if self._jitter > 0:
|
||||
delay += random.uniform(0.0, self._jitter)
|
||||
if delay > 0:
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
async def acquire(self) -> None:
|
||||
"""Block until at least ``min_interval`` has elapsed since the last call."""
|
||||
if self._min_interval <= 0:
|
||||
|
||||
Reference in New Issue
Block a user