浅念
46e3880c0b
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
37 lines
1.3 KiB
Python
37 lines
1.3 KiB
Python
"""Shared pytest fixtures for the yunke-server test suite."""
|
|
|
|
import pytest
|
|
|
|
from app.config import get_settings
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_settings_cache():
|
|
"""Ensure each test starts with a fresh Settings cache.
|
|
|
|
Tests that mutate env vars and call `get_settings()` must also clear the
|
|
`lru_cache` themselves; this fixture is a safety net that runs after every
|
|
test so a cached `Settings` from one test never leaks into the next.
|
|
"""
|
|
yield
|
|
get_settings.cache_clear()
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _disable_request_throttle(monkeypatch):
|
|
"""Zero out the conservative request-pace settings for the test suite.
|
|
|
|
Production defaults (``request_min_interval=1.0``, ``request_jitter=0.8``)
|
|
would make every mocked request sleep ~1-1.8s and balloon the suite's
|
|
runtime. In tests we want the *behaviour* (header stealth, backoff, cap)
|
|
without the wall-clock cost, so we pin the throttle env vars to 0. The
|
|
defaults themselves are still asserted directly via ``Settings`` field
|
|
introspection in ``test_config.py``. The cache is cleared on either side
|
|
so the zeroed env is picked up the first time a test reads settings.
|
|
"""
|
|
get_settings.cache_clear()
|
|
monkeypatch.setenv("REQUEST_MIN_INTERVAL", "0")
|
|
monkeypatch.setenv("REQUEST_JITTER", "0")
|
|
yield
|
|
get_settings.cache_clear()
|