浅念
46e3880c0b
Centralize scraping stealth in YunkeClient.request() so automated traffic
resembles the normal CRM web app and is paced conservatively to avoid
triggering 云客 risk control (pulls M6 rate-limit work forward):
- app/config.py: add 6 tunable settings with conservative defaults
(request_min_interval=1.0, request_jitter=0.8, max_requests_per_run=0,
risk_backoff_{base=2.0, max=60.0, attempts=4}).
- app/scraper/base.py: RateLimiter gains optional jitter + a new wait()
method (fixed min+jitter per-call sleep); gap-based acquire() is unchanged
for backward compat (default jitter=0.0).
- app/session/yunke_client.py: every request now carries browser-identical
headers (UA/sec-ch-ua/accept/accept-language/referer) merged with the auth
headers (per-request overrides win); awaits the shared jittered limiter
before sending; enforces a per-run request cap (RequestCapReached); and
backs off exponentially on risk signals (HTTP 429 or 200 body carrying
频繁/风控/限制/稍后), distinct from the existing 401 relogin-once path.
- tests/conftest.py: zero the throttle env in the test suite so the
conservative prod defaults don't slow unit tests.
- tests/test_yunke_client_stealth.py: 9 new tests (browser headers,
override, limiter spy, 429 backoff, 200-marker backoff, persistent-429
raise, volume cap, unlimited, 401-relogin preserved). TDD red->green.
Suite: 171 -> 181 passing. ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
57 lines
2.5 KiB
Python
57 lines
2.5 KiB
Python
from functools import lru_cache
|
|
from pathlib import Path
|
|
|
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
|
|
|
|
|
class Settings(BaseSettings):
|
|
"""Application settings sourced from environment / .env file."""
|
|
|
|
model_config = SettingsConfigDict(
|
|
env_file=".env",
|
|
env_file_encoding="utf-8",
|
|
extra="ignore",
|
|
)
|
|
|
|
yunke_base_url: str = "https://crm2.yunkecn.com"
|
|
yunke_username: str = ""
|
|
yunke_password: str = ""
|
|
db_url: str = "mysql+asyncmy://user:pass@localhost:3306/yunke"
|
|
redis_url: str = "redis://localhost:6379/0"
|
|
media_dir: Path = Path("./media")
|
|
# Fernet key for encrypting stored credentials. Required for real runs;
|
|
# empty default lets tests construct Settings without a key.
|
|
encryption_key: str = ""
|
|
# Shared bearer token for the viewing API. Empty (default) -> auth DISABLED
|
|
# (we trust the network boundary; this is a local/intranet tool). When set,
|
|
# every API request must present this value via ``Authorization: Bearer
|
|
# <token>`` or ``?token=<token>``. See app/api/auth.py.
|
|
api_token: str = ""
|
|
|
|
# --- Anti-risk-control / stealth tuning ---------------------------------
|
|
# These are the knobs that pace automated scraping so it resembles the
|
|
# normal web app. Defaults are deliberately CONSERVATIVE (slow + jittered):
|
|
# roughly one request every ~1.0..1.8s. Loosen these only once we have
|
|
# evidence the CRM tolerates a faster pace. See app.session.yunke_client.
|
|
# Lower bound on seconds between successive upstream requests.
|
|
request_min_interval: float = 1.0
|
|
# Random extra 0..jitter seconds added on top of ``request_min_interval``
|
|
# per request, so the cadence is not a perfectly periodic signature.
|
|
request_jitter: float = 0.8
|
|
# Hard cap on the number of upstream requests a single client may send in
|
|
# one run. 0 = unlimited; >0 = enforce, then raise RequestCapReached.
|
|
max_requests_per_run: int = 0
|
|
# Exponential backoff parameters applied when the CRM signals risk control
|
|
# (HTTP 429 or a 200 body carrying a 风控/限流 marker). Delay for retry N
|
|
# (0-indexed) is ``risk_backoff_base * 2**N`` clamped to ``risk_backoff_max``.
|
|
risk_backoff_base: float = 2.0
|
|
risk_backoff_max: float = 60.0
|
|
# Number of retries attempted after the initial request before giving up.
|
|
risk_backoff_attempts: int = 4
|
|
|
|
|
|
@lru_cache
|
|
def get_settings() -> Settings:
|
|
"""Return the process-wide Settings singleton."""
|
|
return Settings()
|