feat: RSA login encryption + confirmed wechat type codes
Replace the Phase-0 placeholder password encryption with real RSA
(PKCS#1 v1.5). encrypt_password now builds an RSA public key from the
getSecureKey modulus/exponent (RSAPublicNumbers, e=int(hex,16),
n=int(hex,16)), encrypts the UTF-8 password, and returns hex (documented
assumption -- base64 is the one-line fallback noted in the docstring).
get_secure_key now returns the inner data dict to match the new contract;
pc_login keeps {account, password} with a comment on the username/password
and account/pwd fallbacks if pcLogin rejects.
Extend normalize_wechat_type with the Phase-0 confirmed codes:
2 -> image, 3 -> image_legacy, 4 -> video, 10 -> system (1 -> text and
-2 -> divider unchanged; 6/9/13/15/21/22 stay unknown until confirmed).
Tests:
- encrypt_password round-trip: generate an RSA keypair, build secure_key
from its public numbers, encrypt, then decrypt with the private key and
assert the original password comes back. Pins 512-hex output for a
2048-bit key. No network.
- Parser parametrize table extended for the new codes (and a downstream
message-fixture assertion updated from unknown -> image_legacy).
Suite: 87 -> 97 passing. ruff check clean.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -4,12 +4,12 @@ The yunke CRM uses opaque integer type codes (``type`` / ``messageType``) on
|
||||
WeChat message records. The storage layer prefers stable enum strings so that
|
||||
media dispatch (image / voice / video / file) and presentation (text vs.
|
||||
date divider) can be keyed off a single normalized value rather than the
|
||||
raw code, whose exact mapping for non-text codes is a Phase-0 unknown.
|
||||
raw code.
|
||||
|
||||
This module is intentionally tiny: one total function that maps the
|
||||
documented codes and falls through to ``"unknown"`` for everything else.
|
||||
It MUST NOT raise — every record pulled during a scrape passes through here,
|
||||
and one unexpected code must not abort the whole pull. The raw code is
|
||||
Phase-0 confirmed codes and falls through to ``"unknown"`` for everything
|
||||
else. It MUST NOT raise — every record pulled during a scrape passes through
|
||||
here, and one unexpected code must not abort the whole pull. The raw code is
|
||||
preserved on the record's ``raw_json`` so the persistence layer can recover
|
||||
it when needed.
|
||||
"""
|
||||
@@ -20,26 +20,40 @@ from typing import Any
|
||||
|
||||
__all__ = ["normalize_wechat_type"]
|
||||
|
||||
# Documented WeChat message type codes (see Task 1.6 spec).
|
||||
# * ``1`` — plain text (and sticker text). The only non-media type we map
|
||||
# today; everything else falls through to ``"unknown"``.
|
||||
# Phase-0 confirmed WeChat message type codes (see
|
||||
# docs/superpowers/specs/phase0-findings.md §2). Captured by live inspection
|
||||
# of the yunke CRM.
|
||||
# * ``1`` — plain text (and sticker text).
|
||||
# * ``-2`` — date divider. NOT a real message; the scraper uses this label as
|
||||
# its drop signal (filtered out, never yielded, never persisted).
|
||||
# Image / voice / video / file / transfer / redpacket / system codes are
|
||||
# observed in the wild but their exact integer values are a Phase-0 unknown.
|
||||
# When confirmed, extend this table — for now they pass through as
|
||||
# ``"unknown"`` so the message is still ingested rather than dropped.
|
||||
# * ``2`` — image / sticker (webp). ``file`` / ``fileTh`` are direct URLs;
|
||||
# download + dedupe by ``md5``.
|
||||
# * ``3`` — image, legacy format. ``file`` is an opaque ID (not a URL) and
|
||||
# needs a separate download endpoint; flagged distinctly so the media layer
|
||||
# can route it.
|
||||
# * ``4`` — video (mp4). ``file`` is a direct URL.
|
||||
# * ``10`` — system / link / card / invite. ``file`` is a support URL (non-media);
|
||||
# the media layer skips downloading.
|
||||
# Other codes (``6``, ``9``, ``13``, ``15``, ``21``, ``22`` — voice / file /
|
||||
# transfer / redpacket / quote / etc.) are observed but unconfirmed; they pass
|
||||
# through as ``"unknown"`` so the message is still ingested (raw code kept on
|
||||
# ``raw_json``) rather than dropped. Extend this table as each is confirmed.
|
||||
_WECHAT_TYPE_NAMES: dict[int, str] = {
|
||||
1: "text",
|
||||
-2: "divider",
|
||||
2: "image",
|
||||
3: "image_legacy",
|
||||
4: "video",
|
||||
10: "system",
|
||||
}
|
||||
|
||||
|
||||
def normalize_wechat_type(raw: Any) -> str:
|
||||
"""Map an upstream WeChat ``type`` / ``messageType`` code to a stable string.
|
||||
|
||||
Returns ``"text"`` for code ``1``, ``"divider"`` for code ``-2``, and
|
||||
``"unknown"`` for any other / missing / unparsable value.
|
||||
Confirmed mappings: ``1→text``, ``-2→divider``, ``2→image``,
|
||||
``3→image_legacy``, ``4→video``, ``10→system``. Any other / missing /
|
||||
unparsable value returns ``"unknown"``.
|
||||
|
||||
Never raises: an unexpected type must not break the scrape. The raw code
|
||||
is preserved on ``raw_json`` so the persistence layer can recover it.
|
||||
|
||||
@@ -6,8 +6,8 @@ Public surface:
|
||||
company, cookies snapshot, raw payload).
|
||||
* :func:`app.session.login.get_secure_key` — ``GET /newuc/login/getSecureKey``.
|
||||
* :func:`app.session.login.pc_login` — ``POST /usercenter/login/pcLogin``.
|
||||
* :func:`app.session.login.encrypt_password` — Phase-0 swappable password
|
||||
encryption (see file for the open TODO).
|
||||
* :func:`app.session.login.encrypt_password` — RSA (PKCS#1 v1.5) password
|
||||
encryption using the ``getSecureKey`` modulus/exponent.
|
||||
* :class:`app.session.yunke_client.YunkeClient` — the authenticated HTTP
|
||||
client every scraper uses.
|
||||
"""
|
||||
|
||||
@@ -1,37 +1,42 @@
|
||||
"""yunke CRM login flow: ``getSecureKey`` -> ``pcLogin``.
|
||||
|
||||
Real-API facts (from inspection — see design doc §6.1)
|
||||
Real-API facts (from Phase-0 reconnaissance — see
|
||||
``docs/superpowers/specs/phase0-findings.md`` §1)
|
||||
------------------------------------------------------
|
||||
* ``GET /newuc/login/getSecureKey`` kicks off the login chain.
|
||||
* ``GET /newuc/login/getSecureKey`` returns the RSA public key as
|
||||
``{"data": {"public_exponent": "10001", "modulus": "<hex>"}}``.
|
||||
* The password is **RSA-encrypted (PKCS#1 v1.5)** with that public key
|
||||
(exponent ``0x10001`` = 65537, modulus = the returned hex) — see
|
||||
:func:`encrypt_password`.
|
||||
* ``POST /usercenter/login/pcLogin`` exchanges credentials for a session.
|
||||
* After login every data request to ``crm2.yunkecn.com`` carries the headers
|
||||
``userid`` / ``company`` / ``source=yunkepc`` / ``channel=4`` /
|
||||
``content-type=application/json`` plus the login session cookie (handled
|
||||
by the httpx client's cookie jar, not by this module).
|
||||
|
||||
Phase-0 unknowns (clearly marked)
|
||||
---------------------------------
|
||||
The exact ``pcLogin`` request body, the password encryption scheme (almost
|
||||
certainly RSA using the modulus/exponent from ``getSecureKey``), and the
|
||||
precise response JSON shape were NOT captured. To keep the session layer
|
||||
end-to-end runnable today:
|
||||
Remaining inferences (clearly marked)
|
||||
-------------------------------------
|
||||
The exact ``pcLogin`` request-body field names and the precise response JSON
|
||||
shape were not fully captured. To keep the session layer end-to-end runnable:
|
||||
|
||||
* :func:`encrypt_password` is a deterministic, documented placeholder. The
|
||||
real RSA implementation drops in here later without touching call-sites.
|
||||
* :func:`pc_login` sends ``{"account", "password"}`` (password = the RSA-hex).
|
||||
If ``pcLogin`` rejects, the next variants to try are ``{"username",
|
||||
"password"}`` / ``{"account", "pwd"}`` — see phase0-findings §1.
|
||||
* :func:`pc_login` parses ``userid`` / ``company`` out of a ``data`` object
|
||||
with several reasonable key-name variants. When the real shape is captured
|
||||
only this parser needs to change.
|
||||
|
||||
Tests mock both endpoints, so the placeholders are exercised end-to-end.
|
||||
Tests mock both endpoints, so the flow is exercised end-to-end.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from cryptography.hazmat.primitives.asymmetric import padding
|
||||
from cryptography.hazmat.primitives.asymmetric.rsa import RSAPublicNumbers
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data structures
|
||||
@@ -67,31 +72,46 @@ class LoginResult:
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Password encryption — Phase-0 placeholder
|
||||
# Password encryption — RSA (PKCS#1 v1.5), Phase-0 confirmed
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def encrypt_password(plain: str, secure_key: dict[str, Any]) -> str:
|
||||
"""Encrypt ``plain`` using the key material in ``secure_key``.
|
||||
"""RSA-encrypt ``plain`` (PKCS#1 v1.5) and return the ciphertext as hex.
|
||||
|
||||
Phase-0 TODO: real RSA encryption per ``getSecureKey``.
|
||||
``secure_key`` is the parsed ``data`` object from
|
||||
``GET /newuc/login/getSecureKey`` (Phase-0 confirmed shape)::
|
||||
|
||||
The browser-side crypto was not captured. Based on the endpoint name
|
||||
(``getSecureKey``) and the typical Chinese-CRM login idiom, the real
|
||||
implementation almost certainly RSA-encrypts the password using a
|
||||
modulus/exponent (or PEM public key) returned in ``secure_key`` and
|
||||
then base64-encodes the ciphertext. Until that is confirmed by a real
|
||||
capture, this function returns base64(UTF-8(plain)) so the login flow
|
||||
is end-to-end runnable and deterministic for tests.
|
||||
{"public_exponent": "10001", "modulus": "<hex>"}
|
||||
|
||||
Drop-in contract for the real implementation:
|
||||
def encrypt_password(plain: str, secure_key: dict[str, Any]) -> str:
|
||||
# 1. Pull modulus/exponent (or PEM) out of secure_key.
|
||||
# 2. RSA-OAEP (or PKCS#1 v1.5) encrypt UTF-8(plain).
|
||||
# 3. Return base64(ciphertext)."
|
||||
Both fields are hex strings. ``public_exponent`` is typically ``"10001"``
|
||||
(65537); ``modulus`` is the RSA modulus. An :class:`RSAPublicNumbers`
|
||||
public key is built from ``int(modulus, 16)`` / ``int(exponent, 16)``,
|
||||
used to PKCS#1 v1.5-encrypt UTF-8(plain), and the ciphertext is returned
|
||||
as a lowercase hex string.
|
||||
|
||||
Output-format assumption: **hex**. Phase-0 confirmed the public-key shape
|
||||
but not the ciphertext encoding; hex is the common Chinese-CRM convention
|
||||
and is what this implementation uses. If ``pcLogin`` ever rejects the
|
||||
body, base64 is the first alternative to try — both are one-line changes
|
||||
here (``ciphertext.hex()`` <-> ``base64.b64encode(ciphertext).decode()``).
|
||||
|
||||
Raises ``ValueError`` if ``secure_key`` is missing either field.
|
||||
"""
|
||||
# Phase 0 TODO: real RSA encryption per getSecureKey.
|
||||
return base64.b64encode(plain.encode("utf-8")).decode("ascii")
|
||||
exp_hex = secure_key.get("public_exponent")
|
||||
mod_hex = secure_key.get("modulus")
|
||||
if not exp_hex or not mod_hex:
|
||||
raise ValueError(
|
||||
"secure_key must contain 'public_exponent' and 'modulus' (hex"
|
||||
" strings) — got: "
|
||||
f"{secure_key!r}"
|
||||
)
|
||||
|
||||
e = int(str(exp_hex), 16)
|
||||
n = int(str(mod_hex), 16)
|
||||
public_key = RSAPublicNumbers(e, n).public_key()
|
||||
ciphertext = public_key.encrypt(plain.encode("utf-8"), padding.PKCS1v15())
|
||||
return ciphertext.hex()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -100,15 +120,23 @@ def encrypt_password(plain: str, secure_key: dict[str, Any]) -> str:
|
||||
|
||||
|
||||
async def get_secure_key(client: httpx.AsyncClient) -> dict[str, Any]:
|
||||
"""Call ``GET {base}/newuc/login/getSecureKey`` and return the JSON body.
|
||||
"""Call ``GET {base}/newuc/login/getSecureKey`` and return the key material.
|
||||
|
||||
The exact shape of the secure-key payload is a Phase-0 unknown; this
|
||||
function returns the whole decoded JSON so :func:`encrypt_password` (and
|
||||
eventually the real RSA step) can pick whatever fields it needs.
|
||||
Phase-0 confirmed return shape::
|
||||
|
||||
{"code": 10000, "data": {"public_exponent": "10001", "modulus": "<hex>"}}
|
||||
|
||||
Returns the inner ``data`` dict (the ``public_exponent`` / ``modulus`` key
|
||||
material) so callers — in particular :func:`encrypt_password` — receive
|
||||
the key directly. If the wrapper is ever absent, the whole payload is
|
||||
returned as a defensive fallback.
|
||||
"""
|
||||
resp = await client.get("/newuc/login/getSecureKey")
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
payload = resp.json()
|
||||
if isinstance(payload, dict) and isinstance(payload.get("data"), dict):
|
||||
return payload["data"]
|
||||
return payload
|
||||
|
||||
|
||||
async def pc_login(
|
||||
@@ -119,17 +147,22 @@ async def pc_login(
|
||||
) -> LoginResult:
|
||||
"""Call ``POST {base}/usercenter/login/pcLogin`` and parse a LoginResult.
|
||||
|
||||
Phase-0 assumptions about the request body (not captured):
|
||||
``{"account": <username>, "password": <encrypt_password(...)>}``.
|
||||
Real key names (``account`` vs ``loginName`` vs ``username``) and any
|
||||
additional required fields (captcha, deviceId, ...) are to be confirmed;
|
||||
only this function needs to change when they are.
|
||||
Request body inference (field names NOT yet captured — see
|
||||
phase0-findings §1)::
|
||||
|
||||
Phase-0 assumptions about the response (not captured): JSON body with a
|
||||
``data`` object containing ``userid`` + ``company``. Several reasonable
|
||||
key-name variants are tried (see :func:`_pick`). The session cookie is
|
||||
read back from the httpx client's cookie jar — ``pcLogin``'s
|
||||
``Set-Cookie`` lands there automatically.
|
||||
{"account": <username>, "password": <encrypt_password(...)>}
|
||||
|
||||
where ``password`` is the RSA-hex from :func:`encrypt_password`. If
|
||||
``pcLogin`` rejects this, the next variants to try are
|
||||
``{"username", "password"}`` or ``{"account", "pwd"}``; any additional
|
||||
required fields (captcha, deviceId, ...) land here too. Only this
|
||||
function changes when the real shape is captured.
|
||||
|
||||
Response inference (NOT yet captured): JSON body with a ``data`` object
|
||||
containing ``userid`` + ``company``. Several reasonable key-name variants
|
||||
are tried (see :func:`_pick`). The session cookie is read back from the
|
||||
httpx client's cookie jar — ``pcLogin``'s ``Set-Cookie`` lands there
|
||||
automatically.
|
||||
"""
|
||||
body = {
|
||||
"account": username,
|
||||
|
||||
Reference in New Issue
Block a user