Files
xhs_crawler_claude/app/__pycache__/scraper.cpython-311.pyc
T

194 lines
37 KiB
Plaintext
Raw Normal View History

2026-01-12 18:58:23 +08:00
§
2026-01-12 19:07:00 +08:00
gÔdi4aãó¸—dZddlZddlZddlZddlZddlmZddlmZmZm Z m
2026-01-12 18:58:23 +08:00
Z
ddl m Z m
Z
mZmZmZddlmZddlmZGd„d e¦«ZGd
„d ¦«ZdS) uL
XiaoHongShu Scraper - 核心采集逻辑
2026-01-12 19:07:00 +08:00
支��线程和多线程模�
2026-01-12 18:58:23 +08:00
éN)Údatetime)ÚListÚDictÚOptionalÚ
AsyncIterator)Úasync_playwrightÚBrowserÚBrowserContextÚPageÚ TimeoutError)Úconfig)Ú
LogManagercó(‡—eZdZdZdefˆfd„ ZˆxZS)ÚRiskControlExceptionu 风控异常Úreasoncó^•—||_t¦« d|›�¦«dS)Nõ触å�‘风控: )rÚsuperÚ__init__)ÚselfrÚ __class__s €ú/tmp/tmpdw5zp1n_/app/scraper.pyrzRiskControlException.__init__s1ø€ØˆŒ Ý
‰Œ×ÒÐ2¨&Ð2Ð2Ñ3Ô3Ð3Ð3Ð3ó)Ú__name__Ú
__module__Ú __qualname__Ú__doc__ÚstrrÚ
__classcell__)rs@rrrsJø€€€€€ØÐð4˜sð4ð4ð4ð4ð4ð4ð4ð4ð4ð4rrc
ó6—eZdZdZd1dejdefd„Zd„Zd„Z d1de
d e
d
e e
fd „Z d „Z
d2de
de
ded
e e
fd„Zd1d
e
de
de e
fd„Zdedefd„Zdedefd„Zdedeee
ffd„Zd1ded
e e
fd„Zd1ded
e e
deefd„Zd1ded
e e
de efd„Zde
d ee
defd!„Zde
d"eefd#„Zd3dede
d
e
defd%„Zd&ee
fd'„Z ded(ejd
e
fd)„Z!d4d&ee
d+efd,„Z"d-„Z#d5d&ee
d/e
d+efd0„Z$dS)6ÚXiaoHongShuScraperuå°�红书采集器NÚ event_queueÚ log_managercóL—||_||_||_d|_d|_dS)NF)r"r#ÚbrowserÚ
is_runningÚ_stop_requested)rr"r#r%s rrzXiaoHongShuScraper.__init__s.€Ø&ˆÔØ&ˆÔ؈Œ ؈ŒØ$ˆÔÐÐrcƒó*K—|js‰t¦« ¦«ƒd{V—†|_|jj t jt j¬¦«ƒd{V—†|_|  dd¦«ƒd{V—†dSdS)u6åˆ�始化æµ�览器(如果未传入æµ�览器实例)N)ÚheadlessÚargsÚSUCCESSuæµ�览器åˆ�始化完æˆ�)
r%rÚstartÚ
playwrightÚchromiumÚlaunchr
ÚHEADLESSÚ BROWSER_ARGSÚemit_log©rs rÚ init_browserzXiaoHongShuScraper.init_browser!sÂèè€àŒ|ð GÝ$4Ñ$6Ô$6×$<Ò$<Ñ$>Ô$>Ð>Ð>Ð>Ð>Ð>Ð>ˆDŒOØ!%¤Ô!9×!@Ò!@ÝœÝÔ(ð"Añ"ô"ððððððˆDŒLð—-’-  Ð+EÑFÔFÐ FÐ FÐ FÐ FÐ FÐ FÐ FÐ FÐ Fð
 Gð GrcƒóÚK—|jr|j ¦«ƒd{V—†|jr|j ¦«ƒd{V—†| dd¦«ƒd{V—†dS)u关闭æµ�览器NÚINFOuæµ�览器已关闭)r%Úcloser-Ústopr2r3s rÚ
close_browserz XiaoHongShuScraper.close_browser+s“èè€à Œ<ð 'Ø”,×$Ò$Ñ&Ô&Ð &Ð &Ð &Ð &Ð &Ð &Ð &Ø Œ?ð )Ø”/×&Ò&Ñ(Ô(Ð (Ð (Ð (Ð (Ð (Ð (Ð (Ø�mŠm˜FÐ$8Ñ9Ô9Ð9Ð9Ð9Ð9Ð9Ð9Ð9Ð9Ð9rÚlevelÚmessageÚ thread_idcƒóŠK—|j |||¦«}|j ddi|¥¦«ƒd{V—†dS)uå�‘é€�日志事件ÚtypeÚlogN)r#Úappendr"Úput)rr:r;r<Ú log_entrys rr2zXiaoHongShuScraper.emit_log3syèè€ðÔ$×+Ò+¨E°7¸IÑFÔFˆ ðÔ×"Ò"Ø �Eð$
àð$
ñ ô ð ð ð ð ð ð ð ð ð rc‹óRK—|j ddi|¥¦«ƒd{V—†dS)uå�‘é€�进度更新r>ÚprogressN©r"rA)rÚkwargss rÚ
emit_progressz XiaoHongShuScraper.emit_progress>s[èè€àÔ×"Ò"Ø �Jð$
àð$
ñ ô ð ð ð ð ð ð ð ð ð rrÚuser_idÚstatusÚ
note_countcƒóVK—|j d||||dœ¦«ƒd{V—†dS)uå�‘é€�用户状æ€�Ú user_status)r>rHrIrJr<NrE)rrHrIrJr<s rÚemit_user_statusz#XiaoHongShuScraper.emit_user_statusEsbèè€àÔ×"Ò"Ø!ØØØ$Ø"ð $
ð$
ñ ô ð ð ð ð ð ð ð ð ð rÚ current_usercƒóTK—|j d|||dœ¦«ƒd{V—†dS)uå�‘é€�线程状æ€�Ú
thread_status)r>r<rIrNNrE)rr<rIrNs rÚemit_thread_statusz%XiaoHongShuScraper.emit_thread_statusOs_èè€àÔ×"Ò"Ø#Ø"ØØ(ð $
ð$
ñ ô ð ð ð ð ð ð ð ð ð rÚcontextÚreturncƒóæK—| ¦«ƒd{V—†}| tj¦«ƒd{V—†| dd¦«ƒd{V—†| dd¦«ƒd{V—† | tj›d�tjdz¬¦«ƒd{V—†| d d
¦«ƒd{V—†| tj ¬ ¦«ƒd{V—†| dd ¦«ƒd{V—†| 
¦«ƒd{V—†d
S#t$r:| dd¦«ƒd{V—†| 
¦«ƒd{V—†YdSwxYw)u<
处�登录
返回是�已登录
NÚWARNINGu'请在æµ�览器中完æˆ�扫ç �登录...r6u等待登录确认...z/**éè©Útimeoutr+u登录æˆ�功ï¼�)Úpathu登录状æ€�å·²ä¿�å­˜TÚERRORu登录超时,请é‡�试F) Únew_pageÚgotor
Ú LOGIN_URLr2Ú wait_for_urlÚXIAOHONGSHU_BASE_URLÚ
LOGIN_TIMEOUTÚ
storage_stateÚAUTH_STATE_FILEr7ÚPlaywrightTimeoutError)rrRÚpages rÚ handle_loginzXiaoHongShuScraper.handle_loginXsèè€ð
×%Ò%Ñ'Ô'Ð'Ð'Ð'Ð'Ð'Ð'ˆØ�iŠi�Ô(Ñ)Ô)Ð)Ð)Ð)Ð)Ð)Ð)Ð)à�mŠm˜IÐ'PÑQÔQÐQÐQÐQÐQÐQÐQÐQØ�mŠm˜FÐ$;Ñ<Ô<Ð<Ð<Ð<Ð<Ð<Ð<Ð<ð à×#Ò#ÝÔ.Ð3Ð3Ð3ÝÔ,¨tÑ3ð$ñôð
ð
ð
ð
ð
ð
ð
ð—-’-  Ð+<Ñ=Ô=Ð =Ð =Ð =Ð =Ð =Ð =Ð =ð×'Ò'­VÔ-CÐ'ÑDÔDÐ DÐ DÐ DÐ DÐ DÐ DÐ DØ—-’- Ð(?Ñ@Ô@Ð @Ð @Ð @Ð @Ð @Ð @Ð @à—*’*‘,”,Ð Ð Ð Ð Ð Ð Ð Ø�4øå%ð ð ð Ø—-’- Ð)CÑDÔDÐ DÐ DÐ DÐ DÐ DÐ DÐ DØ—*’*‘,”,Ð Ð Ð Ð Ð Ð Ð Ø�5�5ð øøøsÁ;B/D,Ä,AE0Å/E0Útarget_contextÚsource_contextcƒótK—| ¦«ƒd{V—†}| |¦«ƒd{V—†dS)u å¤�制 Cookies 到目标 ContextN)ÚcookiesÚ add_cookies)rrfrgris rÚ copy_cookieszXiaoHongShuScraper.copy_cookiesxsUèè€à&×.Ò.Ñ0Ô0Ð0Ð0Ð0Ð0Ð0Ð0ˆØ×(Ò(¨Ñ1Ô1Ð1Ð1Ð1Ð1Ð1Ð1Ð1Ð1Ð1rrdcƒóÎK— | ¦«ƒd{V—†}tjD]}||vr dd|›�fcSŒdS#t$r}ddt |¦«›�fcYd}~Sd}~wwxYw)u\
检测页�是�触�风控
返回 (是�触�, 触�原因)
NTu检测到关键è¯�: )FÚFu检测失败: )Úcontentr
Ú
RISK_KEYWORDSÚ Exceptionr)rrdrnÚkeywordÚes rÚcheck_risk_controlz%XiaoHongShuScraper.check_risk_control}sµèè€ð
 4Ø ŸLšL™NœNÐ*Ð*Ð*Ð*Ð*Ð*ˆGÝ!Ô/ð
Bð
B�ؘgÐ%Ð%ØÐ!A¸Ð!AÐ!AÐAÐAÐAÐAð&à�9øÝð 4ð 4ð 4ØÐ3­3¨q©6¬6Ð3Ð3Ð3Ð 3Ð 3Ð 3Ð 3Ð 3Ð 3øøøøð 4øøøs„4<¹<¼
A$ÁAÁA$ÁA$cƒó<K—| dd|›d�|¬¦«ƒd{V—†d}d}|tjk�r=| d¦«ƒd{V—†}||kr#| dd|›d�|¬¦«ƒd{V—†nù|}t jtjtj¦«}| d ¦«ƒd{V—†}| d
||z›d �¦«ƒd{V—†t jtj tj
¦«}tj |¦«ƒd{V—†|d z
}|d
zdkr/| 
|¦«ƒd{V—†\} }
| rt|
¦«‚|tjk�°=| dd|›d|›d�|¬¦«ƒd{V—†dS)u3
模拟人类间歇性平滑滚动
ÚDEBUGÚ[u#] 开始平滑滚动加载笔记...©r<Nrzdocument.body.scrollHeightu] 已滚动到底部zwindow.scrollYz<
window.scrollTo({
top: zI,
behavior: 'smooth'
});
ééu] 滚动完æˆ�,共滚动 u 次)r2r
ÚMAX_SCROLL_COUNTÚevaluateÚrandomÚrandintÚSCROLL_STEP_MINÚSCROLL_STEP_MAXÚuniformÚSCROLL_PAUSE_MINÚSCROLL_PAUSE_MAXÚasyncioÚsleeprsr) rrdr<Ú scroll_countÚ last_heightÚcurrent_heightÚ scroll_stepÚscroll_positionÚ
pause_timeÚis_riskrs rÚhuman_like_scrollz$XiaoHongShuScraper.human_like_scroll‹sEèè€ð�mŠm˜GÐ%W¨Ð%WÐ%WÐ%WÐclˆmÑmÔmÐmÐmÐmÐmÐmÐmÐmàˆ Øˆ à�VÔ4Ò4Ñ4à#'§=¢=Ð1MÑ#NÔ#NÐNÐNÐNÐNÐNÐNˆNð Ò,Ð,Ø—m’m GÐ-P°Ð-PÐ-PÐ-PÐ\e�mÑfÔfÐfÐfÐfÐfÐfÐfÐfØà(ˆKõ!œ.­Ô)?ÅÔAWÑXÔXˆKØ$(§M¢MÐ2BÑ$CÔ$CÐCÐCÐCÐCÐCÐCˆOð—-’-ð!à)¨KÑ7ð!ð!ð!ñôð
ð
ð
ð
ð
ð
ð
õ œ­Ô(?ÅÔAXÑYÔYˆJÝ”- 
Ñ+Ô+Ð +Ð +Ð +Ð +Ð +Ð +Ð +à ˜AÑ ˆLð˜aÑ 1Ò$Ð$Ø(,×(?Ò(?ÀÑ(EÔ(EÐ"EÐ"EÐ"EÐ"EÐ"EÐ"E‘�˜Øð7Ý.¨vÑ6Ô6Ð6ðC�VÔ4Ò4Ñ4ðF�mŠm˜GÐ%a¨Ð%aÐ%aÈ|Ð%aÐ%aÐ%aÐmvˆmÑwÔwÐwÐwÐwÐwÐwÐwÐwÐwÐwrc
ƒó~K—| dd|›d�|¬¦«ƒd{V—†g} gd¢}g}|D]X} | |¦«ƒd{V—†}|r(|}| dd|›d|›�|¬¦«ƒd{V—†nŒI#t$rYŒUwxYw|D]µ}| d¦«ƒd{V—†} | r–d | vr’|  d
¦«rt
j›| ›�} | d ¦«ƒd{V—†}
|
r|
 ¦«ƒd{V—†nd } n#t$rd } YnwxYw|  | | dd
…dœ¦«Œ¶| dd|›dt|¦«›d�|¬¦«ƒd{V—†|S#t$r=} | dd|›dt| ¦«›�|¬¦«ƒd{V—†gcYd} ~ Sd} ~ wwxYw)uc
从用户主页��笔记�片
返回 [{'url': '...', 'title': '...'}]
rurvu] æ��å�–笔记å�¡ç‰‡...rwN)za[href*='/explore/']z+section[class*='note'] a[href*='/explore/']z,div[class*='note-item'] a[href*='/explore/']u] 使用选择器: Úhrefz /explore/ú/z span, divõ 无标题é2)ÚurlÚtitler6u ] æ��å�–到 õ
æ�¡ç¬”è®°rZu] æ��å�–笔记å�¡ç‰‡å¤±è´¥: ) r2Úquery_selector_allrpÚ
get_attributeÚ
startswithr
r_Úquery_selectorÚ
inner_textr@Úlenr)
rrdr<Ú note_urlsÚ selectorsÚlinksÚselectorÚelementsÚlinkr’Ú
title_elemr“rrs
rÚextract_note_cardsz%XiaoHongShuScraper.extract_note_cards¹sòèè€ð
�mŠm˜GÐ%K¨Ð%KÐ%KÐ%KÐW`ˆmÑaÔaÐaÐaÐaÐaÐaÐaÐaàˆ ð- ðððˆIð ˆEØ%ð
ð
�ðØ%)×%<Ò%<¸XÑ%FÔ%FÐFÐFÐFÐFÐFÐF�HØðØ (˜Ø"Ÿmšm¨GÐ5a¸Ð5aÐ5aÐW_Ð5aÐ5aÐmv˜mÑwÔwÐwÐwÐwÐwÐwÐwÐwؘðøõ!ðððØ�Hðøøøðð
ð
�Ø ×.Ò.¨vÑ6Ô6Ð6Ð6Ð6Ð6Ð6Ð6�Øð˜;¨#Ð-Ð-à—~’~ cÑ*Ô*ðDÝ!'Ô!<ÐC¸cÐCÐC˜ð,Ø+/×+>Ò+>¸{Ñ+KÔ+KÐ%KÐ%KÐ%KÐ%KÐ%KÐ%K˜
ØAKÐ \ j×&;Ò&;Ñ&=Ô&=Ð =Ð =Ð =Ð =Ð =Ð =Ð =ÐQ\˜˜øÝ$ð,ð,ð,Ø +˜˜˜ð,øøøð×$Ò$Ø"Ø!& s¨ s¤ð&ð&ñôðøð
—-’- Ð(]¨IÐ(]Ð(]Å3ÀyÁ>Ä>Ð(]Ð(]Ð(]Ðir�-ÑsÔsÐ sÐ sÐ sÐ sÐ sÐ sÐ sàÐ øåð ð ð Ø—-’- Ð)\¨YÐ)\Ð)\ÕTWÐXYÑTZÔTZÐ)\Ð)\Ðhq�-ÑrÔrÐ rÐ rÐ rÐ rÐ rÐ rÐ r؈IˆIˆIˆIˆIˆIøøøøð øøøsg¨
E5³AA9Á6E5Á9
BÂE5ÂBÂA
E5Ã9DÄ
E5Ä DÄE5ÄDÄAE5Å5
F<Å?2F7Æ1F<Æ7F<c
ƒóŒK— tjtjtjަ«ƒd{V—†| |¦«ƒd{V—†\}}|rt|¦«‚i} | tj ¦«ƒd{V—†}|r| 
¦«ƒd{V—†nd|d<n#t$rd|d<YnwxYw | tj ¦«ƒd{V—†}|r| 
¦«ƒd{V—†nd|d<n#t$rd|d<YnwxYw | tj
¦«ƒd{V—†}|r| 
¦«ƒd{V—†nd|d<n#t$rd|d<YnwxYw| ¦«ƒd{V—†} | | ddg¦«|d <| | gd
¢¦«|d <| | gd ¢¦«|d
<|j|d<|S#t$r‚t$r<}
| dd|›dt%|
¦«›�|¬¦«ƒd{V—†Yd}
~
dSd}
~
wwxYw)u0
从笔记详情页��数�
Nr�r“rmÚ publish_timeÚlocationu评论ÚcommentÚ
comment_count)u点赞u赞ÚlikeÚ
like_count)uæ”¶è—�ÚstarÚcollectÚ
collect_countÚnote_urlrZrvu] æ��å�–详情失败: rw)rƒr„r|r€r
ÚPAGE_LOAD_DELAYrsrr˜ÚSELECTOR_DETAIL_TITLEr™rpÚSELECTOR_DETAIL_PUBLISH_TIMEÚSELECTOR_DETAIL_LOCATIONrnÚ_extract_countr’r2r) rrdr<r‹rÚdatar¡Ú time_elemÚloc_elemrnrrs rÚextract_note_detailz&XiaoHongShuScraper.extract_note_detailñsAèè€ð5 å”-¥¤µÔ0FÐ GÑHÔHÐ HÐ HÐ HÐ HÐ HÐ HÐ Hð%)×$;Ò$;¸DÑ$AÔ$AÐAÐAÐAÐAÐAÐA‰OˆG�VØð
3Ý*¨6Ñ2Ô2Ð2àˆDð
,Ø#'×#6Ò#6µvÔ7SÑ#TÔ#TÐTÐTÐTÐTÐTÐT�
ØAKÐ \ j×&;Ò&;Ñ&=Ô&=Ð =Ð =Ð =Ð =Ð =Ð =Ð =ÐQ\��W‘
�
øÝð
,ð
,ð
,Ø +��W‘
�
�
ð
,øøøð
*Ø"&×"5Ò"5µfÔ6YÑ"ZÔ"ZÐZÐZÐZÐZÐZÐZ� ØGPÐ'X¨Y×-AÒ-AÑ-CÔ-CÐ'CÐ'CÐ'CÐ'CÐ'CÐ'CÐ'CÐVX��^Ñ$Ð$øÝð
*ð
*ð
*Ø')��^Ñ$Ð$Ð$ð
*øøøð
&Ø!%×!4Ò!4µVÔ5TÑ!UÔ!UÐUÐUÐUÐUÐUÐU�ØBJÐ#R¨×)<Ò)<Ñ)>Ô)>Ð#>Ð#>Ð#>Ð#>Ð#>Ð#>Ð#>ÐPR��ZÑ Ð øÝð
&ð
&ð
&Ø#%��ZÑ Ð Ð ð
&øøøð!ŸLšL™NœNÐ*Ð*Ð*Ð*Ð*Ð*ˆGð%)×$7Ò$7¸À(ÈIÐAVÑ$WÔ$WˆD�Ñ !ð"&×!4Ò!4°WÐ>WÐ>WÐ>WÑ!XÔ!XˆD�Ñ ð%)×$7Ò$7¸ÐA^ÐA^ÐA^Ñ$_Ô$_ˆD�Ñ !ð $œxˆD�Ñ àˆKøå#ð ð ð Ø Ýð ð ð Ø—-’- Ð)V¨YÐ)VÐ)VÍcÐRSÉfÌfÐ)VÐ)VÐbk�-ÑlÔlÐ lÐ lÐ lÐ lÐ lÐ lÐ lØ�4�4�4�4�4øøøøð øøøsˆ„A!G3Á&AB-Â,G3Â-B?Â<G3Â>B?Â?G3ÃAD
Ä G3Ä
DÄG3ÄDÄG3Ä AE'Å&G3Å'E9Å6G3Å8E9Å9A9G3Ç3IÈ1H>È>IrnÚkeywordsc óÊ—ddl}|D]Û}|›d�}| ||¦«}|r¼| d¦«}d|vr5tt | dd¦«¦«dz¦«cSd| ¦«vrGtt | ¦« dd¦«¦«d z¦«cSt|¦«cSŒÜdS)
u从页é�¢å†…容æ��å�–计数rNu[\s::]*([\d.]+)rxu万rmi'ÚkrV)ÚreÚsearchÚgroupÚintÚfloatÚreplaceÚlower)rrnr·rºrqÚpatternÚmatchÚ count_strs rr²z!XiaoHongShuScraper._extract_count,sü€àˆ ˆ ˆ Øð *ð *ˆGà Ð5Ð5Ð5ˆGØ—I’I˜g wÑ/Ô/ˆEØð
*Ø!ŸKšK¨™NœN� à˜IÐ%Ð%Ý�u Y×%6Ò%6°u¸bÑ%AÔ%AÑBÔBÀUÑJÑKÔKÐKÐKÐKؘIŸOšOÑ-Ô-Ð-Ð-Ý�u Y§_¢_Ñ%6Ô%6×%>Ò%>¸sÀBÑ%GÔ%GÑHÔHÈ4ÑOÑPÔPÐPÐPÐPå˜y™>œ>Ð)Ð)Ð)ð
*ðˆqrÚ new_notescƒóPK—tj tj|›d�¦«}g}tj |¦«rŒ t
|dd¬¦«5}tj|¦«}|  dg¦«}ddd¦«n #1swxYwYn6#t$r)}|  dd|›�¦«ƒd{V—†Yd}~nd}~wwxYwd „|D¦«}d
} |D]A}