Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(route/nytimes): enhance full text fetch #17292

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

dzx-dzx
Copy link
Contributor

@dzx-dzx dzx-dzx commented Oct 24, 2024

Involved Issue / 该 PR 相关 Issue

Close #

Example for the Proposed Route(s) / 路由地址示例

/nytimes/en

New RSS Route Checklist / 新 RSS 路由检查表

  • New Route / 新的路由
  • Anti-bot or rate limit / 反爬/频率限制
    • If yes, do your code reflect this sign? / 如果有, 是否有对应的措施?
  • Date and time / 日期和时间
    • Parsed / 可以解析
    • Correct time zone / 时区正确
  • New package added / 添加了新的包
  • Puppeteer

Note / 说明

@github-actions github-actions bot added the Route label Oct 24, 2024
});
await page.goto(link, {
waitUntil: 'domcontentloaded',
if (request.url().includes('https://www.nytimes.com/svc/onsite-messaging/query') || request.url().includes('https://meter-svc.nytimes.com/meter.js')) {

Check failure

Code scanning / CodeQL

Incomplete URL substring sanitization High

'
https://www.nytimes.com/svc/onsite-messaging/query
' can be anywhere in the URL, and arbitrary hosts may come before or after it.
@github-actions github-actions bot added the Auto: Route Test Complete Auto route test has finished on given PR label Oct 24, 2024
Copy link
Contributor

Successfully generated as following:

http://localhost:1200/nytimes/en - Success ✔️
<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>纽约时报中文网 - 英文原版</title>
    <link>https://cn.nytimes.com</link>
    <atom:link href="http://localhost:1200/nytimes/en" rel="self" type="application/rss+xml"></atom:link>
    <description>纽约时报中文网 - 英文原版 - Powered by RSSHub</description>
    <generator>RSSHub</generator>
    <webMaster>contact@rsshub.app (RSSHub)</webMaster>
    <language>en</language>
    <lastBuildDate>Thu, 24 Oct 2024 17:34:11 GMT</lastBuildDate>
    <ttl>5</ttl>
    <item>
      <title>「英」</title>
      <description></description>
      <link>https://www.nytimes.com/2024/10/23/world/asia/modi-xi-putin-brics.html</link>
      <guid isPermaLink="false">https://www.nytimes.com/2024/10/23/world/asia/modi-xi-putin-brics.html</guid>
      <pubDate>Thu, 24 Oct 2024 04:46:24 GMT</pubDate>
    </item>
    <item>
      <title>「英」iPhone为何中国制造?</title>
      <description></description>
      <link />
      <guid isPermaLink="false">「英」iPhone为何中国制造?</guid>
      <pubDate>Thu, 24 Oct 2024 02:36:02 GMT</pubDate>
    </item>
    <item>
      <title>「英」</title>
      <description></description>
      <link>https://www.nytimes.com/2024/10/13/upshot/trump-black-hispanic-voters-harris.html</link>
      <guid isPermaLink="false">https://www.nytimes.com/2024/10/13/upshot/trump-black-hispanic-voters-harris.html</guid>
      <pubDate>Wed, 23 Oct 2024 20:36:19 GMT</pubDate>
    </item>
  </channel>
</rss>

Copy link
Contributor

Successfully generated as following:

http://localhost:1200/nytimes/en - Failed ❌
HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Failed to launch the browser process! undefined
[70:85:1025/130646.379152:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[70:70:1025/130646.380015:ERROR:ozone_platform_x11.cc(243)] Missing X server or $DISPLAY
[70:70:1025/130646.380024:ERROR:env.cc(257)] The platform failed to initialize.  Exiting.


TROUBLESHOOTING: https://pptr.dev/troubleshooting
Route: /nytimes/:lang?
Full Route: /nytimes/en
Node Version: v22.10.0
Git Hash: ed45c15f

Copy link
Contributor

Successfully generated as following:

http://localhost:1200/nytimes/en - Failed ❌
HTTPError: Response code 503 (Service Unavailable)

Error Message:<br/>Error: Failed to launch the browser process! undefined
[71:86:1025/133404.490503:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[71:71:1025/133404.491328:ERROR:ozone_platform_x11.cc(243)] Missing X server or $DISPLAY
[71:71:1025/133404.491339:ERROR:env.cc(257)] The platform failed to initialize.  Exiting.


TROUBLESHOOTING: https://pptr.dev/troubleshooting
Route: /nytimes/:lang?
Full Route: /nytimes/en
Node Version: v22.10.0
Git Hash: a84437fd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto: Route Test Complete Auto route test has finished on given PR core enhancement Route
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant