Make `plugins` required

Changed

• Make plugins required and manually imported from dist/plugins

Handle Missing Header

Fixed

• Handle missing content-type header

Extract icons

Fixed

• Extract icons from webpages
• Add sailthru image support

Normalize `Snippet` Further

Changed

• Make single Snippet intersected type, use Entity types for object properties
• Define options explicitly for fetch (e.g. accept)

Fixed

• Improved headline extraction
• Decode HTML entities in JSON-LD
• Support @graph in JSON-LD to handle Yoast SEO plugin better

Rewrite Package, Remove Two Step Process

Changed

• Rewrite package to expose only a scrape method which does whatever it wants internally - the only requirement is on the output matching the defined TypeScript result interface
• Removed built-in HTTP request method, make it a required option to initialize the scraper
• Enable the ability to configure plugins

Refactor

Changed

• Everything, basically. The scrape result is similar, as ever, but the extracted result is much improved while how the pipeline actually runs has been overhauled to support middleware and plugins.

Extract `contentUrl`

Changed

• Return the final contentUrl as fallback when none exists

Raw Encoding

Changed

• Set encodingFormat to the raw response encoding

Response URL

Fixed

• Correctly re-use response URL (over original URL) when resolving local links

Icons

Added

• Extract icons from the HTML page
• Support JSON OEmbed resources

Fixed

• Correct encodingFormat for HTML pages
• Handle stream error when aborting the HTTP response