2y ago

Is it ok (ethical) to scrape and display full-page content when blog author did not include in it RSS?

Is it ok (ethical) to scrape and display full-page content when blog author did not include in it RSS? | Lobsters

Hi everyone, I’m seeking advice and opinions. I’m building a web-based RSS reader/search engine/discovery tool. Like any RSS reader, my app fetches content from feeds and displays it to subscribers. Often, blog authors include only a short summary to the RSS, and the user has to visit the blog website to read the full content. My app also attempts to scrape the full webpage of the blog post for search indexing purposes (respecting

<code>

robots.txt

</code>

, of course). It also saves the HTML content for archiving purposes, like Internet Archive (if the author disallows

<code>

ia_archiver

</code>

user agent, I also honour that and don’t archive). So, since the app might already store the full content, my dilemma now is whether it’s ok (ethical) to show the full article in my reader? This view is never public, so only registered users who subscribe to the blog can see it. But still, it feels wrong, because it’s not even like browser’s “reader mode” — the user does not visit the original page at all. Not ok because:

Authors who only include a short summary in the RSS do so precisely because they want readers to visit their website.
Visiting the original blog is a much more personal experience than reading all blogs from the same UI of the reader app; bloggers craft their digital gardens for visitors!
Some blogs include styles, math, scripts, etc. which aren’t rendered correctly elsewhere after scraping.

Ok because:

It’s a nicer UX for the reader?

Curious what others think.