13ft – A site similar to 12ft.io but self-hosted

Countercounterpoint - Maybe I have news subscriptions for periodicals I regularly read, but don't feel like paying for a monthly subscription to read one random article from some news outlet I don't regularly read that someone linked on social media or HN.

as much as I circumvent paywalls myself, it does feel like overkill to setup software to do it always. Sites spend money to produce quality content.

Somewhat related comparison, Is a human choosing to do this theft really better than a neural network scraping content for its own purposes?

Paying for it doesn’t make the site less miserable to use. One of the stupid things about piracy is that it tends to also be the best available version of the thing. You’re actively worse off having paid for it. (Ads, denial, DRM in general, MBs of irrelevant JS, etc don’t go away with money, but do with piracy)

I will never willingly give a journalist my money.

No. Paywalled content should not be indexed by search engines. The implicit contract I have with the search engine is that it is showing me things that I can see. The publishers and search engines pulled a bait and switch here by whitelisting googlebot. So it's fair game to view the publisher's website with googlebot. That's what the engineers spent their time working on. It would be unfair to let that go to waste.

Is there a way to pay for journalistic content that doesn't involve participating in the extensive tracking that those websites perform on their visitors?

I love to read the news but I don't love that the news reads me.

The Venn diagram of people who truly can't afford a New York times subscription, and even know what Docker is looks like two circles.

I agree, but would like for a way to pay for an article, or a single day, week, or month of access. Just like I could buy a single one-off issue of a publication a couple of times before starting a long term relationship with it. Not all publications support this, and some like the NY Times require chatting with a representative to cancel the subscription. I see a lot of talk about physical media around film and music, but not being able to buy single issues of any magazine or newspaper anonymously when the circumstances call for it, is a great loss for public discourse.

Pay walls don't get folks there, how ever noble the sellers of information they to brand it.

100%. And sometimes that form of payment is putting up with ads, etc. I routinely back out of sites that suddenly take over the screen with a popup or take up large chunks with video or animations. Same as opting not to go in a particular store. But I also stick around and occasionally use products advertised to me. Shocking, I know.

It would cost me about $500/month if I subscribe to every paywall that appears in front of me.

Counterpoint, meet me where I want to spend my money and I will. Not giving every publisher on the planet a sub.

I pay for the sites I visit regularly.

But when somebody shares an article with me and I want to see what I’ve been sent, I’m not going to buy a $15 monthly subscription to some small-town newspaper in Ohio just because they’ve decided to paywall their content in that way.

I'll do that as soon as one-click-to-cancel becomes law. I refuse to subject myself to predatory business practices so they won't see my money until a legislative body starts working on behalf of the people.

I fully agree with the sentiment! I support and do pay for sources I read frequently.

Sadly payment models are incompatible with how most people consume content – which is to read a small number of articles from a large number of sources.

As of 21 June 2023, there were 52,642 distinct sites submitted to the front page.

Counting those with 100 or more appearances, that falls to 149.

Doing a manual classification of news sites, there are 146.

Even at a modest annual subscription rate of $50/year ($1/week per source), that's a $7,300 subscriptions budget just to be able to discuss what's appearing on Hacker News from mainstream news sources.

Oh, and if you want per-article access at, say, $0.50 per article, that's $5,475 to read a year's worth of HN front-page submissions (10,950 articles/year), and that is just based on what is captured on the archive. In practice far more articles will appear, if only briefly, on the front page each day.

Which is among the reasons I find the "just subscribe" argument untenable. Some sort of bundling payment arrangement is required.

<https://news.ycombinator.com/item?id=36832354>

My alternative suggestion is global access to content through a universal content syndication tax, or fee assessed through ISPs, on a progressive basis. See:

"Why won't (some) people pay for the news?" (2022)

<https://diaspora.glasswings.com/posts/867c94d0ba87013aca4144...>

Discussed 4 days ago on HN: <https://news.ycombinator.com/item?id=41249571>

I know of very few sites that let you pay to get zero ads or affiliate links. The ones that let you pay still show you affiliate links.

I wasn't even thinking about paywalls, the first thing I did was check to see if cookie banners and "Sign in with Google" popups went away. There's so many user-unfriendly things that you constantly deal with, any amount of browsing is just a bad experience without putting up defenses like this.

why pay a monthly subscription if we're going to be bombarded by legally required popups and other internal promotional stuff that hooks you to the site anyway?

set the user agent header to the googlebot one

Also, how effective is this really? Don’t the big news sites check the IP address of the user agents that claim to be GoogleBot?

Personally I find it nice for sending articles to friends.

If this is all it's doing then you could also just use this extension: https://requestly.com/

Create a rule to replace user agent with "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I just tried it and seems to work.

I always do DevTools -> Network Conditions to set UA, at least in Chrome.

It says it blocks ads and other things, too. I imagine the use case is someone wanting this for multiple devices/people so they don't have to set up an extension for every platform/device individually. I have no idea how effective it is

I use this extension which has a decent UI: https://webextension.org/listing/useragent-switcher.html

This does not work anymore?

https://addons.mozilla.org/en-US/firefox/addon/random_user_a...

"The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access."

It appears anyone can read any new NYT article in the Internet Archive. I use a text-only browser. I am not using Javascript. I do not send a User-Agent header. Don't take my word for it. Here is an example:

https://web.archive.org/web/20240603124838if_/https://www.ny...

If I am not mistaken the NYT recently had their entire private Github repository, a very large one, made public without their consent. This despite the staff at the NYT being "well-versed" in whatever it is the HN commenter thinks they are well versed in.

They probably asynchronously verify that the IP address actually belongs to googlebot, then ban the IP when it fails.

Synchronously verifying it, would probably be too slow.

You can verify googlebot authenticity by doing a reverse dns lookup, then checking that reverse dns name resolves correctly to the expected IP address[0].

[0]: https://developers.google.com/search/docs/crawling-indexing/...

FWIW, if you happen to be based in the U.S., you might find that your local public library provides 3-day NYT subscriptions free of charge, which whilst annoying is probably easier than fighting the paywall. Of course this only applies to the NYT.

We need a P2P swarm for content. Just like Bittorrent back in the day. Pop in your favorite news article (or paraphrase it yourself), and everyone gets it.

With recommender systems, attention graph modeling, etc. it'd probably be a perfect information ingestion and curation engine. And nobody else could modify the algorithm on my behalf.

i tried to access my own website and it says internal server error. i also tried to access Youtube and it said the same.

Some sites do work, but others such as WSJ just give a blank page. Worse, Economist actively blocks this through Cloudflare.

That's the topic of an essay I'd written a couple of years ago and just discussed on HN last week:

"Why won't (some) people pay for the news?"

<https://diaspora.glasswings.com/posts/867c94d0ba87013aca4144...>

<https://news.ycombinator.com/item?id=41249571>

Upshot: Numerous legitimate reasons. I'd very much like to see universal syndication / superbundling on a distributed progressive tax and/or ISP tollkeeper basis, with some additional details for the fiddly bits.

As for subscribing to content submitted to general discussion sites, such as HN or others:

As of 21 June 2023, there were 52,642 distinct sites submitted to the HN front page.

Counting those with 100 or more appearances, that falls to 149.

Doing a manual classification of news sites, there are 146.

Even at a modest annual subscription rate of $50/year ($1/week per source), that's a $7,300 subscriptions budget just to be able to discuss what's appearing on Hacker News from mainstream news sources.

Oh, and if you want per-article access at, say, $0.50 per article, that's $5,475 to read a year's worth of HN front-page submissions (10,950 articles/year), and that is just based on what is captured on the archive. In practice far more articles will appear, if only briefly, on the front page each day.

Which is among the reasons I find the "just subscribe" argument untenable. Some sort of bundling payment arrangement is required.

<https://news.ycombinator.com/item?id=36832354>

(Source: own work based on scraping the HN front-page archive from initiation through June 2023.)

Is there some way to support journalism across publications?

Apple News+, Inkl, PressReader, (maybe more). Others if you want more magazine-focused subscriptions.

Articles should come with license agreements, just like open source software nowadays. Free for personal entertainment, but if you try to make money from the information in the article or otherwise commercialize it, you can fuck right off.

This attitude is why journalism is dying. There is value to an undissected payment to the publisher that gives them revenue surety and lets them fund a variety of reporting, even if you don't personally find it interesting or relevant. This is exactly how journalism x publishing worked with the "rivers of gold" from classifieds/advertising during the golden age (also: this is exactly how Bell Labs, Google 20% time, etc were/are funded. The incessant need to pay for only the small portion you directly consume/find interesting kills this sort of thing).

While this is generally true for legacy publications (impossible to cancel!), I mostly enjoy paying for niche-topic newsletters from a single source. A great example is a former newspaper journalist who was laid off and now produces his own newsletter focused on a single college football team. He probably makes more now than when a newspaper employee. I am a happy subscriber. I pay for a handful of these. I also subscribe to newsletters like "Morning Brew." while free and ad-supported, it is well done.

I just came from chicagotribune.com where they tried to entice me with a Flash Sale of one year’s access for a total of $1. Sounds great, but I took advantage of it a year or so back and regretted it due to how annoying they were with advertisements, newsletters, etc…. It’s pretty amazing that the tactics can be so annoying that they can make me regret a $1 purchase.

A decade or 2 ago, there were some talks in several countries about creating a global licensing aggreement where people would just pay one single tax payment per year and have access to everything without being called pirates.

But medias / arts publishers weren't happy with that idea.

Wish you could just have the option to pay 50 cents to unlock a single article.

I agree with you. Where the gap lies for me is that I can’t just buy one meal, I have to sign up for the yearly meal plan even if I just want one meal.

A few years ago I tracked how many times I visited some of the various paywalls sites for their articles, and typically it was between 5-10 times per year. One was 30, so I paid for a subscription to them, but I can’t justify several dozens of dollars for 5 articles on many other sites. If I can’t access their content because a bypass doesn’t work then so be it, however I wasn’t willing to pay for that content either. I feel like it’s the classic misconception regarding piracy by the movie industry - I wasn’t willing to pay money for it in the first place, so it’s not lost revenue (unfortunately).

I was actually discussing this overall problem with my wife the other day, and I came to the conclusion that I basically want a “Netflix” but for news - I subscribe to one place and I get access to a whole range of publications. That’s worth it to me. I very much don’t see it happening though, sadly.

They choose to make the content available to parties that I can impersonate. I respect their decision by impersonating them.

This uses a couple of classic fallacies:

1. Stop comparing digital products (data) to physical products (food).

2. Don't use the word "take", nothing is taken, only copied.

3. "They get to choose how the world sees their creation" Not necessarily. This is a pretty big assumption that lies at the heart of the conflict between the rights of the author and the rights of the public.

They get to choose how the world sees their creation

Then they should not publicize it. They could license only to Google, but Google isn't interested. Instead, publishers need to publicize, which is... expected? Once they publicize, they can't claim the public is not allowed to read. It's like sharing a printed newspaper with my friends. Publishers shouldn't be able to prevent it.

Yes. It's funny how people will claim they only block ads because they allegedly want to pay for good content, but cannot, or claim that piracy is just a service problem. Yet when asked to put their money where their mouth is they instead just continue to openly try and get stuff for free. It's pure entitlement and disrespect for other's labor

As for the "I can; therefore I will" justifications: I can steal from my local corner shop. It's very unlikely they'd catch me. Yet I do not

Because I was stupid and bought an iPhone which means that is not possible.

I would run headless chrome to fetch website and block all fetch requests from ublock origin list. This would give you a "remote" adblocker that works anywhere.

... but at that point just install pihole.

uBO is fine for clearing up crud after you've accessed an article.

It's not useful for piercing paywalls / registration walls in the first place.

It's also not universally available, particularly on mobile devices and/or Google Chrome.

Or noscript browser plugin if you want to completely stop js

https://noscript.net/

most publications are often available for free via public library.

Via public library in the USA. Other countries exist and as far as I’ve gathered aren’t typically given this level of free access.

NYT's onion site is also free.

I just had anecdotal success with it last week and the atlantic, but before that it has been very hit and miss.

Yeah I also have been using archive.today as well, since 12ft hasn't worked on NYT in forever

Very rarely works.

It seems to me that google should not allow a site to serve different content to their bot than they serve to their users.

That would be the fair thing to do and was Google's policy for many years, and still is for all I know. But modern Google stopped caring about fairness and similar concerns many years ago.

I disagree, I think the current approach actually makes for a better and more open web long term. The fact is that either you pay for content, or the content has to pay for itself (which means it's either sponsored content or full of ads). Real journalism costs money, there's no way around that. So we're left with a few options:

Option a) NYT and other news sites makes their news open to everyone without paywall. To finance itself it will become full of ads and disgusting.

Option b) NYT and other news sites become fully walled gardens, letting no-one in (including Google bots). It won't be indexed by Google and search sites, we won't be able to find its context freely. It's like a discord site or facebook groups: there's a treasure trove of information out there, but you won't be able to find it when you need it.

Option c): NYT and other news sites let Google and search sites index their content, but asks the user to pay to access it.

"organizing the world's information"

This is called cloaking[0] and is against Google's policies for many years. But they don't care

[0] https://en.wikipedia.org/wiki/Cloaking

It moved to a Russian equivalent of GitHub or Gitlab (because of where its hosted, YMMV): https://gitflic.ru/project/magnolia1234/bpc_uploads

BPC also has the option to spoof the User-agent as well when using the "custom sites" option:

• set useragent to Googlebot, Bingbot, Facebookbot or custom

• set referer (to Facebook, Google, Twitter or custom; ignored when Googlebot is set)

Yes, but it may be taken down via DMCA soon. See this DMCA request: https://github.com/github/dmca/blob/master/2024/08/2024-08-0...

It mentions bpc_updates in the takedown request....

...or 11ft8, which can open anything

I kind of like the implied concept that self-hosting is the last-foot problem.

14ft. Similar to 13ft but implemented in Rust.

Reminds me of Montague Street Bridge. [1]

[1] https://howmanydayssincemontaguestreetbridgehasbeenhit.com/

The next step

Should that read "Previous"? :)

The docker-compose.yaml file is where you specify the ports you want to expose. It looks like by default it's 5000:5000 (5000 outside and inside the container). You will need to change it and then run docker-compose up -d.

You can change it to something like 5133:5000 and access the instance through localhost:5133

They probably never make their full articles available without a login. Google Search can probably index them well enough with just the excerpt.

Simple as that.

Apologies for submitting it here if it caused any sense of being overwhelmed. Hopefully FOSS is supportive here instead of overwhelming.

Thanks for sharing the project with the internet!

"organizing the world's information" or 'maximizing revenue?' I don't know - somehow either argument justifies this