Statement regarding the ongoing Sourcehut outage

I run a service which was recently DDOS'd. It came immediately after a request from the police services in India to give them data on a specific user. They claimed it was terrorism related. We explained our policy on that (there is a procedure but it requires lengthy legal formalities in our jurisdiction). Within hours our service was taken down. It lasted about 12 hours. We never received any ransom request. We were left with 3 possibilities:

1. Indian police/government

2. Original user

3. Total coincidence

When it happens you just realize: this should not be possible. This is a game that hugely benefits the most powerful players whether they just don't bother solving the problem or actively play in it.

IMO. There is no way Indian Police is competent enough to DDOS your service.

They definitely are.

Beat cops definitely aren't, but most police forces in India are state level and a couple are federal, and will usually have a Task Force devoted to offensive and defensive security, and if not, will contract out to companies like Appin Security Group (who were able to force Reuters and SentinelOne to remove their report into how the private-public cyber model in India operated [0])

Also, after the 2008 Mumbai Attacks, the Indian Ministry of Home Affairs began working on coordinating and centralizing Metadata Analysis and Monitoring [1][2][3], because they unable to trace the VoIP calls used by LeT [4].

If it was actually National Security related, it would have went through a couple fusion centers that the Indian government formed [5][6]. If it was some local PD asking the platform, they probably wouldn't have the capabilities to DDOS

[0] - https://www.reuters.com/investigates/special-report/usa-hack...

[1] - https://en.m.wikipedia.org/wiki/Central_Monitoring_System

[2] - https://en.m.wikipedia.org/wiki/NATGRID

[3] - https://en.m.wikipedia.org/wiki/DRDO_NETRA

[4] - https://www.wired.com/2008/12/mumbai-and-voip/

[5] - https://en.m.wikipedia.org/wiki/National_Cyber_Coordination_...

[6] - https://en.m.wikipedia.org/wiki/Indian_Cyber_Crime_Coordinat...

So, Indian police is actually able to do quite the lot of pressure when terrorism is involved... but when Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it, and leak the first raid to the scammers beforehand? [1]

[1] https://www.youtube.com/watch?v=UdEELggaY5Q

Indian police is actually able to do quite the lot of pressure when terrorism is involved

Because terrorism related stuff falls under the Indian equivalent of Homeland Security, even if it's local PD triaging.

Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it

Raids that are not National Security related need coordination between the Federal Government and State Government in India. The scamming call centers are located in a state called West Bengal. West Bengal is ruled by a regional opposition party called the TMC. West Bengal removed it's consent for Federal Police in India to raid without the consent of West Bengal Police. This ended up in the Supreme Court for a couple years [0].

On top of that, the scamming call centers are closely tied to the ruling party machinery in that state, as you need to get a license from the state government to operate a call center, and these kinds of call centers will often be donating to local political parties to look the other way.

Watch Jamtara on Netflix. It's a good overview on the economics of scam calls in India.

[0] - https://www.deccanherald.com/india/cbi-independent-legal-ent...

Ah, so that is the reason why it's always West Bengal that's mentioned in SP and the other scambaiter videos? I had wondered about that before - I thought that it's mostly because he has infiltrated some local scammer coordination group on Whatsapp.

Many thanks for the context!

Np. India is a federal democracy like the US. The same kind of state-federal clashes that happen in the US happen in India.

Think of Indian democracy as being similar to American democracy in the 1890s-1930s, when local despots like Huey Long and populists like William Jennings Bryan roamed the planet.

that is the reason why it's always West Bengal

Yep. In other states they either will get raided by the Federal Police (eg. CBI) or the economics of running a call center doesn't make sense.

To run a scamming call center you need a low cost English speaking population AND Political Backing. Most states in India will have 1, but not 2 (or at least, not for call centers).

Don't forget hacking activists computers and phones to plant files to convict them of terrorism. [0]

[0] https://www.wired.com/story/modified-elephant-stan-swamy-hac...

That's local/state level police.

That entire apparatus is rotten due to the incentive structure - if you as a cop don't listen to politicians, you'll get a last minute transfer to some village in the middle of nowhere with no running water

Government authorities purchasing shady cyberweapons is a well documented issue. It would a pleasant surprise if there was any government on Earth that didn't do such things out of respect for basic human rights.

https://hn.algolia.com/?q=nso

DDoS-for-hire services exist. They don't have to build their own.

It's not too hard to DDOS a website. Here's the 9th google result for booter service:

https://nightmarestresser.net/

How sure are you that the request came from the actual police in India, and not somebody else? I.e. someone who might get irrationally upset when you did not give them what you wanted?

"someone who might get irrationally upset when you did not give them what you wanted"

Like the police in India?

I would assume that the actual police have their own traditional methods of being irrationally vindictive, and would not normally restort to DDoS.

Indian police will have a lot of trouble beating you if you live in another country.

Google Hardeep Singh Nijjar.

Still a lot of trouble.

Their government might though..

Those traditional methods don't often work very well across borders like a DDoS does. The post at the top of the thread didn't state whether or not they were based in India too.

Yes we went through a bit of procedure to establish that. They have also been in touch with the authorities in our jurisdiction since.

In fact we were contacted by multiple police authorities from different regions.

Agreed, I've been through something like this before and it was not legit. They'd compromised a Bangladeshi police email account and used it to try to get data out of us.

IMO what is surprising about this is that the service didn't already have DDOS protection of some sort (or did it?) It's been a pretty standard practice to add that to any public-facing service for a decade or two now, hasn't it? Cloudflare is free/cheap for smaller services and there are many other options too.

I'd just assume that any service is going to get randomly DDOSed soon after launch, even without any sort of blackmail/targeted attack. Even if you can figure out who did it, there's pretty much no chance of chasing them down.

My takeaway from this wouldn't be "this shouldn't be possible" but "this has been commonplace for decades, and will keep happening, and we should prepare for the next one".

It doesn't really matter whether it's some corrupt local agency or some script kiddie... half the world's connected, on various shitty devices, and DDOSes are gonna keep happening.

Sourcehut is big enough that Cloudfare is neither cheap nor free, I think there was a discussion about this in another post.

Nor is it a good thing that Cloudfare has keys to half of the internet.

Wasn't talking about Sourcehut, but this comment's parent.

And there are alternatives to Cloudflare.

The point is just that DDOSes have been common for decades and they shouldn't be a surprise for anyone. They're an inherent part of the internet protocols we have and the freedom of routing. Saying "they shouldn't exist" is like saying bad actors shouldn't exist. Sure, that's a nice thought, but they do exist (and have existed long before Cloudflare) and we can't just pretend otherwise and believe that our service won't be affected. It's just a matter of time/luck.

Is sr.ht free?

We were recently DDOS'd, too. Every time it happens, I wonder, who benefits from it (or even cares about us)? Competitors? Someone's bored? A disgruntled customer? And we'll probably never know.

Every time it happens, I wonder, who benefits from it

Cloudflare. And any other DDoS protection vendors.

Not saying they caused it but obviously they benefit the same way roof tile manufacturers benefit from hurricanes.

Cloudflare benefits so long you buy protection from them. It doesn't need to be you. So long as you think you need their services it doesn't matter if you actually do. They also have the ability to keep you running if your project explodes in popularity.

Yes my org was recent DDoSed too repeatedly. It was honestly inexplicable. We're still not sure why.

Which police service? Some Military and Internal Security forces have the name "Police", eg. CRPF, ITBP.

If it was one of these forces within the MHA, I wouldn't be surprised, because it falls under "National Security"

Godot, codeberg, sourcehut… any other websites? Looks like somebody is targeting open source related websites.

Somebody is demonstrating their DDOS capabilities to be able to sell capacity to malicious buyers

Statements like this don't help either

"However, this is not an ordinary DDoS attack; the attacker posesses considerable resources and is operating at a scale beyond that which we have the means to mitigate ourselves."

The Chinese government routinely runs DDoS against GitHub because it hosts VPN software that can be used to circumvent the Great Firewall of China. They use the Great Cannon of China, which is the offensive side of the Great Firewall, effectively turning ordinary Chinese Internet users into an unwitting army of DDoS-ers.

Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.

Pardon my lack of knowledge, is it possible to block all requests that originate from a particular country?

Not really, because your neighbor's security camera is likely participating in the attack. See "Mirai botnet" for example.

Also why it's called DDoS and not DoS.

I work in reliability and I've sat at the intersection of this question before.

Kind of, but not really. A lot of times you'll see UDP blocked from EMEA, which stops a good amount of attacks but doesn't solve the problem. It also creates problems for services that rely on UDP like VOIP. These days, even if the command originates from EMEA many of the participants are IOT devices that've been compromised - and those may live in the host country!

Blocking an entire country can do something, sometimes, but it also opens up a wormhole of optics when users who are not knowingly part of malicious activity complain they can't access a service that the rest of the world can. Of course, the host country that operates with a decent degree of CYA acts like they have no idea why someone would do such a thing.

Mitigating this stuff long term is often a game of 4D chess on a rotating board.

While the attack might be sponsored by one country, the servers often come from a wide number of places.

Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.

At that point I wonder why peering partners of Chinese ISPs used in these attacks don't go and drop connectivity unless the abusive traffic gets stopped.

We did just (Americans and allies) Iranian proxies…non-news really when run through the lens of current political environment.

The things under attack are the message, not the attack itself. XD

Microsoft's Github?

Microsoft has very little to gain from DDoSing sourcehut as at the end of the day GitHub’s competition is like flies, AND Microsoft has a lot to loose if it was discovered they did it

Companies are made of people. Those whose wealth and wellbeing depend on Github being the monopoly may be interested in destroying the competition before it grows into a concern.

It would be really stupid for Microsoft, the country's richest company, to go after a tiny competitor via extralegal means. Absolutely would make no sense. The liability they would incur would be astronomical

Not to mention the free publicity for their competitor if they're caught. "We're so good, Microsoft was scared of us"

Cloudflare should just run a public ddos sink with live traffic dashboard so the attackers can demonstrate their power there and leave the rest of us alone. /s

The funniest thing is that this doesn’t strike me as a terrible idea, though maybe it looks bad from a marketing perspective:

“Look, if you’re trying to sell DDoS services, go right ahead, demonstrate your ability on our infrastructure. That way you’ll also know not to target our infrastructure”

But at the same time that might genuinely be a positive for the net. Just like the drug epidemic — you can’t stop people from doing it, but you can reduce the potential harm

Good for the internet but then there would be less attacks on real websites, who is cloudflare going to charge people for protection? Could be a good nonprofit idea though, get some hosting companies together and donate band with to a central target to get people to stop targeting their actual customers.

This is straight out of the onion. I love it so much.

They should use some crappy websites like Twitter to do that.

There is a -high- probability that twitter is where they disseminate their propaganda, so that would be like punching themselves in the face.

Oh… didn’t think of that. Makes sense.

Yeah that seems most likely. The targets will have been chosen for being likely to attract the attention of the relevant communities.

Or someone is trying to make something being hosted there unavailable.

What is the motivation of ddos attacks, in general? I assume it comes with some risk of being criminally prosecuted, so there must be some upside. Is it shakedown attempts, or competitor's sabotage? Neither seems too plausible in this case.

it comes with some risk of being criminally prosecuted

Has there been actual cases of prosecution?

https://www.justice.gov/opa/pr/former-operator-illegal-boote...

https://www.justice.gov/opa/pr/criminal-charges-filed-los-an...

https://www.justice.gov/opa/pr/individual-pleads-guilty-part...

Specifically, have there been cases of prosecution for those who hire the services? All of the links given so far appear to be for operators.

One recent example would be Zeekill, who ran a lot of DDoS attacks:

https://krebsonsecurity.com/2023/02/finlands-most-wanted-hac...

Most of lulzsec, for example. https://en.wikipedia.org/wiki/LulzSec

Motivation? The "quoted a number we cannot reasonably achieve within our financial means" is all the motivation "someone" needs. So yeah "shakedown" or "protection racket".

Are you suggesting that CloudFlare is behind the same ddos attack that they're selling protection against? Because that's an absurd accusation.

No, its more for the "fun" of making firms lose money, kinda LulzSec [0].

[0]: https://en.wikipedia.org/wiki/LulzSec

Why is it absurd? Any such provider has something to gain by scaring us into believing we need their protection.

Not the parent, but obviously if the cost of CloudFlare protection is $XXX million, a protection racket can say: pay us 10% of $XXX M to make this problem go away. It's routinely deployed against highly profitable online businesses like gambling, but I doubt even at a 90% discount (and assuming SourceHut were willing to pay off criminals, which I doubt), they are not likely to be a profitable target for extortion and some other motive must be at play.

Most obvious is: someone doesn't like part of a code hosted on the website, and it is easy to take down. There was probably a cursory warning that we don't know about.

script-kiddies, state actors, misconfigured botnets...

Motivation can often be like why everything is re-written: "Because we can" or just "Because it's there".

I'm not in the circle of these things, but history/news suggests that many are performed by those under the age of adult prosecution, or from countries that don't care, so there is minimal risk, and even when there is those involved are not the sort to believe they will get caught.

For dark net markets, it was commonly used to extract a ransom or to make other websites aware of their DDoS capabilities so that others pay the ransom too. But for a clear web website it's probably not a monetary incentive

Ransom, taking out competitors (e.g. gambling sites), censorship.

The Chinese government regularly targets GitHub because it hosts VPN software.

Often it's as simple as unhinged individuals renting an attack because they feel wronged by their victim. That's a big reason why Discord became so popular (Skype and Teamspeak revealed your IP address) and why the vast majority of online games stopped using direct P2P networking.

Interesting to learn the reason HN was down:

You may have noticed that Hacker News was down on January 10th; we believe that was ultimately due to Cogent’s heavy handed approach to mitigating the DDoS targetting SourceHut (sorry, HN, glad you got it sorted).

It was also in https://news.ycombinator.com/item?id=38939532 but I did not see it earlier.

to be fair, you reach a certain point when the only options left for DDoS protection are Remote Triggered blackholing on your edge.

This results in an entire /24 network not being routed to your network and being dropped by your peer instead.

That's not true. I work for a company that sells DDoS mitigation products to large network operators like Cogent for helping them deal with exactly these kinds of attacks in a much more sane manner than, "oy, just blackhole the destination and head out to lunch."

Either Cogent didn't buy our product (or a competitor's equivalent), or they have a network op who's a fresher and only knows how to blackhole things. Either way, it's a bad look for Cogent.

That is not a new look for cogent.

I remember at one point back in 2009 or so I wasn’t able to download from one of SourceForge’s mirrors. Turned out the reason for that was that Cogent had cut off my ISP (Telia) for some weird reason. IIRC Cogent and Telia hadn’t agreed on a peering policy, so Cogent just said whatever, we’ll cut you off (:

Are there upstream providers in the US or EU that are known to deal with this kind of attacks in a more thoughtful way?

We were a Cogent customer for years. Blackhole was there one and only tool for DDoS. Eventually we just left.

Which is a totally reasonable way of going about the problem, of course. /s

Nobody has ever accused Cogent of being reasonable or competent.

For a blackhole /32 (single IP) I believe is allowed in most ISP/INP.

Many ISP also have DDOS mitigation using bgp flowspec to block dirty traffic only and let valid traffic pass through.

Surprised he put that shout out in, considering he doesn't like Hacker News.

Only DD knows, but I guess even if he dislikes HN, he still might not wish for them to be blackholed.

Likewise for the reach out to CloudFlare given their reverse proxy is actively blocked from source hut and anyone using source hut pages.

i’m confused; is he saying that Cogent black-holed sourcehut’s IP addresses and that somehow affected HN’s IP addresses? they’re in completely separate IP allocations and ASNs...

We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means, but we are investigating other solutions which may be more affordable and have a few avenues for research today, though we cannot disclose too many details without risking alerting the attackers to our plans.

Maybe someone higher up in Cloudflare here can escalate this subject and help them along for a proper quote?

Why should cloudflare be expected to foot the bill instead of sourcehut?

Helping out a useful service? Though who knows really. Cost of services and the price they are sold at are only loosely correlated!

Though honestly it’s totally possible that cloudflare was offering a real discount but it’s still just very high for something like sourcehut.

Yes, that could be the case.

I am unable to edit my original post anymore to reflect that is also my opinion :(

Maybe they were misquoted, who knows. Connections are a nice thing to have and I've never felt bad about asking, knowing the answer could full well be no, but a second opinion is useful when your business is on the line.

Maybe they were misquoted

99.999999% chance they were not.

Just because the quote is expensive doesn't make it not a proper quote. Either CloudFlare does it for PR for free or they need paid for a proper quote.

Let me rephrase: Maybe they can negotiate a different quote?

The PR gained in the development community to help a service beloved by hackers in this site could be very beneficial.

Paid for in exposure

So you want them to be paid in exposure? If a company offers a service that another needs, but doesn't pay for, should they provide the service for free for them? If GitHub goes down, should CF provide the service to them for good PR?

I feel awful for sourcehut, but that doesn't mean that cloudflare should be obliged to support them for free or below cost rates.

I don’t think people realize that a lot of Cloudflare’s free/cheap services are squarely focused on HTTP. Multi protocol stuff is on the enterprise plans where they make their money.

Why not showing the quote? Perhaps other companies could offer a better deal.

This might be due to how outrageous Cloudflare’s pricing is for protecting non-HTTP services like SSH (Cloudflare Spectrum.)

We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means

I don't get it. Cloudflare proudly advertises unmetered DDoS protection on any plan level. Is that just a lie, or what am I missing here? They don't need to be on a custom Enterprise plan.

They are only doing unmetered DDoS for HTTP traffic - for websites. File storage etc. is not allowed. And git falls into that realm.

I see, that explains it.

To expand on that, Cloudflare's standard product is their HTTP reverse proxy. To proxy arbitrary TCP/UDP traffic, you need to use their Cloudflare Spectrum service (https://www.cloudflare.com/en-gb/application-services/produc...), which is metered.

I think they might be looking in something more similar to Magic Transit, but that depends on how they set up their infrastructure.

wow $1/GB?

would be cheaper to pay a developer to add websocket support to openssh

Unlimited free is almost always a lie.

But in this case it literally is? Cloudflare provides unlimited HTTP traffic on all plans, exactly what they claim. Sourcehut needs a different service.

HTTP(S)-only _is_ the limit.

Yeah, because protecting other protocols would require a totally different approach and product. This isn't a conspiracy.

The Cloudflare HTTP CDN cannot protect SSH any more than a condom will make a good umbrella just because they both are designed for protection!

See also "unlimited paid time off".

https://www.cloudflare.com/en-gb/plans/

Looks like level 3 ddos protection is only available on the enterprise plan, it's not included in the unmetered ddos protection.

Layer 3*

Reminder that source hut makes a value judgement on your source code and will ban repositories they don't like https://news.ycombinator.com/item?id=33403780

Why do you need to convert "goes against their principles" to "doesn't like" to make your argument? Is it because it is a weak argument without twisting words?

They're allowed to decide what they'll host, and it looks like they're just being clear about a boundary. Unless they're playing favorites and allowing some crypto, what is the problem?

I don't want my source code vendor to tell me what source code I am allowed to write

They ain't doing that, they are just telling you which code they don't like to host.

What does that have to do with getting DDoSed? Because "you don't like them" they deserve to be DDoSed? Or is this just, "BTW, I am salty." Nobody owes you a centralized location for your distributed source code repositories.

Goes against their principles is an explanation for why they don't like it, so I don't see anything wrong with saying "if they don't like it". OP is correct and under no obligation for explaining their motives.

They and you are both incorrect. You both wish to pretend that because all dolphins are mammals, all mammals are dolphins.

They aren't removing code simply because they don't like it. They have defined a term of service. You can pretend it's more capricious than that, but it isn't the truth unless you have some additional evidence beyond that one thing.

You have an obvious bias given your username. Why should SourceHut involve itself with projects that are at a high risk of being a scam?

I followed that link to see what "don't like" means, and it means "no cryptocurrency projects". That's +1 to SourceHut in my book.

And that's a good thing. A bootstrapped company doesn't have to be a slave to money.

Codeberg went down to DDoS ~12h ago too:

https://status.codeberg.eu/status/codeberg

I wonder why they're thinking about moving to EU as their compatriot seems to be having the same issue and moving to Europe wouldn't fix it. It was mentioned in their public notice.

Their plan to move to the EU via their AMS site is not related to the attacks, but seems to be a general plan they have had for a while. At least this is how I understand their notice.

They were planning to moving to EU for the last 6-8 months. They were making it gracefully.

This attack expedited it greatly.

The EU move wouldn't exactly help, if the attack is specifically directed at Sourcehut. That is unless the ISP and hosting provider in AMS is offering some form of protection as part of their offering. I can't see why the attack wouldn't just follow the services.

This is certainly possible, but it seems they are anticipating this as well.

One of our main concerns right now is finding a way of getting back online on a new network without the DDoS immediately following us there, and we have reason to believe that it will.

That doesn't make sense though, as all the DDOS people have to do is aim their bot army at the new site. Probably just a single parameter in their attack scripts. I mean I'm an idiot and I can launch a DDoS on someone, I just don't have the $$ or compromised army of iot devices to aim at accomplishing that, nor really the will to do others harm. Whether you're in Europe or NA doesn't matter.

If their plan is to get online via the second location, it's likely that said location has a much beefier upstream or built-in filtering, allowing them to absorb these amounts of traffic without being null-routed.

More likely, though, they're restoring the service to a fresh IP range and put the servers behind some kind of DDOS-protection or, alternatively, they simply choose to do the switch now as they need to do a full restore anyway and it's not related to the DDOS mitigation.

You're right, but they were already planning it, so why not?

Maybe they have another plans under that, a better server or set of servers, some hand-rolled mitigations, etc. I have no idea. I'm a user as everybody else.

I assumed that the addresses for the EU server are not publicly available, so if they get some sort of DDOS protection before bringing them up, then the attackers will not know where to target their attack.

Am I correct in assuming these kind of problems are not possible if you are using a major cloud provider instead of renting rack space?

No, that is not a correct assumption. It's also worth remembering that cloud providers are also servers in racks, but they own the building around it to.

For DDoS attacks, you need to have enough capacity to absorb the attack. Major cloud providers tend to have that, as do DDoS mitigation services (Cloudflare amongst others).

We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means [..]

Typically what you want to do though is stop the traffic from reaching you at all, so ideally your network provider, who is upstream from you, blocks the illegitimate traffic so your servers never see it and don't get overwhelmed.

What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage. Imagine if your ISP decided to stop routing traffic to Google. Being hosted on GCP, a major cloud provider, would be of no use, since there wouldn't be a network path to them in the first place.

In general, Cogent seems to be doing a rather bad job at dealing with this attack and there's been fallout for many services beyond Sourchut. Google or AWS or Microsoft might've handled it more gracefully, or might not. Though major cloud providers tend to have their own connectivity between their datacenters, they too have peering/transit agreements with other major network providers. If those upstreams stop forwarding traffic to them, the same thing would happen. It's just less likely to go unnoticed.

Cogent is a massive provider, so you'd think they'd be a bit better at this. But they also have a reputation for being awful.

Cogent is also a low-cost provider, and it shows in their customer service.

What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage.

That's not what happened based on my understanding. The provider nullrouted their traffic (which is common if a customer is under attack), but Sourcehut couldn't talk to the customer support as their support panel wasn't working for them.

For any company with a complicated cloud footprint, how do they avoid "denial-of-money" by having the attacker pick some externally reachable piece of the cloud setup where the attacker can take actions that the company pays for (e.g. download large files from a cloud storage bucket, generating huge bandwidth charges)?

This is a major problem, particularly for those offering any kind of outbound SMS capability, whether for 2FA or just phone number verification.

That service is the easiest and most profitable to abuse, there are certain providers in certain countries that price inbound SMS very steeply, and are willing to share the profits with you. If you if you can get an attack going.

Depends if you buy their ddos protection services or not.

You are probably correct in assuming these kinds of problems are not possible given enough $$$

Still very much possible, in most cases all you're doing at the cloud provider is running virtual hardware through load balancers so they can very much become overwhelmed. Even with more advanced setups, it's still all just hitting a variety of servers, each of which can be affected in different ways.

There are (expensive) ways of mitigating but a project like Sourcehut couldn't afford or justify what will likely be a 5 to 6 figure sum.

If you're using a major cloud provider, a DDoS might mean either that your service hits scaling limits or that you get a massive bill.

Quite ironic when you consider that Drew is a fan of DDoS attacks and transit provider blackholing when it's aimed at websites he doesn't like.

When has Drew said he was a fan of DDoS attacks? When you make claims like these you should really bring receipts.

SourceHut doesn't blackhole content. It is in their terms of service and documentation what they will not accept on their website.

Not GP.

I agree with you that there is no evidence that Drew likes DDoS.

However, his ToS are sufficiently vague to allow him to deplatform people he doesn't like. And he said publicly that he will. [1]

[1]: https://news.ycombinator.com/item?id=33416823

Yes, he doesn't like cryptocurrency projects. I don't see much vague about it.

The other clauses about bigotry are pretty much aligned with GitHub's ToS, which has "GitHub does not tolerate speech that attacks or promotes hate toward an individual or group of people on the basis of who they are, including age, body size, ability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, sexual identity, or sexual orientation", and "We have the right to refuse or remove any User-Generated Content that, in our sole discretion, violates any laws or GitHub terms or policies."

While people who don't like these policies call it "deplatforming", in the US the freedom of association is a protected First Amendment right, with only limited exceptions for certain protected classes like race, ethnicity, and national origin where there is a strong tension between the First Amendment protections and other constitutional protections.

I don't know about Dutch law though.

In private communication, he told me that he would ban me for being Christian.

Yeah, his policies are "aligned" with GitHub's (and I avoid that too), but Drew is on the record saying he will ban people for their opinions, not their actions.

I'm missing a lot of context.

A whole lot of people are Christian, so if this were true we should see evidence of this happening by now on Sourcehut.

If the ban was for membership in an "I am not a Christian" club, then that ban would be entirely reasonable.

How does he learn about any of your opinions without any sort of action? You must express the opinion first, and the act of expressing an opinion is an action.

I can think of many times when people were fired or "de-platformed" due to their opinion, like Jimmy Snyder (a.k.a. "Jimmy the Greek") - https://en.wikipedia.org/wiki/Jimmy_Snyder_(sports_commentat... , so it isn't like banning someone due to expressing a specific opinion is inherently unconscionable.

There are a whole lot of different kinds of Christians. I am a traditional Christian (not a fundamentalist) and tend to be more outspoken.

But doesn't that prove my point? You say that if I said an opinion, I made an action, so you are saying that Source Hut is right to ban me for opinions.

But you and I fundamentally disagree: banning someone for their opinion is not inherently conscionable.

You may find yourself with a "wrong" opinion someday and be banned from places you find important. Will you find it conscionable then?

By the way, expressing an opinion was not the kind of action I meant. I meant doing something that might harm others. My opinions do not call for harm (in fact, they call out what I see as harm), so my opinions cannot cause harm except by some broad definition that "speech is violence."

So why is it okay to ban me? Free Software needs Free Speech. SourceHut, as Drew has admitted, is not the place for that.

Good morning, Gavin. You were shown the door because you are an outspoken transphobe and we were not interested in helping you voice those opinions. You seem to be unable to disentangle this with your Christian identity, but most Christians seem to manage alright.

Because there are different kinds of Christians, Drew. And a lot of the ones I have seen aren't really Christians.

Also, I am not scared of trans people, so why the terrible term? Oh, that's right: it's a pure smear.

But doesn't that prove my point? You would ban me for my opinion even though I have actually done nothing!

Free Software needs Free Speech, Drew. Even for people whose opinions you don't like.

1. Do "PHL, FRE, and AMS" mean something? Or are these just codenames for each site?

2. If I host a service on AWS, Azure, Linode, DigitalOcean am I also susceptible to layer 3 DDoS?

They're generally airport codes, so Philadelphia and Amsterdam... unsure about FRE.

The IATA code FRE is assigned to Fera in the Solomon Islands. Freemont Airport does not have an IATA code.

Naming conventions can be inconsistent, and not every datacenter location has an airport. ASD is often used instead of AMS for Amsterdam.

ASD is the IATA code assigned to Andros Town Airport, in the Bahamas. It might also be a commonly used abbreviation for a medical condition.

Linode went through a rather long DDoS attack a few years back with a few of their data enters being offline for a few days, so I would guess yes there.

Their topology page has information about PHL/AMS and FRE (seems like Fremont datacenter):

https://web.archive.org/web/20240111132224/https://man.sr.ht...

To answer your 2nd question, yes you can be DDoSd, Azure specifically offers a DDoS protection plan which is quite expensive. https://learn.microsoft.com/en-us/azure/ddos-protection/ddos...

1. PHL means Philadelphia, AMS means Amsterdam. Yes, just codenames for sites.

2. Depends if DDoS protection is part of the offer, I suppose.

I wish the sr.ht employees the best of luck with the AMS migration. Luckily git is decentralized enough that I've been able to practically do everything the last few days except that I can't push new releases of software to the canonical upstream repository.

Erm... AWS migration? What am I missing?

A DDoS on AWS would most likely bankrupt Drew and Sourcehut, even if Amazon managed to absorb the traffic. Not to mention the principial issues that they have with AWS in the first place.

AMS ≠ AWS :P

Erm, apologies, thank you for the correction. I'm embarrassed. :)

Erm... AWS migration? What am I missing?

It's A M S, not A W S.

Also, the fact sourcehut does not baby users about git send-email means that collaborative development happening on the platform is uniquely posed to continue

sure, i can `git send-email` to the particular developers i've collaborated with before, but by default my send-email goes to just the list-serve at `~user/project@lists.sr.ht`. pulling out `nc lists.sr.ht 25` seems to show that it's also offline.

so, yes for really important stuff i could get patches through to projects i care about. but in practice the handful of projects i'm involved in on sr.ht seem to be mostly stalled.

Yeah. My comment was largely optimistic, in practice people would rather just wait for service.

That being said, I did send a small patch directly by email just for the fun of it.

This site can’t be reached

DNS_PROBE_FINISHED_NXDOMAIN

I can reach it but here's an archive: https://archive.is/wVtqN

I does work for me, but more people seem to experience that problem:

https://fosstodon.org/@drewdevault/111742324107487646

Oof, my heart goes out to them. My very first week at [large popular public code forge], we were attacked by [state level actor], probably for hosting something that [state] didn't like. In a way, SourceHut and Codeberg getting this kind of attention is an encouraging sign that these alternative forges are starting to gain traction.

I wonder how you could figure out it was [state]. Was there some clear threat made? A blackmail "do this, or else.."?

This is likely a reference to GitHub, which was DoSed by the Chinese government for hosting a repository that was mirroring uncensored Western media on a domain that could not be blocked by the Great Firewall without hobbling the Chinese software industry.

The attribution wasn't subtle - a substantial fraction of Baidu's ads/analytics traffic served to domestic Chinese users was rewritten to hammer that specific repository directly.

NYT coverage at the time: https://www.nytimes.com/2015/03/31/technology/china-appears-...

I suspect a possible DDoS attack from something Kiwi Farms related, it seems they really don't like the guy over there?

https://archive.is/yOObX

If kiwifarms ddosed everyone that they didn't like... plus, if anything KF has been surprisingly resilient to massive DDoS without any 3rd party protection service since they lost CloudFlare and other providers. So for this one specific thing, I think lessons can be learnt from kf on how to mitigate DDoS without 3rd party providers.

They developed their own proof-of-work system to rate limit requests, using SHA256, implementation details here: https://archive.is/dfBVN

They also have servers in multiple countries. Some of these servers being blackholed by Cogent, by the way.

“what if the primary datacenter just disappeared tomorrow?” We ask this question of ourselves seriously, and make serious plans for what we’d do if this were to pass, and we are executing those plans now – though we had hoped that we would never have to.

Respectfully, if you guys asked this question...how come you don't have a cluster slave as a replica in another data center where the whole thing is synced?

A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.

A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.

If you want a Klugey non-production IoT solution that bodges up something really important THESE days, all the cool kids are using ESP32s.

And as much as I think that would be a totally inappropriate solution for src.ht, I kinda wanna go make a "black-hole" switch for my office.

So it turns out that running your own servers isn't always the best idea

HugOps to the Drew, the rest of the Sourcehut crew, and anyone else working on this. Compliments on the clear blog post.

My name is Drew, I’m the founder of SourceHut and one of three SourceHut staff members working on the outage, alongside my colleagues Simon and Conrad. As you have noticed, SourceHut is down. I offer my deepest apologies for this situation. We have made a name for ourselves for reliability, and this is the most severe and prolonged outage we have ever faced. We spend a lot of time planning to make sure this does not happen, and we failed. We have all hands on deck working the problem to restore service as soon as possible.

Drew, you're great, Simon and Conrad are great, you'll get through this, and you will be fine.

Keep doing your great work, forever.

Feel bad for the team to have to deal with this, but I have to say its a great example of how to communicate with your customers so hats off to you!

The real led in the story:

Company (cloudflare) that protects DDoS'ers and booters charges insane protection racket for DDoS'ers and booters.

Relevant article: https://rasbora.dev/blog/I-ran-the-worlds-largest-ddos-for-h...

Sad to see, best of luck to them