return to table of content

Statement regarding the ongoing Sourcehut outage

throw9999away
30 replies
1d7h

I run a service which was recently DDOS'd. It came immediately after a request from the police services in India to give them data on a specific user. They claimed it was terrorism related. We explained our policy on that (there is a procedure but it requires lengthy legal formalities in our jurisdiction). Within hours our service was taken down. It lasted about 12 hours. We never received any ransom request. We were left with 3 possibilities:

1. Indian police/government

2. Original user

3. Total coincidence

When it happens you just realize: this should not be possible. This is a game that hugely benefits the most powerful players whether they just don't bother solving the problem or actively play in it.

leosanchez
10 replies
1d5h

IMO. There is no way Indian Police is competent enough to DDOS your service.

alephnerd
6 replies
1d4h

They definitely are.

Beat cops definitely aren't, but most police forces in India are state level and a couple are federal, and will usually have a Task Force devoted to offensive and defensive security, and if not, will contract out to companies like Appin Security Group (who were able to force Reuters and SentinelOne to remove their report into how the private-public cyber model in India operated [0])

Also, after the 2008 Mumbai Attacks, the Indian Ministry of Home Affairs began working on coordinating and centralizing Metadata Analysis and Monitoring [1][2][3], because they unable to trace the VoIP calls used by LeT [4].

If it was actually National Security related, it would have went through a couple fusion centers that the Indian government formed [5][6]. If it was some local PD asking the platform, they probably wouldn't have the capabilities to DDOS

[0] - https://www.reuters.com/investigates/special-report/usa-hack...

[1] - https://en.m.wikipedia.org/wiki/Central_Monitoring_System

[2] - https://en.m.wikipedia.org/wiki/NATGRID

[3] - https://en.m.wikipedia.org/wiki/DRDO_NETRA

[4] - https://www.wired.com/2008/12/mumbai-and-voip/

[5] - https://en.m.wikipedia.org/wiki/National_Cyber_Coordination_...

[6] - https://en.m.wikipedia.org/wiki/Indian_Cyber_Crime_Coordinat...

mschuster91
3 replies
1d3h

So, Indian police is actually able to do quite the lot of pressure when terrorism is involved... but when Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it, and leak the first raid to the scammers beforehand? [1]

[1] https://www.youtube.com/watch?v=UdEELggaY5Q

alephnerd
2 replies
1d3h

Indian police is actually able to do quite the lot of pressure when terrorism is involved

Because terrorism related stuff falls under the Indian equivalent of Homeland Security, even if it's local PD triaging.

Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it

Raids that are not National Security related need coordination between the Federal Government and State Government in India. The scamming call centers are located in a state called West Bengal. West Bengal is ruled by a regional opposition party called the TMC. West Bengal removed it's consent for Federal Police in India to raid without the consent of West Bengal Police. This ended up in the Supreme Court for a couple years [0].

On top of that, the scamming call centers are closely tied to the ruling party machinery in that state, as you need to get a license from the state government to operate a call center, and these kinds of call centers will often be donating to local political parties to look the other way.

Watch Jamtara on Netflix. It's a good overview on the economics of scam calls in India.

[0] - https://www.deccanherald.com/india/cbi-independent-legal-ent...

mschuster91
1 replies
1d3h

Ah, so that is the reason why it's always West Bengal that's mentioned in SP and the other scambaiter videos? I had wondered about that before - I thought that it's mostly because he has infiltrated some local scammer coordination group on Whatsapp.

Many thanks for the context!

alephnerd
0 replies
1d3h

Np. India is a federal democracy like the US. The same kind of state-federal clashes that happen in the US happen in India.

Think of Indian democracy as being similar to American democracy in the 1890s-1930s, when local despots like Huey Long and populists like William Jennings Bryan roamed the planet.

that is the reason why it's always West Bengal

Yep. In other states they either will get raided by the Federal Police (eg. CBI) or the economics of running a call center doesn't make sense.

To run a scamming call center you need a low cost English speaking population AND Political Backing. Most states in India will have 1, but not 2 (or at least, not for call centers).

janice1999
1 replies
1d3h

Don't forget hacking activists computers and phones to plant files to convict them of terrorism. [0]

[0] https://www.wired.com/story/modified-elephant-stan-swamy-hac...

alephnerd
0 replies
1d3h

That's local/state level police.

That entire apparatus is rotten due to the incentive structure - if you as a cop don't listen to politicians, you'll get a last minute transfer to some village in the middle of nowhere with no running water

soraminazuki
0 replies
1d4h

Government authorities purchasing shady cyberweapons is a well documented issue. It would a pleasant surprise if there was any government on Earth that didn't do such things out of respect for basic human rights.

https://hn.algolia.com/?q=nso

rst
0 replies
1d4h

DDoS-for-hire services exist. They don't have to build their own.

Thorrez
0 replies
1d4h

It's not too hard to DDOS a website. Here's the 9th google result for booter service:

https://nightmarestresser.net/

teddyh
9 replies
1d5h

How sure are you that the request came from the actual police in India, and not somebody else? I.e. someone who might get irrationally upset when you did not give them what you wanted?

mattw2121
6 replies
1d5h

"someone who might get irrationally upset when you did not give them what you wanted"

Like the police in India?

teddyh
5 replies
1d5h

I would assume that the actual police have their own traditional methods of being irrationally vindictive, and would not normally restort to DDoS.

alright2565
3 replies
1d5h

Indian police will have a lot of trouble beating you if you live in another country.

not2b
1 replies
1d2h

Google Hardeep Singh Nijjar.

computerfriend
0 replies
13h21m

Still a lot of trouble.

m-p-3
0 replies
1d4h

Their government might though..

dspillett
0 replies
1d4h

Those traditional methods don't often work very well across borders like a DDoS does. The post at the top of the thread didn't state whether or not they were based in India too.

throw9999away
0 replies
1d5h

Yes we went through a bit of procedure to establish that. They have also been in touch with the authorities in our jurisdiction since.

In fact we were contacted by multiple police authorities from different regions.

Polycryptus
0 replies
1d1h

Agreed, I've been through something like this before and it was not legit. They'd compromised a Bangladeshi police email account and used it to try to get data out of us.

solardev
3 replies
1d2h

IMO what is surprising about this is that the service didn't already have DDOS protection of some sort (or did it?) It's been a pretty standard practice to add that to any public-facing service for a decade or two now, hasn't it? Cloudflare is free/cheap for smaller services and there are many other options too.

I'd just assume that any service is going to get randomly DDOSed soon after launch, even without any sort of blackmail/targeted attack. Even if you can figure out who did it, there's pretty much no chance of chasing them down.

My takeaway from this wouldn't be "this shouldn't be possible" but "this has been commonplace for decades, and will keep happening, and we should prepare for the next one".

It doesn't really matter whether it's some corrupt local agency or some script kiddie... half the world's connected, on various shitty devices, and DDOSes are gonna keep happening.

whatevaa
2 replies
1d2h

Sourcehut is big enough that Cloudfare is neither cheap nor free, I think there was a discussion about this in another post.

Nor is it a good thing that Cloudfare has keys to half of the internet.

solardev
0 replies
1d2h

Wasn't talking about Sourcehut, but this comment's parent.

And there are alternatives to Cloudflare.

The point is just that DDOSes have been common for decades and they shouldn't be a surprise for anyone. They're an inherent part of the internet protocols we have and the freedom of routing. Saying "they shouldn't exist" is like saying bad actors shouldn't exist. Sure, that's a nice thought, but they do exist (and have existed long before Cloudflare) and we can't just pretend otherwise and believe that our service won't be affected. It's just a matter of time/luck.

AzuraIsCool
0 replies
23h49m

Is sr.ht free?

kgeist
3 replies
1d3h

We were recently DDOS'd, too. Every time it happens, I wonder, who benefits from it (or even cares about us)? Competitors? Someone's bored? A disgruntled customer? And we'll probably never know.

rrdharan
1 replies
1d3h

Every time it happens, I wonder, who benefits from it

Cloudflare. And any other DDoS protection vendors.

Not saying they caused it but obviously they benefit the same way roof tile manufacturers benefit from hurricanes.

bluGill
0 replies
1d3h

Cloudflare benefits so long you buy protection from them. It doesn't need to be you. So long as you think you need their services it doesn't matter if you actually do. They also have the ability to keep you running if your project explodes in popularity.

emmp
0 replies
1d1h

Yes my org was recent DDoSed too repeatedly. It was honestly inexplicable. We're still not sure why.

alephnerd
0 replies
1d4h

Which police service? Some Military and Internal Security forces have the name "Police", eg. CRPF, ITBP.

If it was one of these forces within the MHA, I wouldn't be surprised, because it falls under "National Security"

beretguy
24 replies
1d7h

Godot, codeberg, sourcehut… any other websites? Looks like somebody is targeting open source related websites.

nusl
23 replies
1d7h

Somebody is demonstrating their DDOS capabilities to be able to sell capacity to malicious buyers

kramerger
8 replies
1d6h

Statements like this don't help either

"However, this is not an ordinary DDoS attack; the attacker posesses considerable resources and is operating at a scale beyond that which we have the means to mitigate ourselves."

:(

fmajid
6 replies
1d4h

The Chinese government routinely runs DDoS against GitHub because it hosts VPN software that can be used to circumvent the Great Firewall of China. They use the Great Cannon of China, which is the offensive side of the Great Firewall, effectively turning ordinary Chinese Internet users into an unwitting army of DDoS-ers.

Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.

beretguy
4 replies
1d3h

Pardon my lack of knowledge, is it possible to block all requests that originate from a particular country?

dns_snek
1 replies
1d2h

Not really, because your neighbor's security camera is likely participating in the attack. See "Mirai botnet" for example.

treve
0 replies
1d1h

Also why it's called DDoS and not DoS.

oooyay
0 replies
23h22m

I work in reliability and I've sat at the intersection of this question before.

Kind of, but not really. A lot of times you'll see UDP blocked from EMEA, which stops a good amount of attacks but doesn't solve the problem. It also creates problems for services that rely on UDP like VOIP. These days, even if the command originates from EMEA many of the participants are IOT devices that've been compromised - and those may live in the host country!

Blocking an entire country can do something, sometimes, but it also opens up a wormhole of optics when users who are not knowingly part of malicious activity complain they can't access a service that the rest of the world can. Of course, the host country that operates with a decent degree of CYA acts like they have no idea why someone would do such a thing.

Mitigating this stuff long term is often a game of 4D chess on a rotating board.

jszymborski
0 replies
1d3h

While the attack might be sponsored by one country, the servers often come from a wide number of places.

mschuster91
0 replies
1d3h

Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.

At that point I wonder why peering partners of Chinese ISPs used in these attacks don't go and drop connectivity unless the abusive traffic gets stopped.

b4ke
0 replies
1d5h

We did just (Americans and allies) Iranian proxies…non-news really when run through the lens of current political environment.

The things under attack are the message, not the attack itself. XD

sam_lowry_
4 replies
1d6h

Microsoft's Github?

internetter
3 replies
1d5h

Microsoft has very little to gain from DDoSing sourcehut as at the end of the day GitHub’s competition is like flies, AND Microsoft has a lot to loose if it was discovered they did it

sam_lowry_
2 replies
1d5h

Companies are made of people. Those whose wealth and wellbeing depend on Github being the monopoly may be interested in destroying the competition before it grows into a concern.

dingnuts
1 replies
1d3h

It would be really stupid for Microsoft, the country's richest company, to go after a tiny competitor via extralegal means. Absolutely would make no sense. The liability they would incur would be astronomical

Snow_Falls
0 replies
1d3h

Not to mention the free publicity for their competitor if they're caught. "We're so good, Microsoft was scared of us"

progbits
3 replies
1d6h

Cloudflare should just run a public ddos sink with live traffic dashboard so the attackers can demonstrate their power there and leave the rest of us alone. /s

internetter
2 replies
1d5h

The funniest thing is that this doesn’t strike me as a terrible idea, though maybe it looks bad from a marketing perspective:

“Look, if you’re trying to sell DDoS services, go right ahead, demonstrate your ability on our infrastructure. That way you’ll also know not to target our infrastructure”

But at the same time that might genuinely be a positive for the net. Just like the drug epidemic — you can’t stop people from doing it, but you can reduce the potential harm

Snow_Falls
1 replies
1d3h

Good for the internet but then there would be less attacks on real websites, who is cloudflare going to charge people for protection? Could be a good nonprofit idea though, get some hosting companies together and donate band with to a central target to get people to stop targeting their actual customers.

internetter
0 replies
1d2h

This is straight out of the onion. I love it so much.

beretguy
2 replies
1d6h

They should use some crappy websites like Twitter to do that.

EasyMark
1 replies
1d6h

There is a -high- probability that twitter is where they disseminate their propaganda, so that would be like punching themselves in the face.

beretguy
0 replies
1d3h

Oh… didn’t think of that. Makes sense.

wly_cdgr
0 replies
1d1h

Yeah that seems most likely. The targets will have been chosen for being likely to attract the attention of the relevant communities.

numbsafari
0 replies
1d6h

Or someone is trying to make something being hosted there unavailable.

febeling
15 replies
1d8h

What is the motivation of ddos attacks, in general? I assume it comes with some risk of being criminally prosecuted, so there must be some upside. Is it shakedown attempts, or competitor's sabotage? Neither seems too plausible in this case.

raverbashing
4 replies
1d8h

it comes with some risk of being criminally prosecuted

Has there been actual cases of prosecution?

o11c
0 replies
1d1h

Specifically, have there been cases of prosecution for those who hire the services? All of the links given so far appear to be for operators.

dewey
0 replies
1d6h

One recent example would be Zeekill, who ran a lot of DDoS attacks:

https://krebsonsecurity.com/2023/02/finlands-most-wanted-hac...

Propa_ganda
0 replies
1d6h

Most of lulzsec, for example. https://en.wikipedia.org/wiki/LulzSec

Am4TIfIsER0ppos
4 replies
1d6h

Motivation? The "quoted a number we cannot reasonably achieve within our financial means" is all the motivation "someone" needs. So yeah "shakedown" or "protection racket".

arrowsmith
3 replies
1d5h

Are you suggesting that CloudFlare is behind the same ddos attack that they're selling protection against? Because that's an absurd accusation.

n1c00o
0 replies
1d4h

No, its more for the "fun" of making firms lose money, kinda LulzSec [0].

[0]: https://en.wikipedia.org/wiki/LulzSec

mwcampbell
0 replies
1d4h

Why is it absurd? Any such provider has something to gain by scaring us into believing we need their protection.

fmajid
0 replies
1d4h

Not the parent, but obviously if the cost of CloudFlare protection is $XXX million, a protection racket can say: pay us 10% of $XXX M to make this problem go away. It's routinely deployed against highly profitable online businesses like gambling, but I doubt even at a 90% discount (and assuming SourceHut were willing to pay off criminals, which I doubt), they are not likely to be a profitable target for extortion and some other motive must be at play.

nurettin
0 replies
1d7h

Most obvious is: someone doesn't like part of a code hosted on the website, and it is easy to take down. There was probably a cursory warning that we don't know about.

mobiuscog
0 replies
1d7h

script-kiddies, state actors, misconfigured botnets...

Motivation can often be like why everything is re-written: "Because we can" or just "Because it's there".

I'm not in the circle of these things, but history/news suggests that many are performed by those under the age of adult prosecution, or from countries that don't care, so there is minimal risk, and even when there is those involved are not the sort to believe they will get caught.

mardifoufs
0 replies
22h8m

For dark net markets, it was commonly used to extract a ransom or to make other websites aware of their DDoS capabilities so that others pay the ransom too. But for a clear web website it's probably not a monetary incentive

fmajid
0 replies
1d4h

Ransom, taking out competitors (e.g. gambling sites), censorship.

The Chinese government regularly targets GitHub because it hosts VPN software.

dns_snek
0 replies
1d2h

Often it's as simple as unhinged individuals renting an attack because they feel wronged by their victim. That's a big reason why Discord became so popular (Skype and Teamspeak revealed your IP address) and why the vast majority of online games stopped using direct P2P networking.

Aissen
13 replies
1d7h

Interesting to learn the reason HN was down:

You may have noticed that Hacker News was down on January 10th; we believe that was ultimately due to Cogent’s heavy handed approach to mitigating the DDoS targetting SourceHut (sorry, HN, glad you got it sorted).

It was also in https://news.ycombinator.com/item?id=38939532 but I did not see it earlier.

kazen44
8 replies
1d7h

to be fair, you reach a certain point when the only options left for DDoS protection are Remote Triggered blackholing on your edge.

This results in an entire /24 network not being routed to your network and being dropped by your peer instead.

bityard
4 replies
1d5h

That's not true. I work for a company that sells DDoS mitigation products to large network operators like Cogent for helping them deal with exactly these kinds of attacks in a much more sane manner than, "oy, just blackhole the destination and head out to lunch."

Either Cogent didn't buy our product (or a competitor's equivalent), or they have a network op who's a fresher and only knows how to blackhole things. Either way, it's a bad look for Cogent.

dfc
1 replies
1d5h

That is not a new look for cogent.

cpach
0 replies
1d4h

I remember at one point back in 2009 or so I wasn’t able to download from one of SourceForge’s mirrors. Turned out the reason for that was that Cogent had cut off my ISP (Telia) for some weird reason. IIRC Cogent and Telia hadn’t agreed on a peering policy, so Cogent just said whatever, we’ll cut you off (:

tillulen
0 replies
16h42m

Are there upstream providers in the US or EU that are known to deal with this kind of attacks in a more thoughtful way?

booi
0 replies
22h51m

We were a Cogent customer for years. Blackhole was there one and only tool for DDoS. Eventually we just left.

mariusor
1 replies
1d5h

Which is a totally reasonable way of going about the problem, of course. /s

coldacid
0 replies
1d4h

Nobody has ever accused Cogent of being reasonable or competent.

zakki
0 replies
18h48m

For a blackhole /32 (single IP) I believe is allowed in most ISP/INP.

Many ISP also have DDOS mitigation using bgp flowspec to block dirty traffic only and let valid traffic pass through.

gadders
2 replies
1d4h

Surprised he put that shout out in, considering he doesn't like Hacker News.

cpach
0 replies
19h55m

Only DD knows, but I guess even if he dislikes HN, he still might not wish for them to be blackholed.

_kb
0 replies
1d4h

Likewise for the reach out to CloudFlare given their reverse proxy is actively blocked from source hut and anyone using source hut pages.

scour
0 replies
23h8m

i’m confused; is he saying that Cogent black-holed sourcehut’s IP addresses and that somehow affected HN’s IP addresses? they’re in completely separate IP allocations and ASNs...

csunbird
12 replies
1d7h

We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means, but we are investigating other solutions which may be more affordable and have a few avenues for research today, though we cannot disclose too many details without risking alerting the attackers to our plans.

Maybe someone higher up in Cloudflare here can escalate this subject and help them along for a proper quote?

maccard
4 replies
1d6h

Why should cloudflare be expected to foot the bill instead of sourcehut?

rtpg
1 replies
1d6h

Helping out a useful service? Though who knows really. Cost of services and the price they are sold at are only loosely correlated!

Though honestly it’s totally possible that cloudflare was offering a real discount but it’s still just very high for something like sourcehut.

csunbird
0 replies
1d5h

Yes, that could be the case.

I am unable to edit my original post anymore to reflect that is also my opinion :(

EasyMark
1 replies
1d6h

Maybe they were misquoted, who knows. Connections are a nice thing to have and I've never felt bad about asking, knowing the answer could full well be no, but a second opinion is useful when your business is on the line.

KomoD
0 replies
1d3h

Maybe they were misquoted

99.999999% chance they were not.

that_guy_iain
3 replies
1d7h

Just because the quote is expensive doesn't make it not a proper quote. Either CloudFlare does it for PR for free or they need paid for a proper quote.

csunbird
2 replies
1d5h

Let me rephrase: Maybe they can negotiate a different quote?

The PR gained in the development community to help a service beloved by hackers in this site could be very beneficial.

xeromal
0 replies
1d

Paid for in exposure

maccard
0 replies
1d

So you want them to be paid in exposure? If a company offers a service that another needs, but doesn't pay for, should they provide the service for free for them? If GitHub goes down, should CF provide the service to them for good PR?

I feel awful for sourcehut, but that doesn't mean that cloudflare should be obliged to support them for free or below cost rates.

samcat116
0 replies
1d4h

I don’t think people realize that a lot of Cloudflare’s free/cheap services are squarely focused on HTTP. Multi protocol stuff is on the enterprise plans where they make their money.

emmanueloga_
0 replies
1d5h

Why not showing the quote? Perhaps other companies could offer a better deal.

buildbuildbuild
0 replies
1d4h

This might be due to how outrageous Cloudflare’s pricing is for protecting non-HTTP services like SSH (Cloudflare Spectrum.)

andersa
12 replies
1d7h

We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means

I don't get it. Cloudflare proudly advertises unmetered DDoS protection on any plan level. Is that just a lie, or what am I missing here? They don't need to be on a custom Enterprise plan.

tpetry
4 replies
1d7h

They are only doing unmetered DDoS for HTTP traffic - for websites. File storage etc. is not allowed. And git falls into that realm.

andersa
3 replies
1d7h

I see, that explains it.

Deathmax
2 replies
1d5h

To expand on that, Cloudflare's standard product is their HTTP reverse proxy. To proxy arbitrary TCP/UDP traffic, you need to use their Cloudflare Spectrum service (https://www.cloudflare.com/en-gb/application-services/produc...), which is metered.

matteocontrini
0 replies
9h47m

I think they might be looking in something more similar to Magic Transit, but that depends on how they set up their infrastructure.

blibble
0 replies
1d1h

wow $1/GB?

would be cheaper to pay a developer to add websocket support to openssh

donor20
4 replies
1d5h

Unlimited free is almost always a lie.

internetter
2 replies
1d5h

But in this case it literally is? Cloudflare provides unlimited HTTP traffic on all plans, exactly what they claim. Sourcehut needs a different service.

coldacid
1 replies
1d4h

HTTP(S)-only _is_ the limit.

dingnuts
0 replies
1d3h

Yeah, because protecting other protocols would require a totally different approach and product. This isn't a conspiracy.

The Cloudflare HTTP CDN cannot protect SSH any more than a condom will make a good umbrella just because they both are designed for protection!

arrowsmith
0 replies
1d5h

See also "unlimited paid time off".

dinp
1 replies
1d5h

https://www.cloudflare.com/en-gb/plans/

Looks like level 3 ddos protection is only available on the enterprise plan, it's not included in the unmetered ddos protection.

samcat116
0 replies
1d3h

Layer 3*

monero-xmr
9 replies
1d4h

Reminder that source hut makes a value judgement on your source code and will ban repositories they don't like https://news.ycombinator.com/item?id=33403780

plagiarist
5 replies
1d3h

Why do you need to convert "goes against their principles" to "doesn't like" to make your argument? Is it because it is a weak argument without twisting words?

They're allowed to decide what they'll host, and it looks like they're just being clear about a boundary. Unless they're playing favorites and allowing some crypto, what is the problem?

monero-xmr
2 replies
1d3h

I don't want my source code vendor to tell me what source code I am allowed to write

timetraveller26
0 replies
1d2h

They ain't doing that, they are just telling you which code they don't like to host.

plagiarist
0 replies
1d

What does that have to do with getting DDoSed? Because "you don't like them" they deserve to be DDoSed? Or is this just, "BTW, I am salty." Nobody owes you a centralized location for your distributed source code repositories.

seti0Cha
1 replies
1d2h

Goes against their principles is an explanation for why they don't like it, so I don't see anything wrong with saying "if they don't like it". OP is correct and under no obligation for explaining their motives.

plagiarist
0 replies
1d

They and you are both incorrect. You both wish to pretend that because all dolphins are mammals, all mammals are dolphins.

They aren't removing code simply because they don't like it. They have defined a term of service. You can pretend it's more capricious than that, but it isn't the truth unless you have some additional evidence beyond that one thing.

tristan957
0 replies
1d2h

You have an obvious bias given your username. Why should SourceHut involve itself with projects that are at a high risk of being a scam?

red_admiral
0 replies
1d3h

I followed that link to see what "don't like" means, and it means "no cryptocurrency projects". That's +1 to SourceHut in my book.

mwcampbell
0 replies
1d3h

And that's a good thing. A bootstrapped company doesn't have to be a slave to money.

justsomehnguy
9 replies
1d8h

Codeberg went down to DDoS ~12h ago too:

https://status.codeberg.eu/status/codeberg

EasyMark
8 replies
1d6h

I wonder why they're thinking about moving to EU as their compatriot seems to be having the same issue and moving to Europe wouldn't fix it. It was mentioned in their public notice.

karmarepellent
7 replies
1d6h

Their plan to move to the EU via their AMS site is not related to the attacks, but seems to be a general plan they have had for a while. At least this is how I understand their notice.

bayindirh
6 replies
1d5h

They were planning to moving to EU for the last 6-8 months. They were making it gracefully.

This attack expedited it greatly.

mrweasel
5 replies
1d4h

The EU move wouldn't exactly help, if the attack is specifically directed at Sourcehut. That is unless the ISP and hosting provider in AMS is offering some form of protection as part of their offering. I can't see why the attack wouldn't just follow the services.

karmarepellent
2 replies
1d1h

This is certainly possible, but it seems they are anticipating this as well.

One of our main concerns right now is finding a way of getting back online on a new network without the DDoS immediately following us there, and we have reason to believe that it will.
EasyMark
1 replies
19h7m

That doesn't make sense though, as all the DDOS people have to do is aim their bot army at the new site. Probably just a single parameter in their attack scripts. I mean I'm an idiot and I can launch a DDoS on someone, I just don't have the $$ or compromised army of iot devices to aim at accomplishing that, nor really the will to do others harm. Whether you're in Europe or NA doesn't matter.

Sebb767
0 replies
17h28m

If their plan is to get online via the second location, it's likely that said location has a much beefier upstream or built-in filtering, allowing them to absorb these amounts of traffic without being null-routed.

More likely, though, they're restoring the service to a fresh IP range and put the servers behind some kind of DDOS-protection or, alternatively, they simply choose to do the switch now as they need to do a full restore anyway and it's not related to the DDOS mitigation.

bayindirh
0 replies
1d4h

You're right, but they were already planning it, so why not?

Maybe they have another plans under that, a better server or set of servers, some hand-rolled mitigations, etc. I have no idea. I'm a user as everybody else.

alright2565
0 replies
1d2h

I assumed that the addresses for the EU server are not publicly available, so if they get some sort of DDOS protection before bringing them up, then the attackers will not know where to target their attack.

callalex
9 replies
1d8h

Am I correct in assuming these kind of problems are not possible if you are using a major cloud provider instead of renting rack space?

daenney
2 replies
1d7h

No, that is not a correct assumption. It's also worth remembering that cloud providers are also servers in racks, but they own the building around it to.

For DDoS attacks, you need to have enough capacity to absorb the attack. Major cloud providers tend to have that, as do DDoS mitigation services (Cloudflare amongst others).

We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means [..]

Typically what you want to do though is stop the traffic from reaching you at all, so ideally your network provider, who is upstream from you, blocks the illegitimate traffic so your servers never see it and don't get overwhelmed.

What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage. Imagine if your ISP decided to stop routing traffic to Google. Being hosted on GCP, a major cloud provider, would be of no use, since there wouldn't be a network path to them in the first place.

In general, Cogent seems to be doing a rather bad job at dealing with this attack and there's been fallout for many services beyond Sourchut. Google or AWS or Microsoft might've handled it more gracefully, or might not. Though major cloud providers tend to have their own connectivity between their datacenters, they too have peering/transit agreements with other major network providers. If those upstreams stop forwarding traffic to them, the same thing would happen. It's just less likely to go unnoticed.

Cogent is a massive provider, so you'd think they'd be a bit better at this. But they also have a reputation for being awful.

fmajid
0 replies
1d4h

Cogent is also a low-cost provider, and it shows in their customer service.

dewey
0 replies
1d6h

What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage.

That's not what happened based on my understanding. The provider nullrouted their traffic (which is common if a customer is under attack), but Sourcehut couldn't talk to the customer support as their support panel wasn't working for them.

tgsovlerkhgsel
1 replies
1d6h

For any company with a complicated cloud footprint, how do they avoid "denial-of-money" by having the attacker pick some externally reachable piece of the cloud setup where the attacker can take actions that the company pays for (e.g. download large files from a cloud storage bucket, generating huge bandwidth charges)?

miki123211
0 replies
1d5h

This is a major problem, particularly for those offering any kind of outbound SMS capability, whether for 2FA or just phone number verification.

That service is the easiest and most profitable to abuse, there are certain providers in certain countries that price inbound SMS very steeply, and are willing to share the profits with you. If you if you can get an attack going.

preisschild
0 replies
1d8h

Depends if you buy their ddos protection services or not.

eterps
0 replies
1d8h

You are probably correct in assuming these kinds of problems are not possible given enough $$$

esskay
0 replies
1d7h

Still very much possible, in most cases all you're doing at the cloud provider is running virtual hardware through load balancers so they can very much become overwhelmed. Even with more advanced setups, it's still all just hitting a variety of servers, each of which can be affected in different ways.

There are (expensive) ways of mitigating but a project like Sourcehut couldn't afford or justify what will likely be a 5 to 6 figure sum.

JoshTriplett
0 replies
1d7h

If you're using a major cloud provider, a DDoS might mean either that your service hits scaling limits or that you get a massive bill.

cutner
8 replies
1d7h

Quite ironic when you consider that Drew is a fan of DDoS attacks and transit provider blackholing when it's aimed at websites he doesn't like.

tristan957
7 replies
1d2h

When has Drew said he was a fan of DDoS attacks? When you make claims like these you should really bring receipts.

SourceHut doesn't blackhole content. It is in their terms of service and documentation what they will not accept on their website.

gavinhoward
6 replies
1d2h

Not GP.

I agree with you that there is no evidence that Drew likes DDoS.

However, his ToS are sufficiently vague to allow him to deplatform people he doesn't like. And he said publicly that he will. [1]

[1]: https://news.ycombinator.com/item?id=33416823

eesmith
5 replies
23h20m

Yes, he doesn't like cryptocurrency projects. I don't see much vague about it.

The other clauses about bigotry are pretty much aligned with GitHub's ToS, which has "GitHub does not tolerate speech that attacks or promotes hate toward an individual or group of people on the basis of who they are, including age, body size, ability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, sexual identity, or sexual orientation", and "We have the right to refuse or remove any User-Generated Content that, in our sole discretion, violates any laws or GitHub terms or policies."

While people who don't like these policies call it "deplatforming", in the US the freedom of association is a protected First Amendment right, with only limited exceptions for certain protected classes like race, ethnicity, and national origin where there is a strong tension between the First Amendment protections and other constitutional protections.

I don't know about Dutch law though.

gavinhoward
4 replies
19h20m

In private communication, he told me that he would ban me for being Christian.

Yeah, his policies are "aligned" with GitHub's (and I avoid that too), but Drew is on the record saying he will ban people for their opinions, not their actions.

eesmith
1 replies
12h23m

I'm missing a lot of context.

A whole lot of people are Christian, so if this were true we should see evidence of this happening by now on Sourcehut.

If the ban was for membership in an "I am not a Christian" club, then that ban would be entirely reasonable.

How does he learn about any of your opinions without any sort of action? You must express the opinion first, and the act of expressing an opinion is an action.

I can think of many times when people were fired or "de-platformed" due to their opinion, like Jimmy Snyder (a.k.a. "Jimmy the Greek") - https://en.wikipedia.org/wiki/Jimmy_Snyder_(sports_commentat... , so it isn't like banning someone due to expressing a specific opinion is inherently unconscionable.

gavinhoward
0 replies
2h54m

There are a whole lot of different kinds of Christians. I am a traditional Christian (not a fundamentalist) and tend to be more outspoken.

But doesn't that prove my point? You say that if I said an opinion, I made an action, so you are saying that Source Hut is right to ban me for opinions.

But you and I fundamentally disagree: banning someone for their opinion is not inherently conscionable.

You may find yourself with a "wrong" opinion someday and be banned from places you find important. Will you find it conscionable then?

By the way, expressing an opinion was not the kind of action I meant. I meant doing something that might harm others. My opinions do not call for harm (in fact, they call out what I see as harm), so my opinions cannot cause harm except by some broad definition that "speech is violence."

So why is it okay to ban me? Free Software needs Free Speech. SourceHut, as Drew has admitted, is not the place for that.

drewdevault
1 replies
10h20m

Good morning, Gavin. You were shown the door because you are an outspoken transphobe and we were not interested in helping you voice those opinions. You seem to be unable to disentangle this with your Christian identity, but most Christians seem to manage alright.

gavinhoward
0 replies
3h1m

Because there are different kinds of Christians, Drew. And a lot of the ones I have seen aren't really Christians.

Also, I am not scared of trans people, so why the terrible term? Oh, that's right: it's a pure smear.

But doesn't that prove my point? You would ban me for my opinion even though I have actually done nothing!

Free Software needs Free Speech, Drew. Even for people whose opinions you don't like.

MichaelMug
8 replies
1d8h

1. Do "PHL, FRE, and AMS" mean something? Or are these just codenames for each site?

2. If I host a service on AWS, Azure, Linode, DigitalOcean am I also susceptible to layer 3 DDoS?

cr3ative
3 replies
1d7h

They're generally airport codes, so Philadelphia and Amsterdam... unsure about FRE.

Y_Y
2 replies
1d5h

The IATA code FRE is assigned to Fera in the Solomon Islands. Freemont Airport does not have an IATA code.

fmajid
1 replies
1d4h

Naming conventions can be inconsistent, and not every datacenter location has an airport. ASD is often used instead of AMS for Amsterdam.

Y_Y
0 replies
1d4h

ASD is the IATA code assigned to Andros Town Airport, in the Bahamas. It might also be a commonly used abbreviation for a medical condition.

lftl
0 replies
1d6h

Linode went through a rather long DDoS attack a few years back with a few of their data enters being offline for a few days, so I would guess yes there.

gslin
0 replies
1d7h

Their topology page has information about PHL/AMS and FRE (seems like Fremont datacenter):

https://web.archive.org/web/20240111132224/https://man.sr.ht...

grudg3
0 replies
1d6h

To answer your 2nd question, yes you can be DDoSd, Azure specifically offers a DDoS protection plan which is quite expensive. https://learn.microsoft.com/en-us/azure/ddos-protection/ddos...

emersion
0 replies
1d7h

1. PHL means Philadelphia, AMS means Amsterdam. Yes, just codenames for sites.

2. Depends if DDoS protection is part of the offer, I suppose.

MartijnBraam
7 replies
1d7h

I wish the sr.ht employees the best of luck with the AMS migration. Luckily git is decentralized enough that I've been able to practically do everything the last few days except that I can't push new releases of software to the canonical upstream repository.

mariusor
3 replies
1d5h

Erm... AWS migration? What am I missing?

A DDoS on AWS would most likely bankrupt Drew and Sourcehut, even if Amazon managed to absorb the traffic. Not to mention the principial issues that they have with AWS in the first place.

emersion
1 replies
1d5h

AMS ≠ AWS :P

mariusor
0 replies
1d5h

Erm, apologies, thank you for the correction. I'm embarrassed. :)

M2Ys4U
0 replies
1d5h

Erm... AWS migration? What am I missing?

It's A M S, not A W S.

internetter
2 replies
1d5h

Also, the fact sourcehut does not baby users about git send-email means that collaborative development happening on the platform is uniquely posed to continue

colinsane
1 replies
1d4h

sure, i can `git send-email` to the particular developers i've collaborated with before, but by default my send-email goes to just the list-serve at `~user/project@lists.sr.ht`. pulling out `nc lists.sr.ht 25` seems to show that it's also offline.

so, yes for really important stuff i could get patches through to projects i care about. but in practice the handful of projects i'm involved in on sr.ht seem to be mostly stalled.

internetter
0 replies
1d4h

Yeah. My comment was largely optimistic, in practice people would rather just wait for service.

That being said, I did send a small patch directly by email just for the fun of it.

okasaki
2 replies
1d8h

This site can’t be reached

DNS_PROBE_FINISHED_NXDOMAIN

worstname
0 replies
1d8h

I can reach it but here's an archive: https://archive.is/wVtqN

eterps
0 replies
1d8h

I does work for me, but more people seem to experience that problem:

https://fosstodon.org/@drewdevault/111742324107487646

jordemort
2 replies
1d2h

Oof, my heart goes out to them. My very first week at [large popular public code forge], we were attacked by [state level actor], probably for hosting something that [state] didn't like. In a way, SourceHut and Codeberg getting this kind of attention is an encouraging sign that these alternative forges are starting to gain traction.

rapnie
1 replies
1d1h

I wonder how you could figure out it was [state]. Was there some clear threat made? A blackmail "do this, or else.."?

meredydd
0 replies
23h57m

This is likely a reference to GitHub, which was DoSed by the Chinese government for hosting a repository that was mirroring uncensored Western media on a domain that could not be blocked by the Great Firewall without hobbling the Chinese software industry.

The attribution wasn't subtle - a substantial fraction of Baidu's ads/analytics traffic served to domestic Chinese users was rewritten to hammer that specific repository directly.

NYT coverage at the time: https://www.nytimes.com/2015/03/31/technology/china-appears-...

127361
2 replies
23h22m

I suspect a possible DDoS attack from something Kiwi Farms related, it seems they really don't like the guy over there?

https://archive.is/yOObX

mardifoufs
1 replies
22h6m

If kiwifarms ddosed everyone that they didn't like... plus, if anything KF has been surprisingly resilient to massive DDoS without any 3rd party protection service since they lost CloudFlare and other providers. So for this one specific thing, I think lessons can be learnt from kf on how to mitigate DDoS without 3rd party providers.

127361
0 replies
22h1m

They developed their own proof-of-work system to rate limit requests, using SHA256, implementation details here: https://archive.is/dfBVN

They also have servers in multiple countries. Some of these servers being blackholed by Cogent, by the way.

betimsl
1 replies
1d1h

“what if the primary datacenter just disappeared tomorrow?” We ask this question of ourselves seriously, and make serious plans for what we’d do if this were to pass, and we are executing those plans now – though we had hoped that we would never have to.

Respectfully, if you guys asked this question...how come you don't have a cluster slave as a replica in another data center where the whole thing is synced?

A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.

Rantenki
0 replies
1d

A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.

If you want a Klugey non-production IoT solution that bodges up something really important THESE days, all the cool kids are using ESP32s.

And as much as I think that would be a totally inappropriate solution for src.ht, I kinda wanna go make a "black-hole" switch for my office.

throwawaaarrgh
0 replies
1d1h

So it turns out that running your own servers isn't always the best idea

sytse
0 replies
1d

HugOps to the Drew, the rest of the Sourcehut crew, and anyone else working on this. Compliments on the clear blog post.

simonebrunozzi
0 replies
1d

My name is Drew, I’m the founder of SourceHut and one of three SourceHut staff members working on the outage, alongside my colleagues Simon and Conrad. As you have noticed, SourceHut is down. I offer my deepest apologies for this situation. We have made a name for ourselves for reliability, and this is the most severe and prolonged outage we have ever faced. We spend a lot of time planning to make sure this does not happen, and we failed. We have all hands on deck working the problem to restore service as soon as possible.

Drew, you're great, Simon and Conrad are great, you'll get through this, and you will be fine.

Keep doing your great work, forever.

scopeh
0 replies
1d6h

Feel bad for the team to have to deal with this, but I have to say its a great example of how to communicate with your customers so hats off to you!

pierat
0 replies
1d4h

The real led in the story:

Company (cloudflare) that protects DDoS'ers and booters charges insane protection racket for DDoS'ers and booters.

Relevant article: https://rasbora.dev/blog/I-ran-the-worlds-largest-ddos-for-h...

Phelinofist
0 replies
1d5h

Sad to see, best of luck to them