return to table of content

My cat alerted me to a DDoS attack

fragmede
12 replies
6d18h

The princely sum of $5,000. We got that at my employer back in 2016. We got hit by a ddos, and decided to ignore it, though we did dig up some BTC just in case. We enacted a bunch of DDoS protection as a result, costing way more than $5,000, but not paying money to extortionists is worth every penny.

vsnf
10 replies
6d18h

The problem with paying extortion or ransoms is that you incentivize the attacker to come back and do it again. It may have been $5k to pay off one attacker and more than that to build the defense, but now you have defenses and are less likely to suffer attackers in the future. And as you say, not paying money to criminals is inherently worthwhile.

aleksiy123
6 replies
6d18h

On the other hand the attacker may actually have incentive to follow through and hold up their end so as to build a reputation. Making their next victims more likely to just pay.

Somewhere I read that some ransomware had excellent "customer" service for helping you transfer over the payment and promptly restore your files.

wruza
1 replies
6d12h

This stat includes everyone, some of them may not research what hit them. One of my clients decrypted his database twice, after seeing on the internet that they actually send a decryption key.

sedatk
0 replies
6d10h

Oh, got it. So, the next ransomware author should just brand their ransomware as one of the "honest" ones :)

soraminazuki
1 replies
6d17h

Scammers acting like an adult is hard to believe. They’re usually quick to start yelling and cursing in Kitboga videos.

dhosek
0 replies
6d16h

The serious organized crime outfits are very organized. They’ll provide customer support to walk you through purchasing and transferring the bitcoin.

mort96
0 replies
6d11h

I mean it's not like these people are operating under " long lived identities and want to build a long term business relationship with you.

From their side, the logic seems as simple as: repeat the attack -> some chance of more money, don't repeat the attack -> 0% chance of more money

thaumasiotes
0 replies
6d13h

It is often a temptation to a rich and lazy nation

To puff and look important and to say

Though we know we should defeat you

We have not the time to beat you

We will therefore give you cash to go away

And that is called paying the Dane-Geld

And we've proved it again and again

That if once you have paid him the Dane-Geld

You never get rid of the Dane

fragmede
0 replies
6d17h

and this wasn't a poor company, so $5,000 was nothing in the scheme of things

jnsaff2
0 replies
6d11h

About 20 years ago I was kinda accidentally the guy who dealt with the DDoS attacks in the sysadmin team. There was a sequence of extortion emails during about 2 week period:

1. $50k or we attack - didn’t register anything

2. $25k or else - a minor overload on the server but nothing serious.

3. $10k or else - a serious attack which affected the service in a major way.

4. $5k or we really pissed - this time they took down a whole Tier2 ISP and Datacenter in London for a day. Other carriers peering on London Internet Exchange had to blackhole traffic to our service provider and finally kept blackholing one of our IPs for a while. I had to scramble to find a DDoS mitigation service, new DC and servers.

We did not respond to any of the emails. The attackers were also quite dumb, they attacked the web servers which were located in a well connected place.

The money making service of the business was in the Caribbean with a 1,5Mbps T1 and a 0,5Mbps satellite backup. They could have saturated those much easier for much longer and the impact then would have been about $1M revenue loss per hour.

exabrial
10 replies
6d16h

We don’t have very many earthquakes in Kansas… but I remember the first/only one I felt.

I was sound asleep when my Siamese woke me up by pawing my face… he then went and sat on the edge of the bed and growled aggressively (very out of character)… Not 30s later, things started shaking.

No idea how he knew, but it was pretty wild. He passed away in 2020, still miss him.

jmprspret
3 replies
6d16h

Cats and dogs have been known to feel/sense earthquakes before we can!

In the recent NYC ones there are videos of dogs howling before any of the tremors are noticeable by people. This is a common phenomena I believe.

flakes
0 replies
6d13h

Yeah 100%. I live in NYC, and before the quake, our golden retriever started crying and whimpering. A few moments later we felt the quake!

cqqxo4zV46cp
0 replies
6d7h

Homer Simpson: “Somehow the animals are always the first to know”.

btilly
0 replies
6d14h

Yes, it is common. See my sibling comment explaining it.

btilly
3 replies
6d14h

I know how he knew.

There are two types of sound in rock. P and S waves. P waves are pressure waves and go faster. S waves go side to side and are a bit slower. So you cat was woken by a hiss from the P waves, which arrive a bit before the earthquake that you can feel.

See https://manoa.hawaii.edu/exploringourfluidearth/physical/oce... to verify that there are two types of waves, and the P waves arrive first.

earth2mars
1 replies
6d13h

This seems like a good sensor to build for early detection. Is that what the sensors do?

pezezin
0 replies
6d7h

I have been living in Japan for a while, and in my experience you can feel the earthquake coming a few seconds before the big shaking start. I don't know how to describe it, it is not a sound, more like a very slight movement.

nyjah
0 replies
6d15h

Dang, sorry for your loss. That’s a dope memory of the cat tho. As someone that happened find themselves in Taipei a couple weeks ago for the 7.4, my only thought was getting back to my dog, whom I promised I would get back to. She was sorta freaking out before I left; either could sense me leaving, or sense the earthquake I was heading to…

johnnyAghands
0 replies
6d15h

My condolences, what a good boy :(

lmm
9 replies
6d20h

Maybe the phone was silent but still flashing a screen? Mine does that in that mode.

At my first job we had a guy who could spot incidents coming on the monitoring dashboard before they happened. He never managed to explain or even understand what he was looking for and no-one else picked it up, but he would just see something that made him say things were odd, and most of the time we'd get an alert shortly after.

Waterluvian
3 replies
6d17h

Make or get a human to stare at streams long enough and they’ll attune to the patterns. We’re wired for patterns. It doesn’t even have to be conscious and explainable. The signals just suddenly aren’t right.

brookst
1 replies
6d15h

My s/o is convinced she has a tell when we play rock / paper / scissors, and maybe she does, but if so I don’t know it. I just know that if I observe her closely and don’t make a conscious decision about what to throw, I win 80% of the time.

Somewhere deep in my brain, there are neurons that developed for some more evolutionarily-relevant purpose and which are now a little disgusted with how they’re being used.

zmgsabst
0 replies
6d15h

The perennial example being lab techs/equipment operators and machine hum.

leo150
1 replies
6d12h

Wasn’t the name of that guy Colin Laney

defrost
0 replies
6d12h

Working solo w/out Frank Woodley?

vrighter
0 replies
2d13h

happened to me when playing cyberpunk 2077 on psf ataunch (shudder).

I got to a point where I could reliably tell "the game is about to crash, better save." I save, and 10 seconds after resuming the game, it crashes. I still don't know how I could tell.

seanthemon
0 replies
6d19h

We call those guys the canaries and we keep them deep in the mineshaft

praptak
0 replies
6d11h

Maybe some signals just bypass the part of the brain which deals with well defined facts. I read somewhere about a construction foreman (HN comment maybe?) who gained the respect of the crew by having an unusually good hit rate in finding piping in the ground or walls. He started to believe in his superpowers but later came to the conclusion that he just subconsciously learned the typical patterns, plus an occasional non-obvious sign. Something like a vent pipe in the wall of the building telling you that sewer piping is probably below ground.

ahmedfromtunis
9 replies
6d19h

With horrible grammar

Ah, the days before ChatGPT!

On a more serious note, do you think there will ever be a way to stop ddos attacks once and for all?

While all threats are bad, ddos is the most lame type of attacks there is; no special skill or knowledge are needed, just load a script or, heck, pay someone who'll execute it for you as a service.

toast0
3 replies
6d16h

There's application level DDoS, which you generally stop by not doing expensive work for clients that haven't done expensive work for you. Sometimes, easier said than done.

And then there's volumetric DDoS. You can stop this by having more bandwidth than everyone else... but that's pretty hard and it makes you a potential attacker.

Innovation here is in the form of using BGP to disseminate traffic filters. Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible. But I've seen there's some systems with more precision, like drop udp, drop fragments, drop packets to/from udp/tcp port X.

Most of these systems are designed so that these specialized routes don't propagate beyond immediate peers, but potentially, it might be desirable if they did.

kijin
2 replies
6d14h

Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible.

Oh, this poor guy is being DDoS'd, so we're going to make sure that their service remains denied.

Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.

With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.

xorcist
0 replies
6d10h

So the suggestion here is to implement source routing on a whim, with a table of 100+k entries? At your peering point?

An ISP has to do what it has to do to save their business when a large attack hurts their business. This may be what the attacker wants but that's not an excuse to do nothing.

But what machine learning has to do with this is not clear. Null routing protects you against traffic volume. Machine learning sounds like it maybe can help diagnose more sophisticated low-volume attacks. Maybe. You don't want anything remotely compute intensive when mitigating attacks.

toast0
0 replies
6d14h

Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.

If you're running on a single IP, yes. If you're running on multiple IPs, it's not that bad for the one that's being attacked to get its traffic dropped and everything else works. It's not great, but what are you going to do. If you've got enough traffic to overwhelm the inbound on the top of rack switch your box is on, you're not going to be able to really serve any of the good traffic anyway.

With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.

There's usually way too many source addresses to do that, and anyway, routing infrastructure is geared towards looking at destination addresses, not source addresses. Also, each individual source doesn't look that bad --- if I've got 10,000 sources each sending me 1 Mbps of garbage, nobody is going to accept a block for only 1 mbps of sending, and yet, there's 10 Gbps of garbage arriving at my box; if I've got 10 Gbps or better connectivity, no big deal. But, if I'm only on 1 Gbps, I'm getting less than 1 in 10 of my inbound packets. I'd argue, if everything else has a big enough connection, it's probably still no big deal, it should be able to drop packets headed to me, as long as its upstream connection isn't filling up. But once abuse is causing contention that impacts others on my rack, it's probably time to null route.

If it's one of the big botnets with 100,000+ compromised systems, the individual bandwidth is even less. And if the botnet has significant ability to deliver spoofed traffic, source based filtering is meaningless. If it's reflected DDoS, I dunno --- there's value in hunting down the chargen services and removing them from the internet, but that's usually a lot more work.

OTOH, look on the bright side, if your outbound bandwidth is high and you get a lot of inbound DDoS, you may have roughly balanced your usage, and you may qualify for settlement free peering! (IMHO, this has got to be a major part of Cloudflare's business plan)

squarefoot
1 replies
6d8h

> With horrible grammar

Ah, the days before ChatGPT!

The topic made me read that as CatGPT, and now I can't pull it out of my head.

tgsovlerkhgsel
0 replies
6d16h

It's not as simple as "loading a script" - IP addresses (or in the case of IPv6, subnets) are (for the average person) a limited resource, as is bandwidth, and most amplification attacks require IP spoofing which is not possible from most connections.

If it's a volumetric attack, the side with more bandwidth wins (the attacker may be able to amplify here). If it's a load-based/application-level attack, blocking the attacker IPs at the firewall level solves it. This was application level, not (purely) volumetric, since they already had a WAF/Cloudfront.

Identifying attacker IPs to block is a matter of correctly attributing cost to a source IP, correctly attributing benefit (i.e. legit user activity) to a source IP, then blocking the IPs or ranges where the cost significantly exceeds the benefit you see from that IP or range.

That's easier said than done, since cost can come in many forms (e.g. open connections clogging up memory, TLS handshakes, requests that are expensive to parse for your web server, requests that trigger expensive database queries, in/out bandwidth, ...) which is why most just slap Cloudflare (or here, Cloudfront) in front of it and work around with manual rules like in this example.

flafla2
0 replies
6d18h

Cloudflare does a pretty good job of managing it, at the cost of some centralization.

It would be pretty cool if there was a way to DDOS-harden at the protocol layer. Not sure if that’s even possible though

bee_rider
0 replies
6d18h

Maybe if the network was much more distributed and lower bandwidth?

If most of your customers are in Mexico, Canada is DDoSing you, and the pipes between you and Canada start filling up as a result that isn’t a big problem, right? As long as consumer routers on you/Mexico’s side of the Canadian clog don’t decide to help out.

EveryPizza
9 replies
6d18h

Quite some time ago, someone from my family was alerted by their cat when the dishwasher was leaking. Their conclusion was that the cat was either trying to save them or the cat was trying to kill them.

macintux
7 replies
6d18h

One of my all-time favorite novels, Anansi Boys by Neil Gaiman, includes an anecdote: a crow's call wakes up someone who's sleeping outdoors, just as a large cat (a tiger, perhaps) is sneaking up on him.

One character suggests the crow was trying to warn the man. Another posits the bird was bringing the sleeper to the tiger's attention so it could enjoy the scraps after the meal.

derefr
3 replies
6d16h

Odd that the most obvious hypothesis wasn't given: the crow was probably using its alarm call to warn other crows. Crows are social animals, who care about the fates of their "friends and acquaintances" — so they would do that.

But also, on a tangent, there is a bird that does this kind of non-conspecific alarm calling the time as part of its food-gathering strategy: the African fork-tailed drongo.

The drongo gives true alarm calls to food-rival species nearby, to tell them when it has spotted a mutual predator. This leads to these food-rival species coming to rely on these signals. But then, every once in a while, it gives a false alarm, to get the food-rivals to run away for a bit, so it can nab the bugs/berries/etc that the rival would have been eating.

hiddencost
1 replies
6d13h

It's called an allegory

card_zero
0 replies
6d12h

No, a drongo.

thaumasiotes
0 replies
6d13h

There's also an African bird whose foraging strategy includes alerting large mammals to the presence of food that they can harvest and the bird can scavenge.

In fact, there is a family of such birds, the honeywarblers, who locate beehives, then find humans and lead them to the beehives.

Wikipedia says that the behavior is dying out because there aren't enough human foragers.

jmprspret
2 replies
6d16h

There is another Neil Gaiman short story about a cat who goes gallivanting every night and comes back every morning all scuffed up. The owners don't know why since there aren't any other cats around. One night the owner can't sleep and discovers the reason.

Don't want to spoil. It's nice and short and a must-read for cat lovers. "The Price".

http://www.bitchwick.com/amacker/bean/price.html

melodyogonna
0 replies
6d11h

Good read

herodoturtle
0 replies
6d11h

That was a terrifyingly poignant read, thank you.

readyplayernull
0 replies
6d16h

the cat was either trying to save them or the cat was trying to kill them.

An inverted Schrödinger cat.

jameshart
8 replies
6d20h

As always it’s easy to overlook the insider threat. Grammatically dubious extortion emai? Bitcoin ransom? Did it not occur to you that the cat was the one behind the attack?

ordu
3 replies
6d20h

Yeah, cats are notoriously bad at grammar.

addicted
1 replies
6d16h

You do know that’s just to throw humans off their scent right?

Everyone believes they have bad grammar so when they launch the really serious attacks we all think it was a state agency rather than the cats.

cqqxo4zV46cp
0 replies
6d7h

You mean to say that a Nigerian prince is responsible for the xz backdoor?

smarks
0 replies
6d20h

I CAN HAZ DOS ATTAK

shawn_w
1 replies
6d20h

The call is coming from inside the house!

jvm___
0 replies
6d11h

The meow is coming from inside the house

nusl
0 replies
6d9h

Putting the "Bite" in "Bitcoin"

cocoa19
5 replies
6d20h

And I often wonder if on call is justifiable “because you make more money than most professionals”.

tossandthrow
2 replies
6d20h

as with most roles, I think it is negotiable. You have your professional leverage, expected pay and grit. you need to balance these things.

Also, if you can get an equivalent role with less requirements such as being on call, then I guess it is just a question of grabbing it!

hughesjj
1 replies
6d20h

I mean, you have oncall, it's just permanent oncall.

krab
0 replies
6d12h

Not really. If I don't agree to on-call, I do it on best-effort basis. That is: "Oh, I'm camping with kids without a computer. I'll try to help you as much as I can. Did you try Z after X and Y didn't work? Ok, try it and call me back how it went, I'll try to Google something in the meantime." If that would happen too often (more than 1-2x a year), I would try to improve the process or consider switching my job. And my phone is silent during the night.

I wouldn't call this a permanent on-call, just being responsive.

willsmith72
0 replies
6d16h

2 European teams I worked on paid a bonus for on-call duty, and the systems were so stable that enough people volunteered for the few who didn't want it, weren't forced to.

It was pretty great, I took a week shift every month or so except when I was going on holiday, and aside from lugging a backpack with my laptop everywhere, didn't affect my life at all except 1 or 2 minor issues

krab
0 replies
6d12h

You usually get some extra money for the duty. And if you get woken up, the hours you spend are counted towards your normal working hours - so you aren't expected to show up in the morning after putting out a fire. Or you get some more bonus (like 2x hourly pay for the night work). That's about the balance when people are ok doing it.

But it depends on stability of your service. If it is messed up and people are woken up often, then you won't find many volunteers if they have other choice.

swampthinker
4 replies
6d20h

And here I thought your somehow hooked up a cat feeder to alerts.

Regardless, very cute - what’s your cat’s name?

dguo
1 replies
6d17h

Writing this post did make me think that if someone had a well-trained dog, they could hook up a monitoring service to something that makes a particular sound, which tells the dog to alert the person.

Her name was (I sadly lost her to cancer) Bamboo! Because one of the first things she did after I adopted her was to try to eat my bamboo plant.

codetrotter
0 replies
6d16h

And as an added bonus, we could get that dog classified as a service dog :D

“Sir you need to leave that mutt outside!”

“He’s a service dog”

“Why? You don’t look like you have any disabilities”

“Wow. First of all – rude! Second, yeah you are right I don’t but you see he’s my DDoS dog and I need him with me at all times to protect the company servers”

chris_wot
0 replies
6d18h

Funny, that's how the very first customer realised that the Australian telco Optus was down. The wireless cat feeder relied on the Internet and when no food appeared, the cat decided to complain to management.

867-5309
0 replies
6d19h

proposing Danielle of Purrvice

jart
4 replies
6d20h

It's so easy to crush ddos with token buckets that usually the only thing I need my cat to wake me for is when my Discord gets raided.

avg_dev
2 replies
6d19h

never heard of this before. I looked it up https://en.wikipedia.org/wiki/Token_bucket

I think this would be like a firewall or ingress thing that would drop packets that resulted in excess load before they make it to the application server.

sakopov
0 replies
6d19h

It's a common rate limiting algorithm. Here's an interesting article from Stripe on how they use it in their APIs. [1]

[1] https://stripe.com/blog/rate-limiters

hnlmorg
0 replies
6d12h

some types of DDoS. ;)

You could still overload the service with a sufficiently large attack in either volume of connection requests or number of unique IP addresses.

Token buckets are usually part of an overall resilience strategy rather than a silver bullet to solve all denial of service concerns.

Denvercoder9
4 replies
6d20h

we didn’t have a formal on-call rotation yet. That was a deliberate decision, since being on-call is painful, and the team was good about just collectively keeping an eye out for urgent alerts.

That seems like a terrible solution. Yeah, being on-call is painful, but at least I know beforehand when I'll be on-call and get compensated for it. Always being expected to keep an eye out for urgent alerts just sucks all around.

dguo
2 replies
6d17h

I know it sounds bad, but in practice, it really did work fine for us for quite a while.

1. We didn't experience that many incidents that couldn't wait until working hours.

2. There was never an explicit expectation to keep an eye out. We did it anyway because we were at an early-stage startup, and we all deeply cared about making our products work for our customers.

krab
1 replies
6d12h

I know this from a few startups and it really is not that bad. You really triage what should wake you up and what's ok until morning. It works well as long as the technical founder is ok playing a goalie and essentially being always on call (even though others catch a lot of alarms).

It stops working when the company grows and no one understands the whole system and you need on-calls from several teams. Then the company does some formal on call rotation and it's fine again. It hurts during the transition only.

dguo
0 replies
6d3h

Good point about the technical founder. That was certainly the case for us, as our CTO handled many issues himself.

bongodongobob
0 replies
6d20h

Yeah that sounds like on call all the time, that makes no sense.

AtlasBarfed
3 replies
6d19h

Is this an ad for AWS?

xeromal
1 replies
6d14h

Tech forum

Talks about useful tech

"iS tHiS An aStroTurFing Ad"

AtlasBarfed
0 replies
3d23h

The entire article was "I have all these problems, and I use one small trick (aws product) to fix it all!"

It does +1 most of those types of spam farms with a bit more technical discussion, but not really that much.

fragmede
0 replies
6d18h

Kinda reads like one, but if he was on GCP and used their ddos shield then it'd read like an ad for their service instead. Would be better if he'd been a bit more abstract and said cloud provider instead of naming AWS.

mmahemoff
2 replies
6d20h

I thought it was going to be a home server that went into overdrive, heating the room your cat was in or knocking out the aircon.

Anyway, better experience than being woken up by a dozen SMS alerts.

hunter2_
0 replies
6d18h

Cats love to hang out in warm areas, even sunbathe, so I doubt they'd do anything to get attention in that situation! Their body temperature is a few degrees warmer than that of humans.

Operyl
0 replies
6d17h

Cat would be in bliss then, warm things are their new beds.

ed_mercer
2 replies
6d19h

We didn’t reply, though in retrospect, it could have been fun to try to troll them.

Not replying is the only valid answer. Trolling them could potentially put you more on their radar and get targeted for other attacks. And for what?

dguo
0 replies
6d17h

Yep, I know it could only invite more trouble. It was fun to fantasize about though, especially after watching the video I linked to in the post: https://www.youtube.com/watch?v=dWzz3NeDz3E

benraz123
0 replies
6d17h

For the lulz

ro_bit
1 replies
6d20h

So that's why they want us to microchip our pets!

peterburkimsher
0 replies
6d15h

So they can connect to CAT-6 Ethernet.

avg_dev
1 replies
6d19h

But in 9 years, that was the only time she did it while I was sleeping.

... that you know of

nullderef
0 replies
6d5h

Tangential question that rose up regarding availability vs. quality of life.

For a small startup whose products are only available on the US, does it always make sense to do nightly oncall? This doesn't work for some products, but if, for example, you have a site that sells mattresses in the US, would you wake someone up to fix the site at 3AM?

I guess here the main $$ loss would come from accepting so much traffic. But I wonder if we can better differentiate what's worth waking up for.

matricaria
0 replies
5d13h

I had an old set of PC speakers which always made as weird sound a few seconds before a new message arrived on my phone.

johnnyAghands
0 replies
6d15h

You might say, the cat es-cat-lated it...

fuzztester
0 replies
6d18h

cattackstic!

cattackstrophic!

euroderf
0 replies
6d3h

Well, "attack" is just "cat-kat" spelled sideways.

dontdieych
0 replies
6d11h

It's translated by duckduckgo.com's chatgpt interface. don't down vote plz :D

I suspect that I am somewhat sensitive to electromagnetic fields and magnetic fields. There have been times when I have not felt well the next day after sleeping on an electric heating pad, and I have experienced severe discomfort after sleeping on a mattress with magnets.

When I used a CRT monitor, I often had diarrhea if I spent a long time in front of the monitor.

Since using LCD monitors or laptops, those symptoms have disappeared.

When I sleep, there is a wireless router on the right side of my head, and I play youtube videos on my smartphone on the left side. I have strange dreams and wake up early from sleep. However, if I put the smartphone on the right side of my head while sleeping, those symptoms are lessened.

Thus,

Even though there was no sound, wouldn't your cat have sensed that as well?

com
0 replies
6d7h

We once detected a DDOS because all our office phones went down. Silly attackers didn’t realise that our (money-making) APIs weren’t colocated with our public website and phone system.