The princely sum of $5,000. We got that at my employer back in 2016. We got hit by a ddos, and decided to ignore it, though we did dig up some BTC just in case. We enacted a bunch of DDoS protection as a result, costing way more than $5,000, but not paying money to extortionists is worth every penny.
We don’t have very many earthquakes in Kansas… but I remember the first/only one I felt.
I was sound asleep when my Siamese woke me up by pawing my face… he then went and sat on the edge of the bed and growled aggressively (very out of character)… Not 30s later, things started shaking.
No idea how he knew, but it was pretty wild. He passed away in 2020, still miss him.
Cats and dogs have been known to feel/sense earthquakes before we can!
In the recent NYC ones there are videos of dogs howling before any of the tremors are noticeable by people. This is a common phenomena I believe.
Yeah 100%. I live in NYC, and before the quake, our golden retriever started crying and whimpering. A few moments later we felt the quake!
Homer Simpson: “Somehow the animals are always the first to know”.
Yes, it is common. See my sibling comment explaining it.
I know how he knew.
There are two types of sound in rock. P and S waves. P waves are pressure waves and go faster. S waves go side to side and are a bit slower. So you cat was woken by a hiss from the P waves, which arrive a bit before the earthquake that you can feel.
See https://manoa.hawaii.edu/exploringourfluidearth/physical/oce... to verify that there are two types of waves, and the P waves arrive first.
This seems like a good sensor to build for early detection. Is that what the sensors do?
I have been living in Japan for a while, and in my experience you can feel the earthquake coming a few seconds before the big shaking start. I don't know how to describe it, it is not a sound, more like a very slight movement.
Dang, sorry for your loss. That’s a dope memory of the cat tho. As someone that happened find themselves in Taipei a couple weeks ago for the 7.4, my only thought was getting back to my dog, whom I promised I would get back to. She was sorta freaking out before I left; either could sense me leaving, or sense the earthquake I was heading to…
My condolences, what a good boy :(
Maybe the phone was silent but still flashing a screen? Mine does that in that mode.
At my first job we had a guy who could spot incidents coming on the monitoring dashboard before they happened. He never managed to explain or even understand what he was looking for and no-one else picked it up, but he would just see something that made him say things were odd, and most of the time we'd get an alert shortly after.
Make or get a human to stare at streams long enough and they’ll attune to the patterns. We’re wired for patterns. It doesn’t even have to be conscious and explainable. The signals just suddenly aren’t right.
My s/o is convinced she has a tell when we play rock / paper / scissors, and maybe she does, but if so I don’t know it. I just know that if I observe her closely and don’t make a conscious decision about what to throw, I win 80% of the time.
Somewhere deep in my brain, there are neurons that developed for some more evolutionarily-relevant purpose and which are now a little disgusted with how they’re being used.
https://en.wikipedia.org/wiki/Microexpression might be one explanation.
You can somehow subconsciously detect those in your SO without doing it on purpose.
The perennial example being lab techs/equipment operators and machine hum.
Wasn’t the name of that guy Colin Laney
Working solo w/out Frank Woodley?
happened to me when playing cyberpunk 2077 on psf ataunch (shudder).
I got to a point where I could reliably tell "the game is about to crash, better save." I save, and 10 seconds after resuming the game, it crashes. I still don't know how I could tell.
We call those guys the canaries and we keep them deep in the mineshaft
Maybe some signals just bypass the part of the brain which deals with well defined facts. I read somewhere about a construction foreman (HN comment maybe?) who gained the respect of the crew by having an unusually good hit rate in finding piping in the ground or walls. He started to believe in his superpowers but later came to the conclusion that he just subconsciously learned the typical patterns, plus an occasional non-obvious sign. Something like a vent pipe in the wall of the building telling you that sewer piping is probably below ground.
With horrible grammar
Ah, the days before ChatGPT!
On a more serious note, do you think there will ever be a way to stop ddos attacks once and for all?
While all threats are bad, ddos is the most lame type of attacks there is; no special skill or knowledge are needed, just load a script or, heck, pay someone who'll execute it for you as a service.
There's application level DDoS, which you generally stop by not doing expensive work for clients that haven't done expensive work for you. Sometimes, easier said than done.
And then there's volumetric DDoS. You can stop this by having more bandwidth than everyone else... but that's pretty hard and it makes you a potential attacker.
Innovation here is in the form of using BGP to disseminate traffic filters. Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible. But I've seen there's some systems with more precision, like drop udp, drop fragments, drop packets to/from udp/tcp port X.
Most of these systems are designed so that these specialized routes don't propagate beyond immediate peers, but potentially, it might be desirable if they did.
Null routing is the MVP here: this IP is being attacked, so drop traffic to it as soon as possible.
Oh, this poor guy is being DDoS'd, so we're going to make sure that their service remains denied.
Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.
With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.
So the suggestion here is to implement source routing on a whim, with a table of 100+k entries? At your peering point?
An ISP has to do what it has to do to save their business when a large attack hurts their business. This may be what the attacker wants but that's not an excuse to do nothing.
But what machine learning has to do with this is not clear. Null routing protects you against traffic volume. Machine learning sounds like it maybe can help diagnose more sophisticated low-volume attacks. Maybe. You don't want anything remotely compute intensive when mitigating attacks.
Null-routing the target IP helps everybody except the customer who is being attacked: namely, the network operator and their other customers. From the victim's point of view, it's just as frustrating as the attack itself, and gets in the way of troubleshooting.
If you're running on a single IP, yes. If you're running on multiple IPs, it's not that bad for the one that's being attacked to get its traffic dropped and everything else works. It's not great, but what are you going to do. If you've got enough traffic to overwhelm the inbound on the top of rack switch your box is on, you're not going to be able to really serve any of the good traffic anyway.
With modern tooling and a bit of ML, it shouldn't be too hard for multiple ISPs to collectively determine which IPs are currently part of a large botnet. Drop packets from them, not to the victim. DoS the ones who are causing the DDoS.
There's usually way too many source addresses to do that, and anyway, routing infrastructure is geared towards looking at destination addresses, not source addresses. Also, each individual source doesn't look that bad --- if I've got 10,000 sources each sending me 1 Mbps of garbage, nobody is going to accept a block for only 1 mbps of sending, and yet, there's 10 Gbps of garbage arriving at my box; if I've got 10 Gbps or better connectivity, no big deal. But, if I'm only on 1 Gbps, I'm getting less than 1 in 10 of my inbound packets. I'd argue, if everything else has a big enough connection, it's probably still no big deal, it should be able to drop packets headed to me, as long as its upstream connection isn't filling up. But once abuse is causing contention that impacts others on my rack, it's probably time to null route.
If it's one of the big botnets with 100,000+ compromised systems, the individual bandwidth is even less. And if the botnet has significant ability to deliver spoofed traffic, source based filtering is meaningless. If it's reflected DDoS, I dunno --- there's value in hunting down the chargen services and removing them from the internet, but that's usually a lot more work.
OTOH, look on the bright side, if your outbound bandwidth is high and you get a lot of inbound DDoS, you may have roughly balanced your usage, and you may qualify for settlement free peering! (IMHO, this has got to be a major part of Cloudflare's business plan)
> With horrible grammar
Ah, the days before ChatGPT!
The topic made me read that as CatGPT, and now I can't pull it out of my head.
It's not as simple as "loading a script" - IP addresses (or in the case of IPv6, subnets) are (for the average person) a limited resource, as is bandwidth, and most amplification attacks require IP spoofing which is not possible from most connections.
If it's a volumetric attack, the side with more bandwidth wins (the attacker may be able to amplify here). If it's a load-based/application-level attack, blocking the attacker IPs at the firewall level solves it. This was application level, not (purely) volumetric, since they already had a WAF/Cloudfront.
Identifying attacker IPs to block is a matter of correctly attributing cost to a source IP, correctly attributing benefit (i.e. legit user activity) to a source IP, then blocking the IPs or ranges where the cost significantly exceeds the benefit you see from that IP or range.
That's easier said than done, since cost can come in many forms (e.g. open connections clogging up memory, TLS handshakes, requests that are expensive to parse for your web server, requests that trigger expensive database queries, in/out bandwidth, ...) which is why most just slap Cloudflare (or here, Cloudfront) in front of it and work around with manual rules like in this example.
Cloudflare does a pretty good job of managing it, at the cost of some centralization.
It would be pretty cool if there was a way to DDOS-harden at the protocol layer. Not sure if that’s even possible though
Maybe if the network was much more distributed and lower bandwidth?
If most of your customers are in Mexico, Canada is DDoSing you, and the pipes between you and Canada start filling up as a result that isn’t a big problem, right? As long as consumer routers on you/Mexico’s side of the Canadian clog don’t decide to help out.
Quite some time ago, someone from my family was alerted by their cat when the dishwasher was leaking. Their conclusion was that the cat was either trying to save them or the cat was trying to kill them.
One of my all-time favorite novels, Anansi Boys by Neil Gaiman, includes an anecdote: a crow's call wakes up someone who's sleeping outdoors, just as a large cat (a tiger, perhaps) is sneaking up on him.
One character suggests the crow was trying to warn the man. Another posits the bird was bringing the sleeper to the tiger's attention so it could enjoy the scraps after the meal.
Odd that the most obvious hypothesis wasn't given: the crow was probably using its alarm call to warn other crows. Crows are social animals, who care about the fates of their "friends and acquaintances" — so they would do that.
But also, on a tangent, there is a bird that does this kind of non-conspecific alarm calling the time as part of its food-gathering strategy: the African fork-tailed drongo.
The drongo gives true alarm calls to food-rival species nearby, to tell them when it has spotted a mutual predator. This leads to these food-rival species coming to rely on these signals. But then, every once in a while, it gives a false alarm, to get the food-rivals to run away for a bit, so it can nab the bugs/berries/etc that the rival would have been eating.
It's called an allegory
No, a drongo.
There's also an African bird whose foraging strategy includes alerting large mammals to the presence of food that they can harvest and the bird can scavenge.
In fact, there is a family of such birds, the honeywarblers, who locate beehives, then find humans and lead them to the beehives.
Wikipedia says that the behavior is dying out because there aren't enough human foragers.
There is another Neil Gaiman short story about a cat who goes gallivanting every night and comes back every morning all scuffed up. The owners don't know why since there aren't any other cats around. One night the owner can't sleep and discovers the reason.
Don't want to spoil. It's nice and short and a must-read for cat lovers. "The Price".
Good read
That was a terrifyingly poignant read, thank you.
the cat was either trying to save them or the cat was trying to kill them.
An inverted Schrödinger cat.
As always it’s easy to overlook the insider threat. Grammatically dubious extortion emai? Bitcoin ransom? Did it not occur to you that the cat was the one behind the attack?
Yeah, cats are notoriously bad at grammar.
You do know that’s just to throw humans off their scent right?
Everyone believes they have bad grammar so when they launch the really serious attacks we all think it was a state agency rather than the cats.
You mean to say that a Nigerian prince is responsible for the xz backdoor?
I CAN HAZ DOS ATTAK
The call is coming from inside the house!
The meow is coming from inside the house
Putting the "Bite" in "Bitcoin"
And I often wonder if on call is justifiable “because you make more money than most professionals”.
as with most roles, I think it is negotiable. You have your professional leverage, expected pay and grit. you need to balance these things.
Also, if you can get an equivalent role with less requirements such as being on call, then I guess it is just a question of grabbing it!
I mean, you have oncall, it's just permanent oncall.
Not really. If I don't agree to on-call, I do it on best-effort basis. That is: "Oh, I'm camping with kids without a computer. I'll try to help you as much as I can. Did you try Z after X and Y didn't work? Ok, try it and call me back how it went, I'll try to Google something in the meantime." If that would happen too often (more than 1-2x a year), I would try to improve the process or consider switching my job. And my phone is silent during the night.
I wouldn't call this a permanent on-call, just being responsive.
2 European teams I worked on paid a bonus for on-call duty, and the systems were so stable that enough people volunteered for the few who didn't want it, weren't forced to.
It was pretty great, I took a week shift every month or so except when I was going on holiday, and aside from lugging a backpack with my laptop everywhere, didn't affect my life at all except 1 or 2 minor issues
You usually get some extra money for the duty. And if you get woken up, the hours you spend are counted towards your normal working hours - so you aren't expected to show up in the morning after putting out a fire. Or you get some more bonus (like 2x hourly pay for the night work). That's about the balance when people are ok doing it.
But it depends on stability of your service. If it is messed up and people are woken up often, then you won't find many volunteers if they have other choice.
And here I thought your somehow hooked up a cat feeder to alerts.
Regardless, very cute - what’s your cat’s name?
Writing this post did make me think that if someone had a well-trained dog, they could hook up a monitoring service to something that makes a particular sound, which tells the dog to alert the person.
Her name was (I sadly lost her to cancer) Bamboo! Because one of the first things she did after I adopted her was to try to eat my bamboo plant.
And as an added bonus, we could get that dog classified as a service dog :D
“Sir you need to leave that mutt outside!”
“He’s a service dog”
“Why? You don’t look like you have any disabilities”
“Wow. First of all – rude! Second, yeah you are right I don’t but you see he’s my DDoS dog and I need him with me at all times to protect the company servers”
Funny, that's how the very first customer realised that the Australian telco Optus was down. The wireless cat feeder relied on the Internet and when no food appeared, the cat decided to complain to management.
proposing Danielle of Purrvice
It's so easy to crush ddos with token buckets that usually the only thing I need my cat to wake me for is when my Discord gets raided.
never heard of this before. I looked it up https://en.wikipedia.org/wiki/Token_bucket
I think this would be like a firewall or ingress thing that would drop packets that resulted in excess load before they make it to the application server.
It's a common rate limiting algorithm. Here's an interesting article from Stripe on how they use it in their APIs. [1]
Here's the code I use: https://github.com/jart/cosmopolitan/blob/master/net/http/to...
some types of DDoS. ;)
You could still overload the service with a sufficiently large attack in either volume of connection requests or number of unique IP addresses.
Token buckets are usually part of an overall resilience strategy rather than a silver bullet to solve all denial of service concerns.
we didn’t have a formal on-call rotation yet. That was a deliberate decision, since being on-call is painful, and the team was good about just collectively keeping an eye out for urgent alerts.
That seems like a terrible solution. Yeah, being on-call is painful, but at least I know beforehand when I'll be on-call and get compensated for it. Always being expected to keep an eye out for urgent alerts just sucks all around.
I know it sounds bad, but in practice, it really did work fine for us for quite a while.
1. We didn't experience that many incidents that couldn't wait until working hours.
2. There was never an explicit expectation to keep an eye out. We did it anyway because we were at an early-stage startup, and we all deeply cared about making our products work for our customers.
I know this from a few startups and it really is not that bad. You really triage what should wake you up and what's ok until morning. It works well as long as the technical founder is ok playing a goalie and essentially being always on call (even though others catch a lot of alarms).
It stops working when the company grows and no one understands the whole system and you need on-calls from several teams. Then the company does some formal on call rotation and it's fine again. It hurts during the transition only.
Good point about the technical founder. That was certainly the case for us, as our CTO handled many issues himself.
Yeah that sounds like on call all the time, that makes no sense.
Is this an ad for AWS?
Tech forum
Talks about useful tech
"iS tHiS An aStroTurFing Ad"
The entire article was "I have all these problems, and I use one small trick (aws product) to fix it all!"
It does +1 most of those types of spam farms with a bit more technical discussion, but not really that much.
Kinda reads like one, but if he was on GCP and used their ddos shield then it'd read like an ad for their service instead. Would be better if he'd been a bit more abstract and said cloud provider instead of naming AWS.
I thought it was going to be a home server that went into overdrive, heating the room your cat was in or knocking out the aircon.
Anyway, better experience than being woken up by a dozen SMS alerts.
Cats love to hang out in warm areas, even sunbathe, so I doubt they'd do anything to get attention in that situation! Their body temperature is a few degrees warmer than that of humans.
Cat would be in bliss then, warm things are their new beds.
We didn’t reply, though in retrospect, it could have been fun to try to troll them.
Not replying is the only valid answer. Trolling them could potentially put you more on their radar and get targeted for other attacks. And for what?
Yep, I know it could only invite more trouble. It was fun to fantasize about though, especially after watching the video I linked to in the post: https://www.youtube.com/watch?v=dWzz3NeDz3E
For the lulz
So that's why they want us to microchip our pets!
So they can connect to CAT-6 Ethernet.
But in 9 years, that was the only time she did it while I was sleeping.
... that you know of
Reminds me of this book: Dogs that Know When their Owners are Coming Home https://www.sheldrake.org/books-by-rupert-sheldrake/dogs-tha...
Tangential question that rose up regarding availability vs. quality of life.
For a small startup whose products are only available on the US, does it always make sense to do nightly oncall? This doesn't work for some products, but if, for example, you have a site that sells mattresses in the US, would you wake someone up to fix the site at 3AM?
I guess here the main $$ loss would come from accepting so much traffic. But I wonder if we can better differentiate what's worth waking up for.
I had an old set of PC speakers which always made as weird sound a few seconds before a new message arrived on my phone.
You might say, the cat es-cat-lated it...
cattackstic!
cattackstrophic!
Well, "attack" is just "cat-kat" spelled sideways.
It's translated by duckduckgo.com's chatgpt interface. don't down vote plz :D
I suspect that I am somewhat sensitive to electromagnetic fields and magnetic fields. There have been times when I have not felt well the next day after sleeping on an electric heating pad, and I have experienced severe discomfort after sleeping on a mattress with magnets.
When I used a CRT monitor, I often had diarrhea if I spent a long time in front of the monitor.
Since using LCD monitors or laptops, those symptoms have disappeared.
When I sleep, there is a wireless router on the right side of my head, and I play youtube videos on my smartphone on the left side. I have strange dreams and wake up early from sleep. However, if I put the smartphone on the right side of my head while sleeping, those symptoms are lessened.
Thus,
Even though there was no sound, wouldn't your cat have sensed that as well?
We once detected a DDOS because all our office phones went down. Silly attackers didn’t realise that our (money-making) APIs weren’t colocated with our public website and phone system.
The problem with paying extortion or ransoms is that you incentivize the attacker to come back and do it again. It may have been $5k to pay off one attacker and more than that to build the defense, but now you have defenses and are less likely to suffer attackers in the future. And as you say, not paying money to criminals is inherently worthwhile.
On the other hand the attacker may actually have incentive to follow through and hold up their end so as to build a reputation. Making their next victims more likely to just pay.
Somewhere I read that some ransomware had excellent "customer" service for helping you transfer over the payment and promptly restore your files.
“Ransomware Reality Shock: 92% Who Pay Don’t Get Their Data Back” https://www.forbes.com/sites/daveywinder/2021/05/02/ransomwa...
This stat includes everyone, some of them may not research what hit them. One of my clients decrypted his database twice, after seeing on the internet that they actually send a decryption key.
Oh, got it. So, the next ransomware author should just brand their ransomware as one of the "honest" ones :)
Scammers acting like an adult is hard to believe. They’re usually quick to start yelling and cursing in Kitboga videos.
The serious organized crime outfits are very organized. They’ll provide customer support to walk you through purchasing and transferring the bitcoin.
I mean it's not like these people are operating under " long lived identities and want to build a long term business relationship with you.
From their side, the logic seems as simple as: repeat the attack -> some chance of more money, don't repeat the attack -> 0% chance of more money
It is often a temptation to a rich and lazy nation
To puff and look important and to say
Though we know we should defeat you
We have not the time to beat you
We will therefore give you cash to go away
And that is called paying the Dane-Geld
And we've proved it again and again
That if once you have paid him the Dane-Geld
You never get rid of the Dane
and this wasn't a poor company, so $5,000 was nothing in the scheme of things
This might something for you (and the repliers):
From the german chaos computer clubs yearly meeting. Linus talks about what to do and who ransoms work, how "well" the service is and briefly pros and cons of paying.
https://media.ccc.de/v/37c3-12134-hirne_hacken_hackback_edit...
Also a good one was the first part: https://media.ccc.de/v/36c3-11175-hirne_hacken
About 20 years ago I was kinda accidentally the guy who dealt with the DDoS attacks in the sysadmin team. There was a sequence of extortion emails during about 2 week period:
1. $50k or we attack - didn’t register anything
2. $25k or else - a minor overload on the server but nothing serious.
3. $10k or else - a serious attack which affected the service in a major way.
4. $5k or we really pissed - this time they took down a whole Tier2 ISP and Datacenter in London for a day. Other carriers peering on London Internet Exchange had to blackhole traffic to our service provider and finally kept blackholing one of our IPs for a while. I had to scramble to find a DDoS mitigation service, new DC and servers.
We did not respond to any of the emails. The attackers were also quite dumb, they attacked the web servers which were located in a well connected place.
The money making service of the business was in the Caribbean with a 1,5Mbps T1 and a 0,5Mbps satellite backup. They could have saturated those much easier for much longer and the impact then would have been about $1M revenue loss per hour.