Through a series of connections I know a guy that knows a guy that works at Microsoft that was made aware and the changes have been reverted. Give 'er 30 minutes TTL ;)
Through a series of connections I know a guy that knows a guy that works at Microsoft that was made aware and the changes have been reverted. Give 'er 30 minutes TTL ;)
What could a TLA do with this if it had time to plan ahead?
Serve malicious updates from a locally controlled machine, for one. Lord knows about auth.
Do most DNS forwarders not block addresses that resolve to a local IP these days? I know dnsmasq does, and NextDNS too I think.
Why? Having local IPs on a public DNS is a legitimate use case.
As another reply mentioned, to prevent DNS rebinding attacks. The general expectation is you will whitelist domains from which you expect RFC1918 responses.
In fact, some people block domains by routing them to 127.0.0.1 in their host files. I've used private ranges too, in places where loopback might possibly do something funky.
I think most will see it as a DNS rebinding attack [1].
That’s the phrase I was looking for!
Serve malicious updates from a locally controlled machine. Lord knows about auth.
Wouldn't they have to break into my local machine first, plant an update service, and an update? That doesn't seem to scale well at all, and wouldn't it be easier to just break into the machine they want to 'update'?
A fairly prominent update service already runs from the domain microsoft.com Many machines come with it preinstalled.
The erroneous DNS change wouldn't help that sort of exploit. It just redirects attempts to contact microsoft.com to a local address, probably a router.
That's exactly what I said in the first post.
Why doesn't windows update use authentication (eg https)?
They do: the updates are signed so our hypothetical spies would need to have a zero day in Authenticode or to have compromised the signing keys.
I'm trying to figure out how this could have happened, but I control so few IP addresses that many of my DNS entries are manually assigned. And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses.
Anyone have any theories on how this could happen?
* Copy/Paste
* Copilot told me
* Sabotage (internal or external)
Or hard to reason about IaC
my wild idea: counterattack for DDoS
I'll go with Joseph Conrad on this one.
"It's only those who do nothing that make no mistakes, I suppose."
Now the persons that did it have some proof that they did something.
They will surely put some check in place because there should be another adage somewhere that says that you only learn to use the handrails after you fell in the stairs.
erro ergo sum
about ten years ago, apple added a stray record to the apple.com zone
.... a DNAME[1] record
....... that pointed to apple.com
1: https://en.wikipedia.org/wiki/DNAME_record
this had some pretty disastrous results[2]
2: https://mashable.com/archive/apple-tunes-app-store-icloud-pr...
bad things happen everywhere
It continually astounds me that DNAME got standardized. Scary stuff.
My guess is that someone at MS was testing Windows Updates or other changes from a local source. They also had some other DNS updates in their config they were testing. They took all of their config and pushed it out, when they should only have taken the other changes.
I hope that's not their workflow, but few people I've ever worked with have know how to create operating procedures that had half a chance of succeeding, usually it's based on "oh, don't worry, i would never do that"
Certainly just some automation bug, perhaps a few things strung together, like a dev environment setup that leaked into production. A human in the loop making a mistake probably as well.
This is the kind of thing you look at and put up a few guardrails to prevent it happening again.
> And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses.
Clearly the following is not in play for a root domain (Microsoft.com) but assigning a DNS entry to a class C address does have a purpose.
If you have an intranet server, giving it a DNS name allows for HTTPS serving, with an automatic, CA signed, certificate. (Using say LE with DNS challenge.)
I provide this simply as an example of how this might come about.
An entry-level admin is now unemployed, just before the holidays.
I hope not. Failures are on a spectrum and this was unfortunate but probably not malicious. All things considered this should be a lesson learned. There should be more failsafe mechanisms in place so juniors can fail safely and learn from them. The absolute worst thing we can do is shame an individual so they don’t attempt to try new things in fear of ridicule.
There should be more failsafe mechanisms in place so juniors can fail safely and learn from them.
And if not, whoever put the junior in that role is the person responsible for the problem.
well theoretically you could argue the structure of this task should have 'dual control' / multiple people should be involved in the process checking each others work. preferably even split it up people who do not know or interact with each other on a regular basis. yes it would be slower but its important to get it correct.
might as well throw in some automated poke-yoke or whatever too.
in that case there is no fault in any of the juniors or operators, the fault is in management for failing to implement infrastructure to force a critical process to have more than one control
> An entry-level admin is now unemployed, just before the holidays.
I highly doubt that entry-level admins at Microsoft have access to DNS for their primary domain. My guess is that this incident is a lot more interesting than that.
Yep, this doesn't seem like the kind of thing that you can just toss a couple approvals on and change at a company as big as Microsoft. How this made it through the review process would be very interesting
The seniors all go on leave and the interns are left to run the place. If they fired the juniors the seniors would have to come back from holiday!
I'm wondering how such a change would get "merged" in to begin with. I imagine even non-network engineers would get this huge itch having a large corporate contain a private IP in the changelist (I'm the non network engineer and can't really explain why it's bad. But it FEELS wrong and sometimes you at least need to use instinct to get another pair of eyes on something).
This is most likely a honest mistake. Smart managers don't fire employees for such mistakes unless their behavior regarding that mistake is inappropriate.
As the story goes, after a junior admin wiped a production database. The boss was asked if he should be fired. To what he answered: "Fire him? No way! Not after such an expensive training." Now, he knows.
Nah, if it's already reverted, they're good to go. A post-mortem with how something like that got through will definitely be on the table though.
How the hell did that pass any sort of responsible review process at Microsoft?
Now Microsoft owns all your home networks, only like the default address on every home router out there...
Now Microsoft owns all your home networks
Only if you’re slumming around 192.168.x.x
I for one only use Class A CIDR, 10.0.0.0/8
No they don’t. Going to Microsoft.com will take you to your router.
click click click -- "it's done, boss."
Microsoft's mindnumbingly dense ClickOps culture strikes again.
at a serious org this would have involved at least some level of oversight or intervention
You have the danger of this backwards: this is a very bad security problem for Microsoft, and not a problem for people outside of MS (except to the extent that we're all indirectly reliant on MS being secure). Pointing a domain at an IP address does not give you any power over than IP address, and you can point a domain at anything you want.
Any risk here is nearly the opposite of what you seem to think it is
No, it's that when you open microsoft.com it could open your router page.
for uninitiated (me), why is it bad?
Well in my case (and a lot of other people), 192.168.1.1 is the local address of my home router. So if I go to microsoft.com I have a 1 in 7 chance of getting my home router instead (if I ignore the certificate warning). Other random breakage will happen depending on what that local address is assigned to for you.
In theory this could be leveraged for hacking, but I think that would require setup in advance.
yep. If a hacker can somehow control 192.168.1.1 or 192.168.0.1 they get access to your microsoft.com cookies at least. I'm sure there are more microsoft specific ways to leverage this too (e.g. data/updates hosted on microsoft.com that misuse HTTPS as a poor man's authentication. The curl | sh crowd are especially susceptible to this problem.)
It's an IP address reserved for private networks:
Since 2 out of the 7 IPs are 192.168 (private ips), 2/7 visitors to microsoft.com will load the private ones assumign equal weight and not get the page to load.
So if you go to microsoft.com with probability 1/7 you'll hit 1.1 on your private network of 192.168 - likely router, and with probability 1/7 you'll hit 1.0 maybe printer
Wait... Can DNS resolvers be configured so that RFC1918 is respected?
I mean: I don't expect anything less from Microsoft than doing stuff like that and it cannot affect me for I nullroute microsoft.com from my unbound server (unboud takes wildcard when nullrouting or NXDOMAINing crap domains like microsoft.com or meta.com etc., which is sweet).
However I'd expect my trusty DNS resolver to also prevent me from anyone not on my private LANs to impersonate addresses reserved for private uses.
Does anyone know here if it's easily doable?
I wouldn't expect it to. I have plenty of RFC-1918 addresses in a subdomain of my public DNS zone for my home network. It's been that way for decades. (Perhaps I should use split DNS, but...)
You're looking for DNS rebinding protection, many DNS servers support it. However there are some cases where things do use private IPs in DNS records outside of the local domain, one example is Plex (e.g. https://support.plex.tv/articles/206225077-how-to-use-secure... suggests turning off DNS rebinding protection) -- although in some cases you can allow particular domains which is a much better way than turning it off entirely.
(See also the sibling comment about microsoft.com being IPv6 only as a result of a particular implementation of DNS rebinding protection: https://news.ycombinator.com/item?id=38704159)
Yes, some can.
Unbound's "private-address" and "private-domain" directives control this.
Similarly, bind9 has "deny-answer-addresses" (with an "except-from" option so you can specify local domains that are allowed to use them):
https://bind9.readthedocs.io/en/v9.18.20/reference.html#cont...
Not sure about others.
My Unbound servers strip RFC out.
Public resolvers keep DNS answers intact because they can carry alt data like how dodgy a SMTP server is.
For the uninitiated, can some traffic get sent to 192.168.1.1. Is it round robin?
Yes it is.
I could be wrong but in my experience OS just selects one random and uses that for some time not round robins it.
Even if that is the case, if it is random, some section of DNS would send traffic to it. Maybe it was OK because most resolvers would ignore the local address on the list??
Wait wait wait wait. Bunny.net accidentally changed their DNS to 127.0.0.1 and took a bunch of their CDN users down today too. Coincidence? Weird day.
<singing voice="Chris Rea">Becoming 'home' for Christmas </singing>
Employees mixing up SSH consoles while setting up their smart home Christmas lights over weekend.
I get it too, with .1.0 as well
Name: microsoft.com
Address: 192.168.1.1
Name: microsoft.com
Address: 192.168.1.0
1.1 is gone but I'm still seeing the 1.0 entry.
Same here with 1.0
They probably asked copilot to manage their DNS servers
"help us bring Microsoft to every home's network"
AI-driven type coercion is a new type of debugging hell.
29% traffic lost
Potential timeouts for clients/workstations trying to reach microsoft.com.
Which entry is picked for use is generally random depending on the client.
Most systems will retry using another entry though on issues connecting through. That said, if you are on a network that is 192.168 based, trying to get to Microsoft.com may just send you to your local router!
that's what happens when you buy address space from the back of a van in the parking lot ;)
How dare you besmirch the reputation of The Awesome Store!
https://theamazingworldofgumball.fandom.com/wiki/Awesome_Sto...
Maybe Sydney tried to breakout…
Good point. Let's watch Microsoft executives & employees closely for signs of panicking over an escaped AGI.
Y'all, instead of the constant confirmed here. Just do an authoritative lookup.
dig +trace +short microsoft.com
NS a.root-servers.net. from server 100.100.100.100 in 10 ms.
NS b.root-servers.net. from server 100.100.100.100 in 10 ms.
NS c.root-servers.net. from server 100.100.100.100 in 10 ms.
NS d.root-servers.net. from server 100.100.100.100 in 10 ms.
NS e.root-servers.net. from server 100.100.100.100 in 10 ms.
NS f.root-servers.net. from server 100.100.100.100 in 10 ms.
NS g.root-servers.net. from server 100.100.100.100 in 10 ms.
NS h.root-servers.net. from server 100.100.100.100 in 10 ms.
NS i.root-servers.net. from server 100.100.100.100 in 10 ms.
NS j.root-servers.net. from server 100.100.100.100 in 10 ms.
NS k.root-servers.net. from server 100.100.100.100 in 10 ms.
NS l.root-servers.net. from server 100.100.100.100 in 10 ms.
NS m.root-servers.net. from server 100.100.100.100 in 10 ms.
RRSIG NS 8 0 518400 20240101050000 20231219040000 46780 . fG/YHtUJu3YMAm9Mlzzvp3xG4UCPG01aYNnlyF1HfAHdZpR+L88CVUcz NFHq9M45KjB7ZTlSFt2JvEyK/8FcavZLOthkXRREbJQswjLCbhiPQCbq tQLF+tKaNYUihqawCfjgZy1i5YwYjmphbjfzwoKo1POtepf0YCIcuLBi nQFw4Lr79O6cjyg6qlYnqaK6z4Xi5qt6ocohJafjs86LuuRo2WvmJ1IK k0ZUoAC6Qyjz4MVhqHMvQGdp7EnzjoL8Y9PTXeUuD6Ixp/Aklj2psLjD TZDPYN1q+zDd1giFyuwNRX9DG1zrxzN2lzQiLWmGKrzP3DvFWL1L2Ts1 FWjy/Q== from server 100.100.100.100 in 10 ms.
;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable.
;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable.
;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable.
A 20.112.250.133 from server 150.171.10.39 in 20 ms.
A 20.231.239.246 from server 150.171.10.39 in 20 ms.
A 20.76.201.171 from server 150.171.10.39 in 20 ms.
A 20.70.246.20 from server 150.171.10.39 in 20 ms.
A 20.236.44.162 from server 150.171.10.39 in 20 ms.
A 192.168.1.0 from server 150.171.10.39 in 20 ms.
Or from a bunch of dnses:
$ export srch="192.168.1.0"; echo "as of $(date '+%s';):"; for dns in 1.1.1.1 8.8.8.8 76.76.2.0 9.9.9.9 208.67.222.222 185.228.168.9 76.76.19.19 94.140.14.14; do dig @${dns} microsoft.com +short | grep "${srch}" > /dev/null; if [ $? == 0 ]; then echo "${dns} still has ${srch} for microsoft.com"; else echo "${dns} no longer has ${srch} for microsoft.com"; fi; done
as of 1703033639:
1.1.1.1 still has 192.168.1.0 for microsoft.com
8.8.8.8 still has 192.168.1.0 for microsoft.com
76.76.2.0 still has 192.168.1.0 for microsoft.com
9.9.9.9 still has 192.168.1.0 for microsoft.com
208.67.222.222 still has 192.168.1.0 for microsoft.com
185.228.168.9 still has 192.168.1.0 for microsoft.com
76.76.19.19 still has 192.168.1.0 for microsoft.com
94.140.14.14 still has 192.168.1.0 for microsoft.com
$ pbpaste | sed 's;^; ;' | pbcopySo let me see if I understand. With this DNS record, if me or Windows tries to hit “microsoft.com” there’s a 1/7 chance it hit my router instead?
Yes
stumbled on this thread, my device just got blocked by my at home router. My dns is 192.168.1.1, any suggestions for how to troubleshoot this?
microsoft.com is currently IPv6-only on my network, because OpenWrt's DNS rebinding protection filters out the A records:
$ ping -4 microsoft.com
ping: microsoft.com: Address family for hostname not supported
$ ping -6 microsoft.com
PING microsoft.com(2603:1030:c02:8::14 (2603:1030:c02:8::14)) 56 data bytes
64 bytes from 2603:1030:c02:8::14 (2603:1030:c02:8::14): icmp_seq=1 ttl=112 time=68.4 msI was getting an empty answer for microsoft.com. Turns out my dnsmasq is blocking it:
$ dig microsoft.com. | grep EDE
; EDE: 15 (Blocked)
resolver.log:Dec 20 00:43:57 router dnsmasq[8172]: possible DNS-rebind attack detected: microsoft.comI don’t know man, putting microsoft.com on your router sounds like a massive reduction in latency. Congrats on the achievement.
Damn you Murphy's law.
Another confirmation here. Whoops!
This seems the opposite of good.
I'm surprised 192.168.1.0 is still there 2.5 hours later https://dnstools.ws/lookup/microsoft.com/A/
Looks like somebody made a booboo
Confirmed here.
I imagine it has something to do with how MS creates bypasses for host files for systems xpsp2 onwards ... by [1] it suggests Win10 still does.
[1] https://superuser.com/questions/1111582/does-microsoft-preve...
Was someone able to generate a *.microsoft.com SSL certificate when doing domain validation on non-microsoft machine ?
This isn’t something that I think should be diluted.
If it’s that simple for a stray record to be included in the dns round robin it could have been bad if it was an external ip with a machine setup by a phisherman especially since control of a domain is all you need to get an ssl cert now.
Couple this with the fact that it’s Microsoft, one of the most relied on companies in our computer world, this is pretty darn horrible.
Microsoft also has some of the phishiest looking domains when you are redirected around the O365 cloud.
100%. Starting with "onmicrosoft.com". A phisher wouldn't really have to control Microsoft.com to take advantage of confusion.
There were several phishing attempts from that domain, onmicrosoft.com, to my personal email account this past week.
Microsoft.com has been constantly trying to fool me into subscribing to their Office tools.
Absolutely, I am so glad I'm not the only person who feels this way. Microsoft does not understand domain names.
They use 1drv.ms as the domain in OneDrive emails, and sometimes it almost looks like the .ms ccTLD belongs to them - it very much doesn't, anyone can register a .ms domain.
windows.net being an Azure domain that third parties can have content under is fitting.
It sometimes looks like they want their users to be phished.
Microsoft is a smart company though, I really hope they can sort this out.
Yeah. Easy to spot when your session expires - first they're still using login.live.com, just to redirect you to login.microsoftonline.com.
Indeed, take a look at the lists of azure and o365 domains, they're all over the place:
https://learn.microsoft.com/en-us/microsoft-365/enterprise/u...
https://learn.microsoft.com/en-us/azure/security/fundamental...
The only thing that competes is the redirecting when you log into any health portal.
For all this to work you need to control the domain. Is that easier than simply breaking into their systems and owning their servers?
That's exactly what they're saying.
I.e. one of the IPs for microsoft.com belongs to $phisher, which means they control (a subset of the traffic going to) the domain. They can't add CNAME records for certificate validation, but LetsEncrypt for example offers HTTP-based validation.
Not sure how Microsoft sets up their certificate pinning, it might not be quite that easy.
It might also be a highly targeted attack on someone with precious information wherein someone was able to hack a simple router and in order to get access to their actual microsoft.com account, they simply setup a phisherman's clone on the router and captured the login/password/2fa and got into the account.
For the Microsoft.com domain, proper, there seem to be no existing CAA rules, allowing each and every CA on earth to issue certificates based on whatever criteria the CA requires. What could possibly go wrong with that approach?
Sometimes it’s easier to bribe then to break
I know myself through a series of connections as well. ;) ;)
If you know yourself through a series of connection then that's a false you.
Not necessarily, in the same sense that I'm my neighbor's neighbor.
That's just a sentence. Not who you are.
https://en.wikipedia.org/wiki/Self-enquiry_(Ramana_Maharshi)
You're also an alien's alien.
192.168.1.1 is gone now, but all authoritative nameservers are still offering 192.168.1.0. Oops.
A few are dropping 192.168.1.0 now:
Only one of those is authoritative. All of the authoritative servers have dropped it. Microsoft has fixed the issue.
This is my favorite HN comment of 2023.
This is my second favorite HN comment of 2023
30 minutes minutes or 30 Windows minutes? :P
Actually, it’s looking more like 6 days. No wait, 30 seconds.
Do we need to restart our Windows machines? :)
I see them both. My TTL will run out at 16:39 PST, though.
TTL appears to be set to an hour. But either way, its been 45 min and the primary ns1-39.azure-dns.com is still offering up 0.1
Few months early April 1 stunt?
GG, Gone for me now.