return to table of content

Moving our Encrypted DNS servers to run in RAM

LinuxBender
39 replies
1d1h

Are these physical servers or VM's? I ask because some VM types can be frozen/shapshot/cloned or live replicatedincluding memory contentsas is done in some VPS providers for lawful requests. From the bare metal host anything can be accessed in a VM or container. Do they have a diagram of their physical setup?

[Edit] - Are there any Mullvad architects here that can help us avoid going down theoretical hypothetical rabbit holes and turtle stacks?

dijit
18 replies
1d1h

Mullvad has been pushing system transparency a lot (which is a security system specifically designed for bare-metal), so it's fairly safe to say that it's based on bare-metal.

https://www.system-transparency.org

LinuxBender
17 replies
1d1h

Maybe it should be safe to say?but do they actually have a diagram of their physical and logical setup that shows they use bare metalonlyand not VM's or containers? Is that documented in audited security controls?

dijit
15 replies
1d1h

Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security.

It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security.

Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question.

They've also discussed booting UEFI directly to VPN nodes:https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...

It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.

LinuxBender
8 replies
1d1h

Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security.

I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.

Containers, btw, cannot be snapshotted.

I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted.

They've also discussed booting UEFI directly to VPN nodes:

That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this.

It's completely weird to argue that they might introduce a layer of insecurity here.

Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentationunless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.

ziddoap
6 replies
1d

They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.

They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit.

On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant.

After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.

https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...

LinuxBender
5 replies
23h44m

I remember that and that case was cool. I will leave my questions in place though because I recall Apple doing the same thing when a terrorist phone was locked and they claimed to not be able to accesswith a big public show/fightit but turns out they could and so could the FBI. My theory is that they did not want the public to lose confidence in their phones.

hathawsh
3 replies
23h11m

Yeah, that Apple case was really interesting. I always guessed that Apple could simply push a software update (possibly to everyone) that happens to open a backdoor for a specific phone. I wonder if that's what they did.

asynchronous
1 replies
20h27m

What happened in that case is the FBI went to a different vendor, and they broke into the iPhone either with a zero day they had developed or more likely, they just cloned the phone hardware thousands of times until they could guess the password.

buildbot
0 replies
20h24m

Yep, Apple had nothing to do with it. They do hand over iCloud data (which now can be E2E)

buildbot
0 replies
20h23m

Maybe - depends on how secure enclave is built. It may have hardware defined limitations on # of tries for the passkey and no way to circumvent that in firmware even.

theshrike79
0 replies
18h20m

Apple would've had to create a specific OS update with a security hole the FBI could've exploited and distributed that.

Courts couldn't force them to do that -> FBI went another way. This was in the iPhone 7 era IIRC.

Currently their stuff is locked down even tighter and Apple has even less ability to hack anyone's phone. Barring a full-on backdoored software update targeted to a specific person - which they refused to do once already.

10000truths
0 replies
19h41m

I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.

"Here is a subpoena compelling you to disclose the data you have on XYZ."

"Sure. Here is the data we have on XYZ."hands over blank page

It's not sidestepping the request. They literally do not have the data, because they don't retain it. And unless there is a specific law mandating that they retain the data, law enforcement have no grounds for punitive action.

wolfgang42
2 replies
1d1h

> Containers, btw, cannot be snapshotted.

lxc-snapshot(1), podman-container-checkpoint(1), and docker-checkpoint-create(1) all beg to differ.

rzzzt
1 replies
22h15m

Some (maybe all) of these are backed by CRIU:https://criu.org/Main_Page

paulddraper
0 replies
20h22m

Docker at least is

kaszanka
1 replies
1d1h

Containers, btw, cannot be snapshotted

Why not?

anonacct37
0 replies
19h26m

criu has been used for live migration for a while.

Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.

Ms-J
0 replies
16h1m

I think it's a very rational question to ask. Is Mullvad using only bare-metal infrastructure, or is there any logical abstractions added to the servers?

Companies make statements all the time which can be twisted to their benefit, but read exactly like what a customer is after. I'm not stating Mullvad does or doesn't do this, btw.

tiagod
0 replies
8h29m

Even if they did... if mullvad was a honeypot, there is nothing that would stop them from posting a fake diagram.

k8sToGo
15 replies
1d1h

Physical server can also be frozen (literally) and then data extracted from RAM.

LinuxBender
8 replies
1d1h

It can but to my knowledge law enforcementin the US anywayhave only done this oncefor a national security incidentin the real worldoutside of a lab. If you have any links to this becoming common practiceoutside of a lab demo or security presentation on their own hardwareI would be interested to save in my own documentation.

They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went.

[Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocationwith a performance hit. Both are enabled by default on modern kernels now.

    init_on_alloc=1 init_on_free=1

mschuster91
7 replies
1d

There exist special UPSes that you can attach to a power line of a server, unplug and unmount it, and keep the server powered all the time, without the server being able to detect that.

If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.

tln
5 replies
1d

Whoa! Any keyword suggestions to find those UPSes?

Eldt
3 replies
1d

I'm having trouble remembering what they're called, but I do remember reading about these. I saw a demonstration where they slightly pulled at the desktop power plug in order to hook a thin device around the plug connectors to provide power, and then they could transport the desktop around.

kl4m
1 replies
23h12m

It becomes way more straightforward if it's a server with redundant power supplies.

tuetuopay
0 replies
23h1m

Actually not because a server with multiple PSUs definitely knows when one loses power. It’s made to raise alerts about a power rail failing. So this makes it harder because you need to keep power on both to not raise any alarm

rwmj
0 replies
21h41m

For laptops, law enforcement use "mouse jigglers" to prevent screen locks and laptops going to sleep, which I guess is similar. Random article:https://www.electronicproducts.com/what-is-a-mouse-jiggler-a...

fmajid
0 replies
23h48m
0cf8612b2e1e
0 replies
21h54m

Just like Seinfeld and saving the Frogger high score.

greentea23
3 replies
1d1h

Are there any known examples of this actually being done by law enforcement or government and working?

ziddoap
2 replies
1d

There are plenty of examples of it working (e.g. researchers from F-Secure presented at SEC-T conference in 2018 with a cold-boot against Bitlocker), but I haven't seen a LEA or government come out and say "we got access to X via a cold-boot attack".

roenxi
0 replies
16h23m

Although, obviously, it was the same story with dragnet spying before Snowden. Sure it was technically possible in a lab environment and law enforcement had incentive to do it, but nobody could turn up any actual evidence.

0cf8612b2e1e
0 replies
21h52m

As RAM densities/speed increase, I wonder if this is still viable. DDR5 had to mandate some baseline ECC given the intrinsic error rate.

xyst
0 replies
21h30m

good ole can of air compressor flipped upside down and quick hands trick

adrian_b
0 replies
1d1h

That is why modern servers have the option to encrypt the DRAM to prevent this attack.

Moreover, there is also the option to have distinct randomly-generated keys for each virtual machine, to make useless the speculative attacks that succeed to peek at the memory of another VM.

AMD has offered these options for years and the future Intel CPUs will also have them.

xyst
1 replies
21h31m

If you are asking this, you probably need to self host your VPN

LinuxBender
0 replies
21h10m

I do this but I do not use them for privacy. I use Tinc [1] to route around internet outages as it can do user-space dynamic mesh routing. Since they are all nodes I pay for they would not hide who I am. While I do control the logging of Tinc I do not control external logging of the server and VPS providers.

For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi.

[1] -https://www.tinc-vpn.org/

WirelessGigabit
0 replies
23h25m

One can fix that by deploying software with the VM pause functionality removed and / or remove disks the bare-metal servers and only PXE-boot them. That way there is nothing to pause.

LinuxBender
0 replies
20h7m

The more I think about this VM's are not required. There are tools that can extract data streams based on specific parameters on demand using eBPF. Perhaps tools like Sysdig/Falco, etc... It might also be useful to know if they custom compile kernels to disable eBPF and if they do any additional specific kernel hardening to strip out or disable on demand debugging capabilities.

throwaway914
20 replies
1d2h

As a naive outsider: Does RAM in 2023 encrypt all its contents?

Can we reliably "forget" the contents of what's in RAM by powering-off and letting some encryption key somewhere fade from a smaller, more volatile piece of memory? I'm curious what has been done to mitigate cold-boot attacks in RAM. What has changed in hardware? Are the keys to encrypt contents of RAM kept in the TPM or something?

jchw
8 replies
1d1h

In this case I think the point of running from RAM is that there isn't non-volatile storage. Of course, someone who has physically compromised the server could do potentially bad things, but this makes it harder to e.g. persistently compromise a server, since it doesn't have storage to store malware on persistently, or e.g. accidentally or intentionally log data, since there is no local storage to log it on.

Encrypting what's in RAM would possibly add some additional protection but realistically if things are done right what's left in RAM will probably be only marginally more interesting than what you can see over the wire; at most I'd expect you could see a tiny amount of active request data. That's just a guess, though.

ignoramous
7 replies
1d

since it doesn't have storage to store malware on persistently, or e.g. accidentally or intentionally log data, since there is no local storage to log it on.

Does this preventrootkits?

but realistically if things are done right what's left in RAM will probably be only marginally more interesting

See:https://en.wikipedia.org/wiki/Cold_boot_attack

jchw
6 replies
1d

Does this prevetrootkits?

That's the idea. If the OS is immutable, where do you put them?

Bootkits require further mitigation to prevent, but Mullvad has also put effort into that, too, with their secure coreboot bootchain.

Of course as an end-user you still can't really fully validate a machine over the network has not been tampered with, so you're obviously ultimately still trusting Mullvad as you'd expect.

See:https://en.wikipedia.org/wiki/Cold_boot_attack

Yes, that's what I'm referring to. I don't expect that there would be a significant amount of requests in RAM, probably just remnants of mainly what was active (and some de-allocated but not cleared remnants?) when the machine was reset.

LinuxBender
5 replies
23h51m

That's the idea. If the OS is immutable, where do you put them?

They would be in the PXE image. There was one time I flubbed on this actually. I had to change export options on our PXE servers to fix something and I forgot to change it backadding my poor excuse of multitasking and being lazy. This was in a Dev environmentwe also used PXE in production believe it or not. A developer sitting across from me said he changed something in the image by mistake and I said,"That's not possi... oh crap."It was cool of them to call it out right away. Attackers probably won't be so kind.

jchw
2 replies
23h41m

For what it's worth though, that's possible to mitigate. I don't believe Mullvad uses PXE and they do have a "secure boot" validation chain, starting from Coreboot on the hardware.

https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...

ignoramous
1 replies
23h34m

It is an improvement, for sure.

secure boot, as painful it is to get right, doesn't help when the vendor runs its own things with elevated privileges, like they are known to do:https://news.ycombinator.com/item?id=34085635

Doubly so, when the said things are not only closed source but vulnerable themselves:https://archive.is/6ClWm

Doescoreboothelp in mitigating vuln in privilged vendor blobs?

jchw
0 replies
23h2m

Well, with Coreboot, you at least know the very first instructions executed by the x86 cores are under your control. That's all it can really do. Whatever else lurks beneath is really impossible for anyone to scrutinize without insane capital or insider knowledge.

Mitigating the risk of Intel ME is obviously something that is complicated. The "state of the art" so-to-speak is really just rendering it non-functional. This isn't strictly related to Coreboot; that said, Coreboot has a page about ME:

https://www.coreboot.org/Intel_Management_Engine

And in general, when you're running Coreboot, you're running significantly less privileged blackbox blobs (possiblynonein some rare cases), and in general, the surface area of Coreboot is going to bedramaticallytinier than any typical UEFI system firmware. So it's absolutely worth it in that regard.

There's never going to be a "perfect" option here with how complex things have gotten, but I think that you canat leastsay that Mullvad has taken almost any reasonable step, and perhaps a few steps that are beyond what many would consider reasonable, towards developing effective mitigation of the risks of rootkits, bootkits and malicious/unwanted vendor code running on their "diskless" architecture.

Of course, youstillhave to trust Mullvad. Trust is definitely a meaty human-y experience, and the cold rational part of our brain craves some kind of technically-bulletproof way toprovethat it's truly safe, secure, and private, but that's just not possible. Best you can do is layer enough assurances on top of each-other in a way that at least some layers are completely independent of others for maintaining your operational security. Far easier said than done, and obviously how rigorously one approaches this will vary on paranoia.

At some point you have to pick your middle-ground I suppose.

generalizations
1 replies
19h33m

we also used PXE in production believe it or not

Wait why would this be a bad thing?

LinuxBender
0 replies
19h25m

It depends on your use case, SLA's and configuration management role/structure discipline. There are different schools of thought ondisklessboot which is not truly diskless as the images have to be shared from somewhere meaning that disk availability goes from 1:1 to many to one increasing the blast radius of outages when the NAS fails or network re-convergence occurs. They all fail regardless of how many millions one spends and in my experience the more millions the more glorious the failure thatshould never happen. For some companies this is fine if they have the discipline to balance their servers across many NAS servers in a strict hierarchy that facilitates a smaller predictable POD structure blast radius. But that's a bigger topic than I could possibly cover in a HN comment. It also depends on how the OS is being used and how writable spacestmpfs vs diskare being used. This will all vary wildly across different business models.

josephcsible
4 replies
1d1h

Isn't that what Intel TME does?

throwaway914
3 replies
1d1h

That does look like it:

https://www.intel.com/content/www/us/en/developer/articles/n...

Looks like AMD has an equivalent:

https://www.tomshardware.com/news/intel-mktme-amd-memory-enc...

I wish there were a sort of checklist that prints out in dmesg saying "Your RAM is encrypted: (check) Your machine has these mitigations against cold-boot attacks: ...some list here... (check, check, check)"

In some contexts, I love how far we've come with secure boot, attestation, speculative execution mitigations, etc. but it's very difficult for the average Linux user to understand what they should expect for hardware security, and then work toward enabling all of it.

I love what Gnome did in the last year in Settings -> Device Security:

https://www.phoronix.com/news/Ubuntu-No-GNOME-Device-Securit...

There's a detailed list here of what it's looking for:

https://forums.debian.net/viewtopic.php?t=152705

Not a fan of all the bound-to-Intel technologies. I wish the concept of what's being secured were identified, then made available in dmesg or through some /sys or /proc interface.

matja
1 replies
22h54m

I have an AMD CPU and I get in my kernel logs during boot:

    [    0.338594] Memory Encryption Features active: AMD SME

throwaway914
0 replies
20h25m

I mean like a checklist of security capabilities that are enabled/disabled, all grouped together in dmesg. That is cool though, it does help

adrian_b
0 replies
1d1h

AMD Epyc servers have offered encrypted memory for many years.

Intel is only now catching up.

tredre3
2 replies
23h39m

AMD offers RAM encryption on their threadripper server CPUs and also in their Ryzen PRO on workstation/laptops (you have to enable it in the bios).

https://www.amd.com/system/files/documents/amd-memory-guard-...

https://www.amd.com/content/dam/amd/en/documents/epyc-busine...

latchkey
0 replies
23h23m

Years ago, I went to AMD and saw this demonstrated live by engineers there. They spun up a VM with it turned off and you could basically grep a string out of ram. Turning it on, grep returned nothing. It was pretty cool. Developed by a bunch of true grey beards over the course of many years.

Cody-99
0 replies
21h38m

On consumer Ryzen chips you can enable it too if your motherboard supports it. Like error correction code (ECC) support on consumer chips it isn't "officially" supported but it works. I know MSI and ASUS (IIRC) have the option in bios for transparent memory encryption.

cookiengineer
2 replies
1d

As a naive outsider: Does RAM in 2023 encrypt all its contents?

No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed).

There's even an EFI reboot flag that could fix this and force-clears the RAM on next boot, but it's usually deactivated.

[1]https://en.wikipedia.org/wiki/Cold_boot_attack(works still on Windows, thanks to BitLocker)

[2]https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...

rwmj
1 replies
21h38m

This isn't correct - the key is stored inside the CPU.

cookiengineer
0 replies
6h40m

Did you read any of my referenced links? You cannot state this when the paper and well known attack methods disprove your statement.

Bitlocker does not store the keys in the CPU, otherwise a coldboot attack while literally transplanting the physical memory rows to a different machine wouldn't work. And it does.

In case you are confusing CPU with a TPM chip, which can optionally store the master volume key but only for activated full disk encryption:

With TPM you can easily bypass that, too, with tools that can read the i2c bus. See here [1]

[1]https://blog.scrt.ch/2021/11/15/tpm-sniffing/

kisamoto
16 replies
1d6h

Mullvad encrypted DNS is also available to all - whether paying for VPN services or not.

In addition they also support optional content blocking[1] via blocklists, just set the desired TLS/HTTPS DNS server.

[1]https://github.com/mullvad/dns-blocklists

SV_BubbleTime
10 replies
22h33m

Can you help me out with the threat model for encrypted DNS?

As I see it, the model is that I need to contact a site, and use DNS to get the IP - but either my ISP or my VPN provider can see I’m visiting that IP whether I looked it up just now, or knew it already.

So unless the model is I’m using a VPN, but am leaking via someone else’s logging DNS, I don’t really get it.

Etheryte
6 replies
22h27m

Personally my main motivation is I don't want anyone to snoop what sites I visit and then sell that data, my ISP included, but there are other issues too. To quote Cloudflare [0]:

Queries could be directed to a resolver that performs DNS hijacking. For example, in the UK, Virgin Media and BT return a fake response for domains that do not exist, redirecting users to a search page. This redirection is possible because the computer/phone blindly trusts the DNS resolver that was advertised using DHCP by the ISP-provided gateway router.

[0]https://blog.cloudflare.com/dns-encryption-explained/

_Algernon_
5 replies
22h20m

Cant your ISP reverse lookup the IPs you connect to based on your IP headers anyways?

jrockway
2 replies
21h59m

They can, but pretty much everything will reverse lookup to Cloudflare or AWS.

gruez
1 replies
16h43m

SNI reveals the exact site though

mmwelt
0 replies
12h41m

As I understand it, this is now being addressed by ESNI[1] and ECH[2]. Hopefully ECH will continue being more widely deployed.

[1]https://www.cloudflare.com/learning/ssl/what-is-encrypted-sn...

[2]https://blog.cloudflare.com/encrypted-client-hello/

fullspectrumdev
1 replies
19h46m

Rather amusingly, the massive centralisation of the internet, which is bad for privacy in every other way, actually protects you here.

eddd-ddde
0 replies
18h24m

Yep, cloudflare dns proxy essentially hides your origin ip addresses.

prosody
0 replies
20h58m

I think it might become a little more useful in some contexts at least once ECH gets wider adoption. In that case afaik only the DNS server and the remote host you’re connecting to would know what FQDN you're connecting to—anyone else would only see the IP of the remote host. If the host serves requests for lots of different FQDNs, that’s less information going around.

johnmaguire
0 replies
20h54m

DNS leaks have historically been a thorn in the side of the Tor project, if I understand correctly.

Sure, you might have a SOCKS proxy configured for the HTTP request, but depending on how the program makes the DNS query, it might bypass that proxy:https://support.torproject.org/misc/check-socks-dns-leaks/

Encrypted DNS isn't a perfect solution, because the request still won't go through the proxy, but at least it won't leak the info to your ISP (assuming you have something else configured as your DNS provider.)

Obviously, this isn't limited to just Tor.

Erratic6576
0 replies
11h23m

Your connection is encrypted through https. Your DNS lookups aren’t yet though. Your ISP or network manager could have access to that data.

A vpn reduces speed and battery life

Etheryte
4 replies
1d2h

FYI the standalone encrypted DNS profiles for macOS (.mobileconfig files) do not work if you're using Little Snitch or Lulu to block unwanted network traffic. This is a limitation of macOS that Apple hasn't bothered to fix.

WirelessGigabit
2 replies
23h27m

Doesn't work how? Little Snitch / Lulu don't use the profile? As in DNS traffic is not encrypted?

Little Snitch / Lulu die?

Do you have like a ticket for more explanation?

Etheryte
1 replies
22h36m

You can have one or the other active at a time, but not both simultaneously. If you have the DNS profile turned on and then proceed to turn on Little Snitch/Lulu, the DNS profile will get turned off by macOS and whatever other DNS configuration you have will take precedence. It's mentioned in passing by Mullvad in the docs [0]:

If you have more than one profile added then macOS seems to use only the last one that you installed. If you remove a profile then it appears that it will not use any of the other profiles.

This isn't unique to Little Snitch/Lulu and Mullvad though, you would run into the same problem with any application that creates a network filter by using a profile, see [1] for example. Since the full blown Mullvad VPN app does not use system profiles, it does work in tandem with Little Snitch/Lulu.

[0]https://mullvad.net/en/help/dns-over-https-and-dns-over-tls#...

[1]https://git.codeproxy.net/paulmillr/encrypted-dns/issues/13

TheNewsIsHere
0 replies
2h3m

As a tangential aside: this limitation on Network Extensions isinfuriating. I can understand why it exists, but one of the results is that if you’re using an Endpoint Security extension, it can’t also act as a filter.

I’ve never been able to get CrowdStrike Falcon and Little Snitch to work on the same machine because of this. It would be nice if you could havenfilters (and a limit onnisn’t unreasonable).

Erratic6576
0 replies
11h26m

I guess the solution is to use unbound

lopkeny12ko
12 replies
1d1h

I don't understand this. Don't all Linux processes "run in RAM"? It sounds like what they haveactuallyaccomplished here is eliminate all disk I/O requirements from their DNS service, i.e. it works against a read-only mounted root filesystem now. That's not really the same as "running in RAM".

Etheryte
6 replies
1d

This is not what's happening here. The systems don't have a disk, period. The bootloader downloads an OS image from a provisioning server into RAM that's verified and then run. Past the bootloader, the system doesn't have access to the provisioning server. Every program the server needs or may need to run at one point is already in memory. Outside of RAM there is nowhere to read more data from, not even a read-only disk, nor anywhere to write it to.

ignoramous
2 replies
1d

If one is already running only "verified" (attested) code (with a read-only partition), diskless doesn't buy you much? Android, for example, does this: Thesystempartition where the OS is remains immutable unless there's a signed update (or the bootloader is "unlocked").

ziddoap
1 replies
1d

They briefly explain some of the reasoning here:https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...

If the computer is powered off, moved or confiscated, there is no data to retrieve.

We get the operational benefits of having fewer breakable parts. Disks are among the components that break often. Therefore, switching away from them makes our infrastructure more reliable.

The operational tasks of setting up and upgrading package versions on servers become faster and easier.

Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.
yjftsjthsd-h
0 replies
18h42m

Yeah, I don't have their opsec concerns but I'm very tempted to try implementing something similar just to enforce having a stateless system (easier to manage IMO) with a nice bonus of no longer caring about host disk failures.

drexlspivey
1 replies
20h31m

In such a setup where does the key to verify the image come from?

ziddoap
0 replies
18h11m

More info about stboot is available here:https://www.system-transparency.org/

and here:https://git.glasklar.is/system-transparency/core/system-tran...

The second link talks about network boot mode and signature validation.

lopkeny12ko
0 replies
22h1m

This seems like a very roundabout and unnecessary solution to a problem that is already solved with tmpfs.

oxygen_crisis
3 replies
1d

Once they have everything running in RAM, they can have racks of servers that contain no persistent storage whatsoever, and are therefore subpoena-proof.

On April 18 [2023] at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant. They intended to seize computers with customer data.

In line with our policies such customer data did not exist. We argued they had no reason to expect to find what they were looking for and any seizures would therefore be illegal under Swedish law. After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.

https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...

8organicbits
2 replies
1d

contain no persistent storage whatsoever, and are therefore subpoena-proof.

Nonsense. The linked post shows that they didn't have the sought information. If the information was present in RAM, they'd just read it from RAM.

theshrike79
0 replies
18h14m

Please tell me how you would go about reading something that happened two weeks ago from RAM?

itslennysfault
0 replies
20h8m

I assume they only hold what the server is doing in any given moment in ram and it is overwritten as new requests come in. By the time the police showed up with the subpoena the data they wanted was long gone.

ziddoap
0 replies
1d1h

In early 2022 we announced the beginning of our migration to using diskless infrastructure with our bootloader known as “stboot”.

We recently announced the completion of our migration to remove all traces of disks in use on our VPN infrastructure.

Is this more clear?

elif
10 replies
1d2h

I would like to see real statistics but my gut feeling from running read/write intensive data applications on SSD and ECC RAM is that both of them fail often enough that this move is somewhat lateral in terms of resiliency.

But in terms of clawing raw performance decimals, I applaud the effort. This would be a fun redis project.

stephen_g
8 replies
1d2h

That's not the point - they promise to keep no logs, and the best way to demonstrate that they don't is to not even have any fixed storage in the servers at all (they presumably netboot from a read-only image). So they have done this with their VPN servers, now they are rolling out the strategy to DNS servers as well.

elif
3 replies
1d2h

That's fine but as long as it is connected to a network you cannot prove a system to be log-free.

The majority of systems I've worked on used a message queue of some kind for logging rather than a file on disk.

I suppose it could be useful to expedite downtime in the case that they are subpoenaed, but I doubt claiming to be diskless would prevent any subpoena.

ziddoap
0 replies
1d

but I doubt claiming to be diskless would prevent any subpoena.

It has already.

Seehttps://news.ycombinator.com/item?id=38220769

bragr
0 replies
1d1h

With this kind of security, you're more guarding against your servers being seized or intelligence agency installing an implant.

I doubt claiming to be diskless would prevent any subpoena

It does in Sweden. They've repeatedly squashed subpoenas

long as it is connected to a network you cannot prove a system to be log-free.

Nothing is perfect. If you read their audits, it's clear admins could get at your traffic if they wanted

ThePowerOfFuet
0 replies
1d1h

This is why they have their architecture audited (and release the audit reports).

https://mullvad.net/en/blog/2022/6/22/vpn-server-audit-found...

Thaxll
2 replies
1d1h

How do they troubleshoot issues without logs exactly?

dboreham
0 replies
1d1h

By shipping the logs to another box.

bragr
0 replies
1d1h

Live monitoring and test instances. They have ssh access to prod

waynesonfire
0 replies
1d1h

Logging doesn't require disks, the logs can be sent over the network.

Lord-Jobo
0 replies
2h31m

You've seen ECC fail often enough? Ours has been incredibly reliable, but we don't have enormous bandwidth. do you have a data sheet or anything showing failure rates?

deknos
10 replies
1d1h

they should at least opensource their implementation. they may have errors there. should we just trust everything?

k8sToGo
2 replies
1d1h

Pretty sure they have been audited recently.

willis936
0 replies
23h8m

If I were a Mullvad customer this is what I would want: regular audits and closed source until everyone was convinced the code was bulletproof.

dewey
0 replies
1d1h
dewey
2 replies
1d1h

Why the entitled attitude? They do a lot of open source contributions already so maybe just give them some time instead of immediately disregarding the whole improvement? Small steps into the right direction.

https://mullvad.net/en/help/open-source

zlg_codes
1 replies
20h25m

Making a suggestion isn't entitlement. Maybe reconsider the foul void that accusation came from.

dewey
0 replies
20h17m

If someone suggests something to me they usually don't say "you should at least". This sounded more like an entitled comment you see every day on open source repositories and making it sometimes not fun to be an open source maintainer.

carlhjerpe
1 replies
20h2m

Their infrastructure kind of is their USP. It'd be like giving their competition their work for free. And it's not about absolute trust, it's about trusting them more than your ISP and other VPN providers. You still have TLS over Mullvad

yjftsjthsd-h
0 replies
18h40m

I'd argue that Mullvad's USP/moat is their reputation, far more than the software they run.

jcul
0 replies
7h15m

They are just using wireguard.

You can connect with wireguard and not run any mullvad software on the client side.

Though that's generally the case with VPN providers, whether openvpn, wireguard, etc.

I guess they could introduce some vulnerability on the server side if they rolled their own wireguard implementation or patched it. Though it seems unlikely to me.

https://mullvad.net/en/help/easy-wireguard-mullvad-setup-lin...

dfc
0 replies
15h33m

They have open sourced a good deal of the infrastructure:

https://github.com/mullvad/system-transparency

j-bos
7 replies
1d

Does anyone know how Mullvad vpn security compares with Proton vpn?

drexlspivey
4 replies
20h38m

"In April 2019, upon the order of the Swiss judiciary in a case of clear criminal conduct, we enabled IP logging against a specific user account which is engaged in illegal activities which contravene Swiss law. Pursuant to Swiss law, the user in question will also be notified and afforded the opportunity to defend against this in court before the data can be used in criminal proceedings"

woofcat
2 replies
20h7m

Who is this in reference to? What is the source? Googling for the quote you gave doesn't have great results.

ziddoap
0 replies
19h28m

See my comment above, but it is in reference to ProtonMail.

You can find many articles about it (search "Protonmail subpoena 2019"), but an example article is here:https://www.privacyaffairs.com/protonmail-surrenders-user-lo...

ProtonMail's own blog post about it is here:https://proton.me/blog/climate-activist-arrest

drexlspivey
0 replies
19h20m
ziddoap
0 replies
20h10m

A quote with no context or attribution always makes it fun to figure out what someone intends to say with their comment.

This quote is in reference to ProtonMail in 2019, a fairly widely reported on case at the time.

This would be a point in favor of Mullvad, as Mullvad has demonstrably squashed a subpoena in the past, thanks to their data-minimization policies (no PII for signup, no PII options for payment, diskless infrastructure, etc.).

Note that this is more related toprivacy, and not as muchsecurity. If the parent comment was specifically talking about security, I would recommend reviewing the software and infrastructure audits made available by either company. I'm not aware of any major fault with either, from a security perspective.

theshrike79
0 replies
18h16m

Mullvad is pretty much the #1 when you want privacy. They have a history of fighting away law enforcement requests, because they really don't have anything to give to them.

Everything runs on RAM and will disappear the second a server is shutdown, they really don't store anything.

And you can buy a Mullvad account by mailing cash to them or with crypto. No need to create an account with an email address or anything.

__turbobrew__
0 replies
16h25m

vpn’s don’t provide security. The only thing they do is circumvent geoblocks and maybe prevent dmca notices when downloading pirated content.

rbut
4 replies
20h12m

I use Cloudflare's DoH DNS (https://1.1.1.1) and tunnel that over Mullvad's VPN.

That way Mullvad can only see the IPs I visit, Cloudflare only see DNS requests coming from the VPN IP, and my ISP sees nothing.

Mullvad's DoH DNS seems nice, but it provides potentially less privacy than the above, so I won't be using it.

ThePowerOfFuet
2 replies
20h6m

That way Mullvad can only see the IPs I visit,

I hate to break it to you, but if they wanted to look at the SNI on all your TLS connections, they could; ECH is not really a thing yet.

rbut
1 replies
19h9m

Yes you're correct if Mullvad is performing packet inspection then there isn't much I can do about that.

My point stands though about not putting all eggs in the one basket and using a different DNS provider than that of the VPN provider.

tucnak
0 replies
7h52m

No, I don't think you understand. "That way Mullvad can only see the IPs I visit" is your point and it _doesn't_ stand. There isn't really multiple ways of looking at it; the point you were making is exactly the point that was refuted, and all this "my point stands" is the weakest shit in the whole white world— a compulsion, almost inertial in nature, where by attempting to save face we present ourselves clueless. Now, all this "putting all eggs in the one basket" and common sense knowledge would be very adorable had you not been wrong, and had it not been a bad advice leaving you at a disadvantage. Good luck

coldblues
0 replies
9h59m

I don't think it's recommended to use another DNS with Mullvad because it makes you more unique.

naet
1 replies
22h50m

I was extremely happy with everything mullvad was doing until they announced they were discontinuing port forwarding, which is important to a niche part of my setup. I understand the reasoning, but if it came back ever I would again happily be a customer.

vGPU
0 replies
13h6m

VPN port forwarding is basically a “botnet operators welcome” sign from what I understand. The benefits of disabling it are greater than the downsides for Mullvad.

herpderperator
1 replies
19h59m

Today we can announce more steps forward - our Encrypted DNS service has also been converted to run from RAM!

Where exactly do services run from if not RAM? The post is very plain with no explanation of what "running from RAM" means. When you execute any binary it gets loaded into RAM as per standard operating system procedures...?

10000truths
0 replies
19h51m

They mean diskless. There's no persistent storage attached to the machines that run the DNS servers, because the entire operating system is loaded into RAM directly from PXE boot. So, if the hardware ever gets seized by law enforcement or rogue actors, there is no risk of exfiltration of private user data.

UberFly
0 replies
16h26m

VPN provider OVPN (ovpn.com) has been doing this as well.