Moving our Encrypted DNS servers to run in RAM(mullvad.net)
mullvad.net
Moving our Encrypted DNS servers to run in RAM
https://mullvad.net/en/blog/moving-our-encrypted-dns-servers-to-run-in-ram
13 comments
FYI the standalone encrypted DNS profiles for macOS (.mobileconfig files) do not work if you're using Little Snitch or Lulu to block unwanted network traffic. This is a limitation of macOS that Apple hasn't bothered to fix.
Doesn't work how? Little Snitch / Lulu don't use the profile? As in DNS traffic is not encrypted?
Little Snitch / Lulu die?
Do you have like a ticket for more explanation?
Little Snitch / Lulu die?
Do you have like a ticket for more explanation?
You can have one or the other active at a time, but not both simultaneously. If you have the DNS profile turned on and then proceed to turn on Little Snitch/Lulu, the DNS profile will get turned off by macOS and whatever other DNS configuration you have will take precedence. It's mentioned in passing by Mullvad in the docs [0]:
> If you have more than one profile added then macOS seems to use only the last one that you installed. If you remove a profile then it appears that it will not use any of the other profiles.
This isn't unique to Little Snitch/Lulu and Mullvad though, you would run into the same problem with any application that creates a network filter by using a profile, see [1] for example. Since the full blown Mullvad VPN app does not use system profiles, it does work in tandem with Little Snitch/Lulu.
[0] https://mullvad.net/en/help/dns-over-https-and-dns-over-tls#...
[1] https://git.codeproxy.net/paulmillr/encrypted-dns/issues/13
> If you have more than one profile added then macOS seems to use only the last one that you installed. If you remove a profile then it appears that it will not use any of the other profiles.
This isn't unique to Little Snitch/Lulu and Mullvad though, you would run into the same problem with any application that creates a network filter by using a profile, see [1] for example. Since the full blown Mullvad VPN app does not use system profiles, it does work in tandem with Little Snitch/Lulu.
[0] https://mullvad.net/en/help/dns-over-https-and-dns-over-tls#...
[1] https://git.codeproxy.net/paulmillr/encrypted-dns/issues/13
As a tangential aside: this limitation on Network Extensions is infuriating. I can understand why it exists, but one of the results is that if you’re using an Endpoint Security extension, it can’t also act as a filter.
I’ve never been able to get CrowdStrike Falcon and Little Snitch to work on the same machine because of this. It would be nice if you could have n filters (and a limit on n isn’t unreasonable).
I’ve never been able to get CrowdStrike Falcon and Little Snitch to work on the same machine because of this. It would be nice if you could have n filters (and a limit on n isn’t unreasonable).
Even when I don’t use any firewall app, the DNS profile doesn’t work on macOS. There is a bug that other apps (and even system) can bypass it.
Not like in iOS, DNS profile work very well.
Not like in iOS, DNS profile work very well.
I guess the solution is to use unbound
Can you help me out with the threat model for encrypted DNS?
As I see it, the model is that I need to contact a site, and use DNS to get the IP - but either my ISP or my VPN provider can see I’m visiting that IP whether I looked it up just now, or knew it already.
So unless the model is I’m using a VPN, but am leaking via someone else’s logging DNS, I don’t really get it.
As I see it, the model is that I need to contact a site, and use DNS to get the IP - but either my ISP or my VPN provider can see I’m visiting that IP whether I looked it up just now, or knew it already.
So unless the model is I’m using a VPN, but am leaking via someone else’s logging DNS, I don’t really get it.
Personally my main motivation is I don't want anyone to snoop what sites I visit and then sell that data, my ISP included, but there are other issues too. To quote Cloudflare [0]:
> Queries could be directed to a resolver that performs DNS hijacking. For example, in the UK, Virgin Media and BT return a fake response for domains that do not exist, redirecting users to a search page. This redirection is possible because the computer/phone blindly trusts the DNS resolver that was advertised using DHCP by the ISP-provided gateway router.
[0] https://blog.cloudflare.com/dns-encryption-explained/
> Queries could be directed to a resolver that performs DNS hijacking. For example, in the UK, Virgin Media and BT return a fake response for domains that do not exist, redirecting users to a search page. This redirection is possible because the computer/phone blindly trusts the DNS resolver that was advertised using DHCP by the ISP-provided gateway router.
[0] https://blog.cloudflare.com/dns-encryption-explained/
Cant your ISP reverse lookup the IPs you connect to based on your IP headers anyways?
Rather amusingly, the massive centralisation of the internet, which is bad for privacy in every other way, actually protects you here.
Yep, cloudflare dns proxy essentially hides your origin ip addresses.
They can, but pretty much everything will reverse lookup to Cloudflare or AWS.
SNI reveals the exact site though
As I understand it, this is now being addressed by ESNI[1] and ECH[2]. Hopefully ECH will continue being more widely deployed.
[1] https://www.cloudflare.com/learning/ssl/what-is-encrypted-sn...
[2] https://blog.cloudflare.com/encrypted-client-hello/
[1] https://www.cloudflare.com/learning/ssl/what-is-encrypted-sn...
[2] https://blog.cloudflare.com/encrypted-client-hello/
DNS leaks have historically been a thorn in the side of the Tor project, if I understand correctly.
Sure, you might have a SOCKS proxy configured for the HTTP request, but depending on how the program makes the DNS query, it might bypass that proxy: https://support.torproject.org/misc/check-socks-dns-leaks/
Encrypted DNS isn't a perfect solution, because the request still won't go through the proxy, but at least it won't leak the info to your ISP (assuming you have something else configured as your DNS provider.)
Obviously, this isn't limited to just Tor.
Sure, you might have a SOCKS proxy configured for the HTTP request, but depending on how the program makes the DNS query, it might bypass that proxy: https://support.torproject.org/misc/check-socks-dns-leaks/
Encrypted DNS isn't a perfect solution, because the request still won't go through the proxy, but at least it won't leak the info to your ISP (assuming you have something else configured as your DNS provider.)
Obviously, this isn't limited to just Tor.
I think it might become a little more useful in some contexts at least once ECH gets wider adoption. In that case afaik only the DNS server and the remote host you’re connecting to would know what FQDN you're connecting to—anyone else would only see the IP of the remote host. If the host serves requests for lots of different FQDNs, that’s less information going around.
Your connection is encrypted through https. Your DNS lookups aren’t yet though. Your ISP or network manager could have access to that data.
A vpn reduces speed and battery life
A vpn reduces speed and battery life
Good point.
One scenario comes to mind: multiple services under the same IP. “cutecats.com” and “hotintercourse.net” might both resolve to the same IP, but a reverse proxy resolves requests differently depending on the host. Then, knowing the IP but not the host, as your ISP does, becomes a meaningful difference.
One scenario comes to mind: multiple services under the same IP. “cutecats.com” and “hotintercourse.net” might both resolve to the same IP, but a reverse proxy resolves requests differently depending on the host. Then, knowing the IP but not the host, as your ISP does, becomes a meaningful difference.
Are these physical servers or VM's? I ask because some VM types can be frozen/shapshot/cloned or live replicated including memory contents as is done in some VPS providers for lawful requests. From the bare metal host anything can be accessed in a VM or container. Do they have a diagram of their physical setup?
[Edit] - Are there any Mullvad architects here that can help us avoid going down theoretical hypothetical rabbit holes and turtle stacks?
[Edit] - Are there any Mullvad architects here that can help us avoid going down theoretical hypothetical rabbit holes and turtle stacks?
Mullvad has been pushing system transparency a lot (which is a security system specifically designed for bare-metal), so it's fairly safe to say that it's based on bare-metal.
https://www.system-transparency.org
https://www.system-transparency.org
Maybe it should be safe to say? but do they actually have a diagram of their physical and logical setup that shows they use bare metal only and not VM's or containers? Is that documented in audited security controls?
Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security.
It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security.
Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question.
They've also discussed booting UEFI directly to VPN nodes: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.
It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security.
Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question.
They've also discussed booting UEFI directly to VPN nodes: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.
> Containers, btw, cannot be snapshotted.
lxc-snapshot(1), podman-container-checkpoint(1), and docker-checkpoint-create(1) all beg to differ.
lxc-snapshot(1), podman-container-checkpoint(1), and docker-checkpoint-create(1) all beg to differ.
Some (maybe all) of these are backed by CRIU: https://criu.org/Main_Page
Docker at least is
Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security.
I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.
Containers, btw, cannot be snapshotted.
I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted.
They've also discussed booting UEFI directly to VPN nodes:
That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this.
It's completely weird to argue that they might introduce a layer of insecurity here.
Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentation unless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.
I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.
Containers, btw, cannot be snapshotted.
I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted.
They've also discussed booting UEFI directly to VPN nodes:
That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this.
It's completely weird to argue that they might introduce a layer of insecurity here.
Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentation unless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.
>They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.
They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit.
>On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant.
>After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.
https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit.
>On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant.
>After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.
https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
I remember that and that case was cool. I will leave my questions in place though because I recall Apple doing the same thing when a terrorist phone was locked and they claimed to not be able to access with a big public show/fight it but turns out they could and so could the FBI. My theory is that they did not want the public to lose confidence in their phones.
Apple would've had to create a specific OS update with a security hole the FBI could've exploited and distributed that.
Courts couldn't force them to do that -> FBI went another way. This was in the iPhone 7 era IIRC.
Currently their stuff is locked down even tighter and Apple has even less ability to hack anyone's phone. Barring a full-on backdoored software update targeted to a specific person - which they refused to do once already.
Courts couldn't force them to do that -> FBI went another way. This was in the iPhone 7 era IIRC.
Currently their stuff is locked down even tighter and Apple has even less ability to hack anyone's phone. Barring a full-on backdoored software update targeted to a specific person - which they refused to do once already.
Yeah, that Apple case was really interesting. I always guessed that Apple could simply push a software update (possibly to everyone) that happens to open a backdoor for a specific phone. I wonder if that's what they did.
What happened in that case is the FBI went to a different vendor, and they broke into the iPhone either with a zero day they had developed or more likely, they just cloned the phone hardware thousands of times until they could guess the password.
Yep, Apple had nothing to do with it. They do hand over iCloud data (which now can be E2E)
Maybe - depends on how secure enclave is built. It may have hardware defined limitations on # of tries for the passkey and no way to circumvent that in firmware even.
> I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.
"Here is a subpoena compelling you to disclose the data you have on XYZ."
"Sure. Here is the data we have on XYZ." hands over blank page
It's not sidestepping the request. They literally do not have the data, because they don't retain it. And unless there is a specific law mandating that they retain the data, law enforcement have no grounds for punitive action.
"Here is a subpoena compelling you to disclose the data you have on XYZ."
"Sure. Here is the data we have on XYZ." hands over blank page
It's not sidestepping the request. They literally do not have the data, because they don't retain it. And unless there is a specific law mandating that they retain the data, law enforcement have no grounds for punitive action.
> Containers, btw, cannot be snapshotted
Why not?
Why not?
criu has been used for live migration for a while.
Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.
Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.
I think it's a very rational question to ask. Is Mullvad using only bare-metal infrastructure, or is there any logical abstractions added to the servers?
Companies make statements all the time which can be twisted to their benefit, but read exactly like what a customer is after. I'm not stating Mullvad does or doesn't do this, btw.
Companies make statements all the time which can be twisted to their benefit, but read exactly like what a customer is after. I'm not stating Mullvad does or doesn't do this, btw.
Even if they did... if mullvad was a honeypot, there is nothing that would stop them from posting a fake diagram.
Physical server can also be frozen (literally) and then data extracted from RAM.
It can but to my knowledge law enforcement in the US anyway have only done this once for a national security incident in the real world outside of a lab. If you have any links to this becoming common practice outside of a lab demo or security presentation on their own hardware I would be interested to save in my own documentation.
They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went.
[Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocation with a performance hit. Both are enabled by default on modern kernels now.
They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went.
[Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocation with a performance hit. Both are enabled by default on modern kernels now.
init_on_alloc=1 init_on_free=1There exist special UPSes that you can attach to a power line of a server, unplug and unmount it, and keep the server powered all the time, without the server being able to detect that.
If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.
If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.
Just like Seinfeld and saving the Frogger high score.
Whoa! Any keyword suggestions to find those UPSes?
I'm having trouble remembering what they're called, but I do remember reading about these. I saw a demonstration where they slightly pulled at the desktop power plug in order to hook a thin device around the plug connectors to provide power, and then they could transport the desktop around.
For laptops, law enforcement use "mouse jigglers" to prevent screen locks and laptops going to sleep, which I guess is similar. Random article: https://www.electronicproducts.com/what-is-a-mouse-jiggler-a...
It becomes way more straightforward if it's a server with redundant power supplies.
Actually not because a server with multiple PSUs definitely knows when one loses power. It’s made to raise alerts about a power rail failing. So this makes it harder because you need to keep power on both to not raise any alarm
[deleted]
[deleted]
That is why modern servers have the option to encrypt the DRAM to prevent this attack.
Moreover, there is also the option to have distinct randomly-generated keys for each virtual machine, to make useless the speculative attacks that succeed to peek at the memory of another VM.
AMD has offered these options for years and the future Intel CPUs will also have them.
Moreover, there is also the option to have distinct randomly-generated keys for each virtual machine, to make useless the speculative attacks that succeed to peek at the memory of another VM.
AMD has offered these options for years and the future Intel CPUs will also have them.
good ole can of air compressor flipped upside down and quick hands trick
Are there any known examples of this actually being done by law enforcement or government and working?
There are plenty of examples of it working (e.g. researchers from F-Secure presented at SEC-T conference in 2018 with a cold-boot against Bitlocker), but I haven't seen a LEA or government come out and say "we got access to X via a cold-boot attack".
Although, obviously, it was the same story with dragnet spying before Snowden. Sure it was technically possible in a lab environment and law enforcement had incentive to do it, but nobody could turn up any actual evidence.
As RAM densities/speed increase, I wonder if this is still viable. DDR5 had to mandate some baseline ECC given the intrinsic error rate.
The more I think about this VM's are not required. There are tools that can extract data streams based on specific parameters on demand using eBPF. Perhaps tools like Sysdig/Falco, etc... It might also be useful to know if they custom compile kernels to disable eBPF and if they do any additional specific kernel hardening to strip out or disable on demand debugging capabilities.
One can fix that by deploying software with the VM pause functionality removed and / or remove disks the bare-metal servers and only PXE-boot them. That way there is nothing to pause.
If you are asking this, you probably need to self host your VPN
I do this but I do not use them for privacy. I use Tinc [1] to route around internet outages as it can do user-space dynamic mesh routing. Since they are all nodes I pay for they would not hide who I am. While I do control the logging of Tinc I do not control external logging of the server and VPS providers.
For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi.
[1] - https://www.tinc-vpn.org/
For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi.
[1] - https://www.tinc-vpn.org/
As a naive outsider: Does RAM in 2023 encrypt all its contents?
Can we reliably "forget" the contents of what's in RAM by powering-off and letting some encryption key somewhere fade from a smaller, more volatile piece of memory? I'm curious what has been done to mitigate cold-boot attacks in RAM. What has changed in hardware? Are the keys to encrypt contents of RAM kept in the TPM or something?
Can we reliably "forget" the contents of what's in RAM by powering-off and letting some encryption key somewhere fade from a smaller, more volatile piece of memory? I'm curious what has been done to mitigate cold-boot attacks in RAM. What has changed in hardware? Are the keys to encrypt contents of RAM kept in the TPM or something?
In this case I think the point of running from RAM is that there isn't non-volatile storage. Of course, someone who has physically compromised the server could do potentially bad things, but this makes it harder to e.g. persistently compromise a server, since it doesn't have storage to store malware on persistently, or e.g. accidentally or intentionally log data, since there is no local storage to log it on.
Encrypting what's in RAM would possibly add some additional protection but realistically if things are done right what's left in RAM will probably be only marginally more interesting than what you can see over the wire; at most I'd expect you could see a tiny amount of active request data. That's just a guess, though.
Encrypting what's in RAM would possibly add some additional protection but realistically if things are done right what's left in RAM will probably be only marginally more interesting than what you can see over the wire; at most I'd expect you could see a tiny amount of active request data. That's just a guess, though.
> since it doesn't have storage to store malware on persistently, or e.g. accidentally or intentionally log data, since there is no local storage to log it on.
Does this prevent rootkits?
> but realistically if things are done right what's left in RAM will probably be only marginally more interesting
See: https://en.wikipedia.org/wiki/Cold_boot_attack
Does this prevent rootkits?
> but realistically if things are done right what's left in RAM will probably be only marginally more interesting
See: https://en.wikipedia.org/wiki/Cold_boot_attack
> Does this prevet rootkits?
That's the idea. If the OS is immutable, where do you put them?
Bootkits require further mitigation to prevent, but Mullvad has also put effort into that, too, with their secure coreboot bootchain.
Of course as an end-user you still can't really fully validate a machine over the network has not been tampered with, so you're obviously ultimately still trusting Mullvad as you'd expect.
> See: https://en.wikipedia.org/wiki/Cold_boot_attack
Yes, that's what I'm referring to. I don't expect that there would be a significant amount of requests in RAM, probably just remnants of mainly what was active (and some de-allocated but not cleared remnants?) when the machine was reset.
That's the idea. If the OS is immutable, where do you put them?
Bootkits require further mitigation to prevent, but Mullvad has also put effort into that, too, with their secure coreboot bootchain.
Of course as an end-user you still can't really fully validate a machine over the network has not been tampered with, so you're obviously ultimately still trusting Mullvad as you'd expect.
> See: https://en.wikipedia.org/wiki/Cold_boot_attack
Yes, that's what I'm referring to. I don't expect that there would be a significant amount of requests in RAM, probably just remnants of mainly what was active (and some de-allocated but not cleared remnants?) when the machine was reset.
That's the idea. If the OS is immutable, where do you put them?
They would be in the PXE image. There was one time I flubbed on this actually. I had to change export options on our PXE servers to fix something and I forgot to change it back adding my poor excuse of multitasking and being lazy. This was in a Dev environment we also used PXE in production believe it or not. A developer sitting across from me said he changed something in the image by mistake and I said, "That's not possi... oh crap." It was cool of them to call it out right away. Attackers probably won't be so kind.
They would be in the PXE image. There was one time I flubbed on this actually. I had to change export options on our PXE servers to fix something and I forgot to change it back adding my poor excuse of multitasking and being lazy. This was in a Dev environment we also used PXE in production believe it or not. A developer sitting across from me said he changed something in the image by mistake and I said, "That's not possi... oh crap." It was cool of them to call it out right away. Attackers probably won't be so kind.
For what it's worth though, that's possible to mitigate. I don't believe Mullvad uses PXE and they do have a "secure boot" validation chain, starting from Coreboot on the hardware.
https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
It is an improvement, for sure.
secure boot, as painful it is to get right, doesn't help when the vendor runs its own things with elevated privileges, like they are known to do: https://news.ycombinator.com/item?id=34085635
Doubly so, when the said things are not only closed source but vulnerable themselves: https://archive.is/6ClWm
Does coreboot help in mitigating vuln in privilged vendor blobs?
secure boot, as painful it is to get right, doesn't help when the vendor runs its own things with elevated privileges, like they are known to do: https://news.ycombinator.com/item?id=34085635
Doubly so, when the said things are not only closed source but vulnerable themselves: https://archive.is/6ClWm
Does coreboot help in mitigating vuln in privilged vendor blobs?
Well, with Coreboot, you at least know the very first instructions executed by the x86 cores are under your control. That's all it can really do. Whatever else lurks beneath is really impossible for anyone to scrutinize without insane capital or insider knowledge.
Mitigating the risk of Intel ME is obviously something that is complicated. The "state of the art" so-to-speak is really just rendering it non-functional. This isn't strictly related to Coreboot; that said, Coreboot has a page about ME:
https://www.coreboot.org/Intel_Management_Engine
And in general, when you're running Coreboot, you're running significantly less privileged blackbox blobs (possibly none in some rare cases), and in general, the surface area of Coreboot is going to be dramatically tinier than any typical UEFI system firmware. So it's absolutely worth it in that regard.
There's never going to be a "perfect" option here with how complex things have gotten, but I think that you can at least say that Mullvad has taken almost any reasonable step, and perhaps a few steps that are beyond what many would consider reasonable, towards developing effective mitigation of the risks of rootkits, bootkits and malicious/unwanted vendor code running on their "diskless" architecture.
Of course, you still have to trust Mullvad. Trust is definitely a meaty human-y experience, and the cold rational part of our brain craves some kind of technically-bulletproof way to prove that it's truly safe, secure, and private, but that's just not possible. Best you can do is layer enough assurances on top of each-other in a way that at least some layers are completely independent of others for maintaining your operational security. Far easier said than done, and obviously how rigorously one approaches this will vary on paranoia.
At some point you have to pick your middle-ground I suppose.
Mitigating the risk of Intel ME is obviously something that is complicated. The "state of the art" so-to-speak is really just rendering it non-functional. This isn't strictly related to Coreboot; that said, Coreboot has a page about ME:
https://www.coreboot.org/Intel_Management_Engine
And in general, when you're running Coreboot, you're running significantly less privileged blackbox blobs (possibly none in some rare cases), and in general, the surface area of Coreboot is going to be dramatically tinier than any typical UEFI system firmware. So it's absolutely worth it in that regard.
There's never going to be a "perfect" option here with how complex things have gotten, but I think that you can at least say that Mullvad has taken almost any reasonable step, and perhaps a few steps that are beyond what many would consider reasonable, towards developing effective mitigation of the risks of rootkits, bootkits and malicious/unwanted vendor code running on their "diskless" architecture.
Of course, you still have to trust Mullvad. Trust is definitely a meaty human-y experience, and the cold rational part of our brain craves some kind of technically-bulletproof way to prove that it's truly safe, secure, and private, but that's just not possible. Best you can do is layer enough assurances on top of each-other in a way that at least some layers are completely independent of others for maintaining your operational security. Far easier said than done, and obviously how rigorously one approaches this will vary on paranoia.
At some point you have to pick your middle-ground I suppose.
Thank you. Appreciate your answers.
> we also used PXE in production believe it or not
Wait why would this be a bad thing?
Wait why would this be a bad thing?
It depends on your use case, SLA's and configuration management role/structure discipline. There are different schools of thought on diskless boot which is not truly diskless as the images have to be shared from somewhere meaning that disk availability goes from 1:1 to many to one increasing the blast radius of outages when the NAS fails or network re-convergence occurs. They all fail regardless of how many millions one spends and in my experience the more millions the more glorious the failure that should never happen. For some companies this is fine if they have the discipline to balance their servers across many NAS servers in a strict hierarchy that facilitates a smaller predictable POD structure blast radius. But that's a bigger topic than I could possibly cover in a HN comment. It also depends on how the OS is being used and how writable spaces tmpfs vs disk are being used. This will all vary wildly across different business models.
AMD offers RAM encryption on their threadripper server CPUs and also in their Ryzen PRO on workstation/laptops (you have to enable it in the bios).
https://www.amd.com/system/files/documents/amd-memory-guard-...
https://www.amd.com/content/dam/amd/en/documents/epyc-busine...
https://www.amd.com/system/files/documents/amd-memory-guard-...
https://www.amd.com/content/dam/amd/en/documents/epyc-busine...
Years ago, I went to AMD and saw this demonstrated live by engineers there. They spun up a VM with it turned off and you could basically grep a string out of ram. Turning it on, grep returned nothing. It was pretty cool. Developed by a bunch of true grey beards over the course of many years.
On consumer Ryzen chips you can enable it too if your motherboard supports it. Like error correction code (ECC) support on consumer chips it isn't "officially" supported but it works. I know MSI and ASUS (IIRC) have the option in bios for transparent memory encryption.
> As a naive outsider: Does RAM in 2023 encrypt all its contents?
No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed).
There's even an EFI reboot flag that could fix this and force-clears the RAM on next boot, but it's usually deactivated.
[1] https://en.wikipedia.org/wiki/Cold_boot_attack (works still on Windows, thanks to BitLocker)
[2] https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...
No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed).
There's even an EFI reboot flag that could fix this and force-clears the RAM on next boot, but it's usually deactivated.
[1] https://en.wikipedia.org/wiki/Cold_boot_attack (works still on Windows, thanks to BitLocker)
[2] https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...
This isn't correct - the key is stored inside the CPU.
Did you read any of my referenced links? You cannot state this when the paper and well known attack methods disprove your statement.
Bitlocker does not store the keys in the CPU, otherwise a coldboot attack while literally transplanting the physical memory rows to a different machine wouldn't work. And it does.
In case you are confusing CPU with a TPM chip, which can optionally store the master volume key but only for activated full disk encryption:
With TPM you can easily bypass that, too, with tools that can read the i2c bus. See here [1]
[1] https://blog.scrt.ch/2021/11/15/tpm-sniffing/
Bitlocker does not store the keys in the CPU, otherwise a coldboot attack while literally transplanting the physical memory rows to a different machine wouldn't work. And it does.
In case you are confusing CPU with a TPM chip, which can optionally store the master volume key but only for activated full disk encryption:
With TPM you can easily bypass that, too, with tools that can read the i2c bus. See here [1]
[1] https://blog.scrt.ch/2021/11/15/tpm-sniffing/
Bitlocker is disk encryption not ram. Ram encryption like AMD's Secure Memory Encryption (SME), transparent secure memory encryption, and Intel's Total Memory Encryption (TME) all store the encryption key used to encrypt the RAM inside the CPU. The encryption key never leaves the CPU so it can't be read from RAM via cold boot attack. As far as I know no attacks have been shown to expose this key (which is changed on every reboot).
Game consoles have been using RAM encryption for over a decade. It's one of the reasons consoles are much harder to attack nowadays.
Game consoles have been using RAM encryption for over a decade. It's one of the reasons consoles are much harder to attack nowadays.
Okay, if we are narrowing the topic now to AMD's SME/SEV and Intel's TME/TDX, I'll bite, too.
First: Only Ryzen PRO or EPYC models support it, which kind of kicks out all mobile or laptop systems already. Then, only Zen3 CPUs work, because previous generations have a boot freeze bug, which wasn't fixed and upstream linux 5.15 as a result disabled the mem encrypt flag by default.
Second: Before you switch topic to SEV, that's only supported for EPYC models, see here [2]
Regarding attacks: At least AMD had an injection attack problem where SEV in EPYC 7xxx and 3xxx processors was confirmed to be affected without AMD confirming the vulnerability (yet...). It was a master thesis iirc from a guy in luebeck.
There are also known sidechannel attacks which void RAM encryption in practice, like Hertzbleed which used frequency scaling to decrypt ECDSA and PIKE SIDH (which is meanwhile known to be unsecure, at least for PIKE). [3]
Google also did an audit on Intel's TDX where they found bugs in loop boundaries, off by one errors and similar feasible attack methods (which haven't been published as a PoC yet, so I grant you that). [4]
So I would still argue that with these very narrow set of available processors (Intel Pro 13th generation for TME and EPYC 7xxx that have both SME and SEV) is highly limited in its availability and also not available for laptop hardware due to them being server CPUs.
Additionally there's been a lot of attack surfaces that have been proven to have access to SME or SEV stored keys in the CPU and there have been other sidechannel attacks which conceptionally are very unlikely to be fixed anytime soon.
So I would still argue that memory encryption in practice is unreliable.
[1] https://lists.freedesktop.org/archives/amd-gfx/2021-October/...
[2] https://github.com/AMDESE/AMDSEV/issues/1
[3] https://www.hertzbleed.com/
[4] https://services.google.com/fh/files/misc/intel_tdx_-_full_r...
Edit: found the AMD injection attack thesis:
[01] https://www.its.uni-luebeck.de/fileadmin/files/theses/MA_Luc...
Another attack on SEV, which was confirmed by others since the USENIX conference. Both of the techniques rely heavily on pattern matching to find the decryption oracles though, and around 16 bytes for their OpenSSH demonstrations.
[02] https://www.its.uni-luebeck.de/fileadmin/files/theses/MA_Luc...
I'm gonna ignore the game consoles statement because that's a specialized set of CPUs which will never land in any laptop anytime soon. Most of them are AMD, so I'd refer to the same articles anyways.
First: Only Ryzen PRO or EPYC models support it, which kind of kicks out all mobile or laptop systems already. Then, only Zen3 CPUs work, because previous generations have a boot freeze bug, which wasn't fixed and upstream linux 5.15 as a result disabled the mem encrypt flag by default.
Second: Before you switch topic to SEV, that's only supported for EPYC models, see here [2]
Regarding attacks: At least AMD had an injection attack problem where SEV in EPYC 7xxx and 3xxx processors was confirmed to be affected without AMD confirming the vulnerability (yet...). It was a master thesis iirc from a guy in luebeck.
There are also known sidechannel attacks which void RAM encryption in practice, like Hertzbleed which used frequency scaling to decrypt ECDSA and PIKE SIDH (which is meanwhile known to be unsecure, at least for PIKE). [3]
Google also did an audit on Intel's TDX where they found bugs in loop boundaries, off by one errors and similar feasible attack methods (which haven't been published as a PoC yet, so I grant you that). [4]
So I would still argue that with these very narrow set of available processors (Intel Pro 13th generation for TME and EPYC 7xxx that have both SME and SEV) is highly limited in its availability and also not available for laptop hardware due to them being server CPUs.
Additionally there's been a lot of attack surfaces that have been proven to have access to SME or SEV stored keys in the CPU and there have been other sidechannel attacks which conceptionally are very unlikely to be fixed anytime soon.
So I would still argue that memory encryption in practice is unreliable.
[1] https://lists.freedesktop.org/archives/amd-gfx/2021-October/...
[2] https://github.com/AMDESE/AMDSEV/issues/1
[3] https://www.hertzbleed.com/
[4] https://services.google.com/fh/files/misc/intel_tdx_-_full_r...
Edit: found the AMD injection attack thesis:
[01] https://www.its.uni-luebeck.de/fileadmin/files/theses/MA_Luc...
Another attack on SEV, which was confirmed by others since the USENIX conference. Both of the techniques rely heavily on pattern matching to find the decryption oracles though, and around 16 bytes for their OpenSSH demonstrations.
[02] https://www.its.uni-luebeck.de/fileadmin/files/theses/MA_Luc...
I'm gonna ignore the game consoles statement because that's a specialized set of CPUs which will never land in any laptop anytime soon. Most of them are AMD, so I'd refer to the same articles anyways.
Limited platforms? Code exploits?
These are a perfectly good way to back up the first half of your original statement, "No. At least not reliably"
But "due to how the keys are stored in RAM" is not true. And that's the part that got objected to.
You're not being nitpicked on a technicality here. You mixed up the techs you were talking about in a way that makes a significant difference. Bringing in all these references might be useful to people, but you could have just said "oh, I meant disk encryption by that part" instead of getting sassy and doubling down on saying incorrect things with "Did you read any of my referenced links? You cannot state this when the paper and well known attack methods disprove your statement."
These are a perfectly good way to back up the first half of your original statement, "No. At least not reliably"
But "due to how the keys are stored in RAM" is not true. And that's the part that got objected to.
You're not being nitpicked on a technicality here. You mixed up the techs you were talking about in a way that makes a significant difference. Bringing in all these references might be useful to people, but you could have just said "oh, I meant disk encryption by that part" instead of getting sassy and doubling down on saying incorrect things with "Did you read any of my referenced links? You cannot state this when the paper and well known attack methods disprove your statement."
In my first comment I literally mentioned BitLocker, which, as I pointed out correctly, stores its keys either in RAM or on the TPM chip.
If you gonna rip apart my comment into parts without context, at least do it with all parts.
Note that I didn't start the discussion about where the key is stored, someone replied to my initial comment - in the context of BitLocker - saying that the CPU stores the key, which is incorrect.
My initial point was not about how secure enclaves work in practice. Though I argued that most implemented follow-up technologies don't work reliably due to how the OS always interfaces with unpatchable firmware bugs.
While I agree this is the only way to fix the coldboot attack problem I don't see the technology as reliable as of today, and I don't understand why you think I am wrong with that statement?
If you don't agree: show papers with counter measures, and how to prevent the exploits and bypasses.
If you gonna rip apart my comment into parts without context, at least do it with all parts.
Note that I didn't start the discussion about where the key is stored, someone replied to my initial comment - in the context of BitLocker - saying that the CPU stores the key, which is incorrect.
My initial point was not about how secure enclaves work in practice. Though I argued that most implemented follow-up technologies don't work reliably due to how the OS always interfaces with unpatchable firmware bugs.
While I agree this is the only way to fix the coldboot attack problem I don't see the technology as reliable as of today, and I don't understand why you think I am wrong with that statement?
If you don't agree: show papers with counter measures, and how to prevent the exploits and bypasses.
> In my first comment I literally mentioned BitLocker,
You did mention bitlocker. But it was unrelated to RAM encryption, and you acted like it was related.
Everything you said about bitlocker was true, but the way you brought up bitlocker was itself the problem.
> someone replied to my initial comment - in the context of BitLocker
No they did not. Your answer did not make a new separate context for just bitlocker. You muddled bitlocker and RAM encryption. They replied in the original context, because you said that RAM encryption stores the key in RAM during standby.
Whether you intended to say that or not, it's what you said. Go back and read the original comments. You directly said that line about RAM encryption, not just disk encryption.
"> > As a naive outsider: Does RAM in 2023 encrypt all its contents?"
"> No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed)."
> I don't see the technology as reliable as of today, and I don't understand why you think I am wrong with that statement?
I never said you were wrong about that.
You did mention bitlocker. But it was unrelated to RAM encryption, and you acted like it was related.
Everything you said about bitlocker was true, but the way you brought up bitlocker was itself the problem.
> someone replied to my initial comment - in the context of BitLocker
No they did not. Your answer did not make a new separate context for just bitlocker. You muddled bitlocker and RAM encryption. They replied in the original context, because you said that RAM encryption stores the key in RAM during standby.
Whether you intended to say that or not, it's what you said. Go back and read the original comments. You directly said that line about RAM encryption, not just disk encryption.
"> > As a naive outsider: Does RAM in 2023 encrypt all its contents?"
"> No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed)."
> I don't see the technology as reliable as of today, and I don't understand why you think I am wrong with that statement?
I never said you were wrong about that.
I was reading my comments again, and I can understand how someone might interpret them differently.
Granted, I should have formulated it probably differently and specifically differed between secure enclaves, memory encryption and full disk encryption problems to be more clear about it.
Granted, I should have formulated it probably differently and specifically differed between secure enclaves, memory encryption and full disk encryption problems to be more clear about it.
Bitlocker isn't RAM encryption, so I don't see how that proves your point. Those are attack methods for typical disk encryption when no RAM encryption is involved.
Any non-ridiculous RAM encryption will keep the key in the CPU. And since the key only needs to last for one boot, it never needs to leave the chip. If there's hardware in the memory controller doing the encryption it can be designed so that key export is impossible.
Any non-ridiculous RAM encryption will keep the key in the CPU. And since the key only needs to last for one boot, it never needs to leave the chip. If there's hardware in the memory controller doing the encryption it can be designed so that key export is impossible.
See my sibling comment.
Isn't that what Intel TME does?
That does look like it:
https://www.intel.com/content/www/us/en/developer/articles/n...
Looks like AMD has an equivalent:
https://www.tomshardware.com/news/intel-mktme-amd-memory-enc...
I wish there were a sort of checklist that prints out in dmesg saying "Your RAM is encrypted: (check) Your machine has these mitigations against cold-boot attacks: ...some list here... (check, check, check)"
In some contexts, I love how far we've come with secure boot, attestation, speculative execution mitigations, etc. but it's very difficult for the average Linux user to understand what they should expect for hardware security, and then work toward enabling all of it.
I love what Gnome did in the last year in Settings -> Device Security:
https://www.phoronix.com/news/Ubuntu-No-GNOME-Device-Securit...
There's a detailed list here of what it's looking for:
https://forums.debian.net/viewtopic.php?t=152705
Not a fan of all the bound-to-Intel technologies. I wish the concept of what's being secured were identified, then made available in dmesg or through some /sys or /proc interface.
https://www.intel.com/content/www/us/en/developer/articles/n...
Looks like AMD has an equivalent:
https://www.tomshardware.com/news/intel-mktme-amd-memory-enc...
I wish there were a sort of checklist that prints out in dmesg saying "Your RAM is encrypted: (check) Your machine has these mitigations against cold-boot attacks: ...some list here... (check, check, check)"
In some contexts, I love how far we've come with secure boot, attestation, speculative execution mitigations, etc. but it's very difficult for the average Linux user to understand what they should expect for hardware security, and then work toward enabling all of it.
I love what Gnome did in the last year in Settings -> Device Security:
https://www.phoronix.com/news/Ubuntu-No-GNOME-Device-Securit...
There's a detailed list here of what it's looking for:
https://forums.debian.net/viewtopic.php?t=152705
Not a fan of all the bound-to-Intel technologies. I wish the concept of what's being secured were identified, then made available in dmesg or through some /sys or /proc interface.
AMD Epyc servers have offered encrypted memory for many years.
Intel is only now catching up.
Intel is only now catching up.
I have an AMD CPU and I get in my kernel logs during boot:
[ 0.338594] Memory Encryption Features active: AMD SMEI mean like a checklist of security capabilities that are enabled/disabled, all grouped together in dmesg. That is cool though, it does help
I contacted Mullvad, who confirmed that they only use bare metal - no virtualization or containerization whatsoever. https://ibb.co/XLDQGGt
I use Cloudflare's DoH DNS (https://1.1.1.1) and tunnel that over Mullvad's VPN.
That way Mullvad can only see the IPs I visit, Cloudflare only see DNS requests coming from the VPN IP, and my ISP sees nothing.
Mullvad's DoH DNS seems nice, but it provides potentially less privacy than the above, so I won't be using it.
That way Mullvad can only see the IPs I visit, Cloudflare only see DNS requests coming from the VPN IP, and my ISP sees nothing.
Mullvad's DoH DNS seems nice, but it provides potentially less privacy than the above, so I won't be using it.
>That way Mullvad can only see the IPs I visit,
I hate to break it to you, but if they wanted to look at the SNI on all your TLS connections, they could; ECH is not really a thing yet.
I hate to break it to you, but if they wanted to look at the SNI on all your TLS connections, they could; ECH is not really a thing yet.
Yes you're correct if Mullvad is performing packet inspection then there isn't much I can do about that.
My point stands though about not putting all eggs in the one basket and using a different DNS provider than that of the VPN provider.
My point stands though about not putting all eggs in the one basket and using a different DNS provider than that of the VPN provider.
No, I don't think you understand. "That way Mullvad can only see the IPs I visit" is your point and it _doesn't_ stand. There isn't really multiple ways of looking at it; the point you were making is exactly the point that was refuted, and all this "my point stands" is the weakest shit in the whole white world— a compulsion, almost inertial in nature, where by attempting to save face we present ourselves clueless. Now, all this "putting all eggs in the one basket" and common sense knowledge would be very adorable had you not been wrong, and had it not been a bad advice leaving you at a disadvantage. Good luck
I don't think it's recommended to use another DNS with Mullvad because it makes you more unique.
I would like to see real statistics but my gut feeling from running read/write intensive data applications on SSD and ECC RAM is that both of them fail often enough that this move is somewhat lateral in terms of resiliency.
But in terms of clawing raw performance decimals, I applaud the effort. This would be a fun redis project.
But in terms of clawing raw performance decimals, I applaud the effort. This would be a fun redis project.
That's not the point - they promise to keep no logs, and the best way to demonstrate that they don't is to not even have any fixed storage in the servers at all (they presumably netboot from a read-only image). So they have done this with their VPN servers, now they are rolling out the strategy to DNS servers as well.
That's fine but as long as it is connected to a network you cannot prove a system to be log-free.
The majority of systems I've worked on used a message queue of some kind for logging rather than a file on disk.
I suppose it could be useful to expedite downtime in the case that they are subpoenaed, but I doubt claiming to be diskless would prevent any subpoena.
The majority of systems I've worked on used a message queue of some kind for logging rather than a file on disk.
I suppose it could be useful to expedite downtime in the case that they are subpoenaed, but I doubt claiming to be diskless would prevent any subpoena.
With this kind of security, you're more guarding against your servers being seized or intelligence agency installing an implant.
> I doubt claiming to be diskless would prevent any subpoena
It does in Sweden. They've repeatedly squashed subpoenas
> long as it is connected to a network you cannot prove a system to be log-free.
Nothing is perfect. If you read their audits, it's clear admins could get at your traffic if they wanted
> I doubt claiming to be diskless would prevent any subpoena
It does in Sweden. They've repeatedly squashed subpoenas
> long as it is connected to a network you cannot prove a system to be log-free.
Nothing is perfect. If you read their audits, it's clear admins could get at your traffic if they wanted
This is why they have their architecture audited (and release the audit reports).
https://mullvad.net/en/blog/2022/6/22/vpn-server-audit-found...
https://mullvad.net/en/blog/2022/6/22/vpn-server-audit-found...
>but I doubt claiming to be diskless would prevent any subpoena.
It has already.
See https://news.ycombinator.com/item?id=38220769
It has already.
See https://news.ycombinator.com/item?id=38220769
Logging doesn't require disks, the logs can be sent over the network.
You've seen ECC fail often enough? Ours has been incredibly reliable, but we don't have enormous bandwidth. do you have a data sheet or anything showing failure rates?
I was extremely happy with everything mullvad was doing until they announced they were discontinuing port forwarding, which is important to a niche part of my setup. I understand the reasoning, but if it came back ever I would again happily be a customer.
VPN port forwarding is basically a “botnet operators welcome” sign from what I understand. The benefits of disabling it are greater than the downsides for Mullvad.
Does anyone know how Mullvad vpn security compares with Proton vpn?
Mullvad is pretty much the #1 when you want privacy. They have a history of fighting away law enforcement requests, because they really don't have anything to give to them.
Everything runs on RAM and will disappear the second a server is shutdown, they really don't store anything.
And you can buy a Mullvad account by mailing cash to them or with crypto. No need to create an account with an email address or anything.
Everything runs on RAM and will disappear the second a server is shutdown, they really don't store anything.
And you can buy a Mullvad account by mailing cash to them or with crypto. No need to create an account with an email address or anything.
"In April 2019, upon the order of the Swiss judiciary in a case of clear criminal conduct, we enabled IP logging against a specific user account which is engaged in illegal activities which contravene Swiss law. Pursuant to Swiss law, the user in question will also be notified and afforded the opportunity to defend against this in court before the data can be used in criminal proceedings"
A quote with no context or attribution always makes it fun to figure out what someone intends to say with their comment.
This quote is in reference to ProtonMail in 2019, a fairly widely reported on case at the time.
This would be a point in favor of Mullvad, as Mullvad has demonstrably squashed a subpoena in the past, thanks to their data-minimization policies (no PII for signup, no PII options for payment, diskless infrastructure, etc.).
Note that this is more related to privacy, and not as much security. If the parent comment was specifically talking about security, I would recommend reviewing the software and infrastructure audits made available by either company. I'm not aware of any major fault with either, from a security perspective.
This quote is in reference to ProtonMail in 2019, a fairly widely reported on case at the time.
This would be a point in favor of Mullvad, as Mullvad has demonstrably squashed a subpoena in the past, thanks to their data-minimization policies (no PII for signup, no PII options for payment, diskless infrastructure, etc.).
Note that this is more related to privacy, and not as much security. If the parent comment was specifically talking about security, I would recommend reviewing the software and infrastructure audits made available by either company. I'm not aware of any major fault with either, from a security perspective.
Note that this applies to Proton Mail.
Proton Mail is considered to be a communication service, and in most countries (including Switzerland), communication services are regulated to some extent. All companies need to comply with the local law.
That being said, Swiss law is very restrictive, and there are a lot of hurdles that one needs to jump through to get a court order. And even with a court order (and has been proved multiple times in court), there is no way to break Proton Mail's encryption. Privacy is not the same as anonymity, and due to the way the internet works, if anonymity is what you are going after, you have to exercise proper infosec and take preventive measures, such as using Tor or VPN.
Under Swiss law, the treatment of VPNs is different, and VPNs can indeed be no-logs, like Proton VPN is. No-logs VPN is also possible in other countries as well. What makes Switzerland different, and possibly unique, is that within the current Swiss legal framework, Proton VPN also does not have forced logging obligations. You can find out more here: https://protonvpn.com/blog/transparency-report/.
That being said, Swiss law is very restrictive, and there are a lot of hurdles that one needs to jump through to get a court order. And even with a court order (and has been proved multiple times in court), there is no way to break Proton Mail's encryption. Privacy is not the same as anonymity, and due to the way the internet works, if anonymity is what you are going after, you have to exercise proper infosec and take preventive measures, such as using Tor or VPN.
Under Swiss law, the treatment of VPNs is different, and VPNs can indeed be no-logs, like Proton VPN is. No-logs VPN is also possible in other countries as well. What makes Switzerland different, and possibly unique, is that within the current Swiss legal framework, Proton VPN also does not have forced logging obligations. You can find out more here: https://protonvpn.com/blog/transparency-report/.
Who is this in reference to? What is the source? Googling for the quote you gave doesn't have great results.
See my comment above, but it is in reference to ProtonMail.
You can find many articles about it (search "Protonmail subpoena 2019"), but an example article is here: https://www.privacyaffairs.com/protonmail-surrenders-user-lo...
ProtonMail's own blog post about it is here: https://proton.me/blog/climate-activist-arrest
You can find many articles about it (search "Protonmail subpoena 2019"), but an example article is here: https://www.privacyaffairs.com/protonmail-surrenders-user-lo...
ProtonMail's own blog post about it is here: https://proton.me/blog/climate-activist-arrest
It's from Protonmail's website https://web.archive.org/web/20190425155330/https://protonmai...
vpn’s don’t provide security. The only thing they do is circumvent geoblocks and maybe prevent dmca notices when downloading pirated content.
VPN provider OVPN (ovpn.com) has been doing this as well.
I don't understand this. Don't all Linux processes "run in RAM"? It sounds like what they have actually accomplished here is eliminate all disk I/O requirements from their DNS service, i.e. it works against a read-only mounted root filesystem now. That's not really the same as "running in RAM".
This is not what's happening here. The systems don't have a disk, period. The bootloader downloads an OS image from a provisioning server into RAM that's verified and then run. Past the bootloader, the system doesn't have access to the provisioning server. Every program the server needs or may need to run at one point is already in memory. Outside of RAM there is nowhere to read more data from, not even a read-only disk, nor anywhere to write it to.
If one is already running only "verified" (attested) code (with a read-only partition), diskless doesn't buy you much? Android, for example, does this: The system partition where the OS is remains immutable unless there's a signed update (or the bootloader is "unlocked").
They briefly explain some of the reasoning here: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
>If the computer is powered off, moved or confiscated, there is no data to retrieve.
>We get the operational benefits of having fewer breakable parts. Disks are among the components that break often. Therefore, switching away from them makes our infrastructure more reliable.
>The operational tasks of setting up and upgrading package versions on servers become faster and easier.
>Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.
>If the computer is powered off, moved or confiscated, there is no data to retrieve.
>We get the operational benefits of having fewer breakable parts. Disks are among the components that break often. Therefore, switching away from them makes our infrastructure more reliable.
>The operational tasks of setting up and upgrading package versions on servers become faster and easier.
>Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.
Yeah, I don't have their opsec concerns but I'm very tempted to try implementing something similar just to enforce having a stateless system (easier to manage IMO) with a nice bonus of no longer caring about host disk failures.
In such a setup where does the key to verify the image come from?
More info about stboot is available here: https://www.system-transparency.org/
and here:https://git.glasklar.is/system-transparency/core/system-tran...
The second link talks about network boot mode and signature validation.
and here:https://git.glasklar.is/system-transparency/core/system-tran...
The second link talks about network boot mode and signature validation.
This seems like a very roundabout and unnecessary solution to a problem that is already solved with tmpfs.
Once they have everything running in RAM, they can have racks of servers that contain no persistent storage whatsoever, and are therefore subpoena-proof.
> On April 18 [2023] at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant. They intended to seize computers with customer data.
> In line with our policies such customer data did not exist. We argued they had no reason to expect to find what they were looking for and any seizures would therefore be illegal under Swedish law. After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.
https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
> On April 18 [2023] at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant. They intended to seize computers with customer data.
> In line with our policies such customer data did not exist. We argued they had no reason to expect to find what they were looking for and any seizures would therefore be illegal under Swedish law. After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.
https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
> contain no persistent storage whatsoever, and are therefore subpoena-proof.
Nonsense. The linked post shows that they didn't have the sought information. If the information was present in RAM, they'd just read it from RAM.
Nonsense. The linked post shows that they didn't have the sought information. If the information was present in RAM, they'd just read it from RAM.
I assume they only hold what the server is doing in any given moment in ram and it is overwritten as new requests come in. By the time the police showed up with the subpoena the data they wanted was long gone.
Please tell me how you would go about reading something that happened two weeks ago from RAM?
>In early 2022 we announced the beginning of our migration to using diskless infrastructure with our bootloader known as “stboot”.
>We recently announced the completion of our migration to remove all traces of disks in use on our VPN infrastructure.
Is this more clear?
>We recently announced the completion of our migration to remove all traces of disks in use on our VPN infrastructure.
Is this more clear?
they should at least opensource their implementation. they may have errors there. should we just trust everything?
Why the entitled attitude? They do a lot of open source contributions already so maybe just give them some time instead of immediately disregarding the whole improvement? Small steps into the right direction.
https://mullvad.net/en/help/open-source
https://mullvad.net/en/help/open-source
Making a suggestion isn't entitlement. Maybe reconsider the foul void that accusation came from.
If someone suggests something to me they usually don't say "you should at least". This sounded more like an entitled comment you see every day on open source repositories and making it sometimes not fun to be an open source maintainer.
They are just using wireguard.
You can connect with wireguard and not run any mullvad software on the client side.
Though that's generally the case with VPN providers, whether openvpn, wireguard, etc.
I guess they could introduce some vulnerability on the server side if they rolled their own wireguard implementation or patched it. Though it seems unlikely to me.
https://mullvad.net/en/help/easy-wireguard-mullvad-setup-lin...
You can connect with wireguard and not run any mullvad software on the client side.
Though that's generally the case with VPN providers, whether openvpn, wireguard, etc.
I guess they could introduce some vulnerability on the server side if they rolled their own wireguard implementation or patched it. Though it seems unlikely to me.
https://mullvad.net/en/help/easy-wireguard-mullvad-setup-lin...
Pretty sure they have been audited recently.
If I were a Mullvad customer this is what I would want: regular audits and closed source until everyone was convinced the code was bulletproof.
Their infrastructure kind of is their USP. It'd be like giving their competition their work for free. And it's not about absolute trust, it's about trusting them more than your ISP and other VPN providers. You still have TLS over Mullvad
I'd argue that Mullvad's USP/moat is their reputation, far more than the software they run.
They have open sourced a good deal of the infrastructure:
https://github.com/mullvad/system-transparency
https://github.com/mullvad/system-transparency
> Today we can announce more steps forward - our Encrypted DNS service has also been converted to run from RAM!
Where exactly do services run from if not RAM? The post is very plain with no explanation of what "running from RAM" means. When you execute any binary it gets loaded into RAM as per standard operating system procedures...?
Where exactly do services run from if not RAM? The post is very plain with no explanation of what "running from RAM" means. When you execute any binary it gets loaded into RAM as per standard operating system procedures...?
They mean diskless. There's no persistent storage attached to the machines that run the DNS servers, because the entire operating system is loaded into RAM directly from PXE boot. So, if the hardware ever gets seized by law enforcement or rogue actors, there is no risk of exfiltration of private user data.
[deleted]
[deleted]
In addition they also support optional content blocking[1] via blocklists, just set the desired TLS/HTTPS DNS server.
[1] https://github.com/mullvad/dns-blocklists