ARM Mac: Why I'm Worried About Virtualization(bmalehorn.com)
bmalehorn.com
ARM Mac: Why I'm Worried About Virtualization
https://bmalehorn.com/arm-mac/
305 comments
It is very likely that ARM-based Macs will lack a performant hypervisor upon release. We will have to see how VMWare responds. I'd bet it will inspire new products and innovation and the desktop space will move towards a less x86-x64 centric world. In the end it is a short term problem. Someone will respond and provide a performant hypervisor that can run on an ARM host and virtualize x86-x64 and ARM guests.
It's true it will cause some pain in the first year or two, but even as a heavy VMWare Fusion user I am really looking forward to the benefits of a vertically integrated laptop.
It's true it will cause some pain in the first year or two, but even as a heavy VMWare Fusion user I am really looking forward to the benefits of a vertically integrated laptop.
I'd say that I'm excited for ARM. That doesn't mean the transition will be seamless or easy.
I know that a big complaint about the move is "great, now I'm doing ARM locally and deploying to x86". I think this is a legitimate concern, for now, but I also strongly believe it is inevitable that, within the next decade, deploying to x86 in the Cloud will be as "weird" as ARM would be today. The benefits are way too numerous.
Well, more accurately, I think it'll be a "I'm on Fargate, oh wow, Fargate runs on ARM, I had no idea" kind of thing. Ok, the article outlines why you may need some idea, but come on; we're talking about one line where I'm downloading the x86 version of a dependency instead of an ARM version. That's an easy fix.
I don't know what this means for open accessibility of hardware. Right now, I could go buy and run locally the Intel Xeon chip powering my app in the cloud; when things move to ARM, it absolutely will be "AWS Graviton" (not sold outside AWS) or "Azure ARM Whatever" (not sold outside Azure). This sucks for accessibility, but, actually, does it? ARM enables the cloud providers to do this; they could never design their own x86 chips. As long as we're all standardized on the same ISA, and the chips generally have the same characteristics, I'm looking forward to a very bright future where vendors are now also competing against one-another in the silicon. And I may not be able to buy an AWS Graviton, but I'm sure (well, hopeful) that one day I'll be able to build an ARM desktop that isn't a Raspberry Pi. AWS will have their chips, Quallcomm has theirs, Apple has theirs, Microsoft and Google have some, and they're all competing against one another.
Ok, maybe this is a pipe dream. But, I'm definitely in the short Intel camp, at least on the long-term.
I know that a big complaint about the move is "great, now I'm doing ARM locally and deploying to x86". I think this is a legitimate concern, for now, but I also strongly believe it is inevitable that, within the next decade, deploying to x86 in the Cloud will be as "weird" as ARM would be today. The benefits are way too numerous.
Well, more accurately, I think it'll be a "I'm on Fargate, oh wow, Fargate runs on ARM, I had no idea" kind of thing. Ok, the article outlines why you may need some idea, but come on; we're talking about one line where I'm downloading the x86 version of a dependency instead of an ARM version. That's an easy fix.
I don't know what this means for open accessibility of hardware. Right now, I could go buy and run locally the Intel Xeon chip powering my app in the cloud; when things move to ARM, it absolutely will be "AWS Graviton" (not sold outside AWS) or "Azure ARM Whatever" (not sold outside Azure). This sucks for accessibility, but, actually, does it? ARM enables the cloud providers to do this; they could never design their own x86 chips. As long as we're all standardized on the same ISA, and the chips generally have the same characteristics, I'm looking forward to a very bright future where vendors are now also competing against one-another in the silicon. And I may not be able to buy an AWS Graviton, but I'm sure (well, hopeful) that one day I'll be able to build an ARM desktop that isn't a Raspberry Pi. AWS will have their chips, Quallcomm has theirs, Apple has theirs, Microsoft and Google have some, and they're all competing against one another.
Ok, maybe this is a pipe dream. But, I'm definitely in the short Intel camp, at least on the long-term.
This seems like a weird benchmark, reading from /dev/urandom and gzipping random data does not seem like something most folks will want to do. It even appears like /dev/urandom speeds differ greatly on various architectures [0] and there are issues with /dev/random being fundamentally slow due to the entropy pool [1] (but I guess this is why the author uses /dev/urandom).
It would be better to measure something more related to what docker users will actually do, like container build time of a common container, and/or latency of HTTP requests to native/emulated containers running on the some container.
One reason to feel positive about the virtualization issues is that Rosetta 2 provides x86->ARM translation for JITs which an ARM-based QEMU could perhaps integrate into it's own binary translation [2].
[0] https://ianix.com/pub/comparing-dev-random-speed-linux-bsd.h... [1] https://superuser.com/questions/359599/why-is-my-dev-random-... [2] https://developer.apple.com/videos/play/wwdc2020/10686/
It would be better to measure something more related to what docker users will actually do, like container build time of a common container, and/or latency of HTTP requests to native/emulated containers running on the some container.
One reason to feel positive about the virtualization issues is that Rosetta 2 provides x86->ARM translation for JITs which an ARM-based QEMU could perhaps integrate into it's own binary translation [2].
[0] https://ianix.com/pub/comparing-dev-random-speed-linux-bsd.h... [1] https://superuser.com/questions/359599/why-is-my-dev-random-... [2] https://developer.apple.com/videos/play/wwdc2020/10686/
It didn’t take months, the time I did it (running Docker on a Pinebook, which was not a great experience). It took a couple of hours to flip some base images away from Alpine, as Debian already has a load of ARM packages built.
So many wrong assumptions ...
1. If emulating aarch64 (arm64) on x86_64 is 6x slower (on your system, btw, it's not an universal constant), it doesn't mean emulating x86_64 on aarch64 will be 6x slower. It'd probably be worse, or at least that's my gut feeling.
2. Generic container images like the Ubuntu mentioned usually have aarch64 (arm64) support, so running the x86_64 image makes no sense for the presented use-case.
3. You won't be able to use most software because they don't release ARM binaries ... and the example uses `wget` && `tar xf`, with no binary signature check. As someone who has been porting stuff from x86_64 to aarch64 for a couple of years, I admit I've seen this pattern frequently. The most obvious solution is to build from sources, which would have been better off on x86_64 too, instead of fetching a prebuilt (and unverified) binary from the internet. Maybe there are some CPU flags the compiler could notice and apply optimizations which are not included in the prebuilt binary.
I'm not an Apple fan and I'm certainly not a fan of cross-architecture development either. I do agree with the general idea behind the article, however I find it a bit hand wavy.
1. If emulating aarch64 (arm64) on x86_64 is 6x slower (on your system, btw, it's not an universal constant), it doesn't mean emulating x86_64 on aarch64 will be 6x slower. It'd probably be worse, or at least that's my gut feeling.
2. Generic container images like the Ubuntu mentioned usually have aarch64 (arm64) support, so running the x86_64 image makes no sense for the presented use-case.
3. You won't be able to use most software because they don't release ARM binaries ... and the example uses `wget` && `tar xf`, with no binary signature check. As someone who has been porting stuff from x86_64 to aarch64 for a couple of years, I admit I've seen this pattern frequently. The most obvious solution is to build from sources, which would have been better off on x86_64 too, instead of fetching a prebuilt (and unverified) binary from the internet. Maybe there are some CPU flags the compiler could notice and apply optimizations which are not included in the prebuilt binary.
I'm not an Apple fan and I'm certainly not a fan of cross-architecture development either. I do agree with the general idea behind the article, however I find it a bit hand wavy.
Are there any excited embedded developers in the crowd? I have done a little embedded work and cross compiling has always been a huge pain in the ass to setup. I know some people have even gone as far as purchasing expensive niche workstations with ARM CPUs specifically to avoid this problem. I feel like having a mainstream ARM platform like the MBP will make compiling software for ARM-based single board computers a breeze.
My 2017 Macbook pro still has quite a lot of life in it, but it seems unlikely I will replace it with another Mac in a few years. Before the Mac I had a Lenovo X1 Carbon running Linux and it was great; and even then was a better development environment in some ways (docker has better filesystem performance, pacman is much better than homebrew). I do use some audio processing applications and my kids play a few games that do not run at all on Linux. I may try WSL again instead of going straight back to Linux, but its hard to imagine Mac will be the best OS for me.
If the worst of every possible thing happens and you avoid the most obvious solutions and one is very, very slow, then yes, you're right to worry.
Or, you could use already-extant Debian ARM releases and spend minutes rather than months switching over.
Or, you could use already-extant Debian ARM releases and spend minutes rather than months switching over.
I'd like to advocate for remote development environments. Most of my day is spent typing into a tmux session on a cloud-hosted box. (I picked up a Magic Keyboard for my 11" iPad Pro, and thanks to Blink it's a great glass terminal. It's not going to work if you're debugging let's say a React app, but I've been very happy on it the last several days churning out Golang.)
Running stuff on your laptop makes it run slow, get hot, and burn battery. I've considered getting a small x86 or ARM media appliance as a (physically local) remote server for when I can't count on an Internet connection. A media PC costs how much? The big holdup has been the tyranny of choice I'm confronted with. (Suggestions are welcome!)
I think very few people would be surprised if the coming of ARM Macs will, along with AWS's ARM moves (and Microsoft's), drive acceptance and adoption of ARM-based server computing. The mechanism won't be anything formal, just the vague pressure that comes from people wanting their programs and libraries to compile locally.
Running stuff on your laptop makes it run slow, get hot, and burn battery. I've considered getting a small x86 or ARM media appliance as a (physically local) remote server for when I can't count on an Internet connection. A media PC costs how much? The big holdup has been the tyranny of choice I'm confronted with. (Suggestions are welcome!)
I think very few people would be surprised if the coming of ARM Macs will, along with AWS's ARM moves (and Microsoft's), drive acceptance and adoption of ARM-based server computing. The mechanism won't be anything formal, just the vague pressure that comes from people wanting their programs and libraries to compile locally.
I would expect about a 5x slowdown running Docker images.
Docker on a Mac utilizes a hypervisor. Hypervisors rely on running the same architecture on the host as the guest, and are about about 1x - 2x as slow as running natively.
Since you're running ARM Mac, these hypervisors can only run ARM Linux. They can't run x86_64 Linux.
What will happen instead? These tools will fall back on emulators.
The loss of Boot Camp is huge. One of the reasons I develop on a Mac is so that I can use a single machine for all development (including macOS, iOS, and Windows development). Most of the time, developing for Windows on Parallels works fine, but there are some cases where it's necessary to boot directly into Windows to test or debug adequately. I hope Apple is able to reach an agreement with Microsoft or at least continues shipping Intel-based Macs until then.
Apple will singlehandedly make 2021 "the year of Linux on the desktop".
This is ridiculous. Your code will just have to support multiple architectures, which is very easy with modern languages and tooling.
I remember the days when having to switch between x86, ppc, sparc, etc was a thing (not to mention the many flavors of UN*X) and we survived. In fact I think it was more fun back before the x86/Linux server domination. Architecture diversity is good.
I'm more convinced dropping dual boot and supporting virtualization is the right move.
Only the host OS is going to have the right drivers for the trackpad, wi-fi, GPU, power management, etc. etc. Through virtualization, the guest OS doesn't have to worry about constantly evolving hardware models.
Virtualized OS performance is already very good, and USB passthrough has existed for a while. Snapshots are a godsend.
What won't work are things like CUDA for eGPUs over Thunderbolt 3, and you'll have to share disk and RAM with the host OS.
But for most use cases it's probably the right choice. (This doesn't address the author's concern about moving away from x86.)
Only the host OS is going to have the right drivers for the trackpad, wi-fi, GPU, power management, etc. etc. Through virtualization, the guest OS doesn't have to worry about constantly evolving hardware models.
Virtualized OS performance is already very good, and USB passthrough has existed for a while. Snapshots are a godsend.
What won't work are things like CUDA for eGPUs over Thunderbolt 3, and you'll have to share disk and RAM with the host OS.
But for most use cases it's probably the right choice. (This doesn't address the author's concern about moving away from x86.)
> Why can't you update the Docker image to also support ARM? You theoretically could switch your backend to run ARM Linux. However, this would take months - renting out ARM instances, re-building all repositories, and a tense switch over.
I don't see why this would be so hard. If anything, I expect to see a massive upswing in things like AWS Graviton2 uptake, and a lot of common Docker images being built with ARM versions out of the box. It might be about a year or so, but eventually we'll be able to just go ARM-native the whole way.
What Apple needs to do is make a first-class, WSL-tier implementation of Docker for Mac for ARM.
I don't see why this would be so hard. If anything, I expect to see a massive upswing in things like AWS Graviton2 uptake, and a lot of common Docker images being built with ARM versions out of the box. It might be about a year or so, but eventually we'll be able to just go ARM-native the whole way.
What Apple needs to do is make a first-class, WSL-tier implementation of Docker for Mac for ARM.
For my most recent project[1], I wanted to see if Amazon’s Graviton instances would be a good choice for my docker deployments (I was deploying MongoDB, an Express server, and several instances of the Janus WebRTC server). I was developing in Pop OS on an x86_64 desktop (since we’re gonna have to start specifying now) and found the toolchain around building ARM64 images to be pretty simple once I got it set up.
I benchmarked some `t2a.nano`s against some `a1.medium`s and found that the `nano`s were sufficient for my needs, so I went with them (they are cheaper than `a1.medium`s, even if the `a1`s have a better price-to-performance ratio).
I didn’t find it too difficult to rebuild any of these projects for cross-architecture usage. Even Janus, which has a TON of C/C++ dependencies (some of which have to be compiled from a particular version of the source) easily built for ARM with no change in the Dockerfile.
So I kind of feel like OP is exaggerating the effort required to migrate servers to ARM. Sure it might be a hassle when you have tons of microservices, but you can move them incrementally, and most things recompile with no changes. And regardless of what architecture your dev machine is, you’ll want to be able to compile for and work with both architectures if you want to get the most out of the infrastructure on offer in 2020.
[1] Shameless plug: https://chrisuehlinger.com/blog/2020/06/16/unshattering-the-...
I benchmarked some `t2a.nano`s against some `a1.medium`s and found that the `nano`s were sufficient for my needs, so I went with them (they are cheaper than `a1.medium`s, even if the `a1`s have a better price-to-performance ratio).
I didn’t find it too difficult to rebuild any of these projects for cross-architecture usage. Even Janus, which has a TON of C/C++ dependencies (some of which have to be compiled from a particular version of the source) easily built for ARM with no change in the Dockerfile.
So I kind of feel like OP is exaggerating the effort required to migrate servers to ARM. Sure it might be a hassle when you have tons of microservices, but you can move them incrementally, and most things recompile with no changes. And regardless of what architecture your dev machine is, you’ll want to be able to compile for and work with both architectures if you want to get the most out of the infrastructure on offer in 2020.
[1] Shameless plug: https://chrisuehlinger.com/blog/2020/06/16/unshattering-the-...
As a Linux tech, I welcome this Apple move honestly.
Having a proper competitor for x86/x64 is a good thing.
The fact docker is slower on ARM (at the moment!) is mostly due to the lack of interests for optimizations.
With Apple starting to produce MacARM machines, and maybe more ARM servers in the wild, docker (and other platforms/frameworks) will start to get more performant on ARM as well.
Having a proper competitor for x86/x64 is a good thing.
The fact docker is slower on ARM (at the moment!) is mostly due to the lack of interests for optimizations.
With Apple starting to produce MacARM machines, and maybe more ARM servers in the wild, docker (and other platforms/frameworks) will start to get more performant on ARM as well.
Do we know that Boot Camp isn’t supported by Big Sur? Or is it just that one can’t run an x86 OS on an ARM architecture? In other words - will I still be able to dual-boot into something like ARM-flavored Linux?
If you stay back on x86 virtualization will be slow, but if you jump to ARM this is great!
> However, this would take months
(infomercial arms)
I'm sure it won't make sense for everyone, but I'm just as sure it will make sense for many.
> However, this would take months
(infomercial arms)
I'm sure it won't make sense for everyone, but I'm just as sure it will make sense for many.
On the flip side, I guess ARM Macs will now allow the use of hypervisors for Android simulators, instead of a full hardware virtualization.
... thus making Android development better on Macs?
... thus making Android development better on Macs?
> ec2 only offers 6 general-purpose ARM instance sizes
m6g, c6g, r6g each support 6 sizes for a total of 24
m6g, c6g, r6g each support 6 sizes for a total of 24
If you hadn't sold yourself out of the free market, you would be able to choose what architecture machine you bought.
> Docker on a Mac utilizes a hypervisor. Hypervisors rely on running the same architecture on the host as the guest, and are about about 1x - 2x as slow as running natively.
That doesnt sound right to me. Perhaps on IO bound tasks, if you are using emulated devices. On CPU bound tasks you should see near native performance.
That doesnt sound right to me. Perhaps on IO bound tasks, if you are using emulated devices. On CPU bound tasks you should see near native performance.
There's another unintended consequence of this virtualization -
docker is already has very high CPU usage on my macbook, anywhere from 50-100%. Because of which it is always hot and toasty. This is caused already caused my screen to start deteriorating (https://www.ifixit.com/Answers/View/567125/Horizontal+line+o...) and the battery has degraded considerably too, even when I'm not coding on it and docker is shut down. This means a significant hit to the longevity of such devices as they not meant to be pushed so hard 40 hours a week. With ARM macs I see it getting even worse.
On the bright side, it looks like low-latency streaming is good enough that as long as you have internet connection, Boot Camp is not really necessary. This works well for me: https://shadow.tech/usen/
A lot of developer-centric focus discussion on how Docker would work (hint: it does), but VirtualBox is still pretty common in the sysadmin world and other industry circles. Moreover, there seems to be no way it will ever work. It will be interesting to see how that turns out.
Any company with a load of binaries built without any effort towards supporting cross platform builds that uses Docker is gonna have a bad day with this. They buy a bunch of new MacBooks and then find they can't use them until they spend a few weeks porting everything.
My guess is that Apple will end up copying Microsoft and providing a WSL style Linux kernel "shim" into Darwin (pretty easy as it's already UNIX) and use Rosetta2 to translate any x86_64 containers to aarch64). No need for a hypervisor.
- I already do cross-arch development day-in and day out between x86 and ARM, and have only run into hard blockers on a library or tool a handful of times. The solve was generally pretty straightforward to either use an ARM-compatible alternative, or to cross-compile it myself.
- We've done this many many times before and it's not that bad. I know I'm not the only one old enough here to remember the days of having heterogeneous fleets across PPC, SPARC, and x86. Or even more recent - different extensions for x86 with different chipset manufacturers.