64 bit OS Raspberry Pi4 Benchmarks(medium.com)
medium.com
64 bit OS Raspberry Pi4 Benchmarks
https://medium.com/@matteocroce/why-you-should-run-a-64-bit-os-on-your-raspberry-pi4-bd5290d48947
17 comments
AES is quite efficient in software. The RPi4 should be able to saturate its network connectivity many times over with SW AES.
A Pentium II 200 MHz could saturate 100 Mbps with AES-128[1], RPi4 has 4 x 1400 MHz cores (with 64-bit ALU & NEON available).
[1] https://www.di.ens.fr/~granboul/recherche/AES/timings.html
edit: There are RPi4 AES and other benchmarks here in this "openssl speed" paste: https://gist.github.com/HimaJyun/f05d3017dfb05a4ccb0def010bb... - indicates 50-80 MB/s = 400-640 Mbps per core.
A Pentium II 200 MHz could saturate 100 Mbps with AES-128[1], RPi4 has 4 x 1400 MHz cores (with 64-bit ALU & NEON available).
[1] https://www.di.ens.fr/~granboul/recherche/AES/timings.html
edit: There are RPi4 AES and other benchmarks here in this "openssl speed" paste: https://gist.github.com/HimaJyun/f05d3017dfb05a4ccb0def010bb... - indicates 50-80 MB/s = 400-640 Mbps per core.
Fast table based software AES implementations leak bits of the secret key in array offsets, so anything that can use the cache on the chip can get the key. A software implementation of AES that is secure against side channel attacks by using bitslicing is slow, you would be better off using chacha20-poly1305.
So you're saying OpenSSL might still be using side channel vulnerable AES code, all these years after there was a big row about it?
This ARMv8 impl according to comments is constant time: https://github.com/openssl/openssl/blob/master/crypto/aes/as...
There are also other ARM AES implementations in the tree: https://github.com/openssl/openssl/tree/master/crypto/aes/as...
AFAICT one is for the hardware AES instructions, and one is a bitsliced (so constant time) impl for 32-bit ARMv7, and one is for low end ARMv4. The latter sounds like it might still be vulnerable to timing sidechannel attacks but no idea if it is still used by default in any configuration...
This ARMv8 impl according to comments is constant time: https://github.com/openssl/openssl/blob/master/crypto/aes/as...
There are also other ARM AES implementations in the tree: https://github.com/openssl/openssl/tree/master/crypto/aes/as...
AFAICT one is for the hardware AES instructions, and one is a bitsliced (so constant time) impl for 32-bit ARMv7, and one is for low end ARMv4. The latter sounds like it might still be vulnerable to timing sidechannel attacks but no idea if it is still used by default in any configuration...
I don't know about openssl's status on old / weak platforms. People who need AES without hardware AES instructions should be using BearSSL code or the rust code that has been reviewed by the author of BearSSL (see e.g. https://github.com/RustCrypto/block-ciphers/issues/65).
I have heard that AES is very complicated to implement in software, if one intends to avoid side-channel leaks.
You don't, especially not GCM mode. You use one of the mature tested versions out there.
To build on this: I was able to implement AES and OCB mode by just reading papers without any code in them. I was, however, not able to implement GCM reliably even by translating a "simple" reference C implementation into scheme. Sure, it worked, but even after 2 rewrites it still did not produce the same output as the simple reference implementation for some edge cases.
All this was done on a just-for-fun basis, but it ended up just making me frustrated so I stopped trying.
All this was done on a just-for-fun basis, but it ended up just making me frustrated so I stopped trying.
I use geli for full disk encryption on FreeBSD.. If you do not load the kernel model "aesni" it is intolerably slow.
They are a recommended extension.
I expect that the crypto instructions are not present on the RPi3/4 because of legal reasons, to allow it to be exported to Iran and such...
Syncthing hashing performance of Rock64 (that has crypto extensions) is ten times that of Raspberry pi 4.
One of the main benefits you get to running a proper aarch64 userspace is ability to run a modern functional firefox install. The 32-bit builds usually don't support anything newer than a very old LTS release. It's been hell trying to build and/or run that target (see: https://bugzilla.mozilla.org/show_bug.cgi?id=1452128 )
This is especially important if considering use as a low-end desktop as Chromium will eat through 4GB of ram like it's nobody's business.
This is especially important if considering use as a low-end desktop as Chromium will eat through 4GB of ram like it's nobody's business.
I'm running Firefox 73 on my BeagleBoard-X15 (Cortex-A15). Ubuntu 18.04.
$ dpkg -l | grep firefox
ii firefox
73.0.1+build1-0ubuntu0.18.04.1 armhf
Safe and easy web browser from Mozilla
$ dpkg -l | grep firefox
ii firefox
73.0.1+build1-0ubuntu0.18.04.1 armhf
Safe and easy web browser from Mozilla
So that's why it's showing up with 59 being the latest versions from the distro for arm/arm64? https://packages.ubuntu.com/bionic/web/firefox
Where can I get it? I didn't think Mozilla offered a precompiled version for aarch64 Linux. I've been using an the Debian ESR release (68.0.5) from the Armbian repository on my NanoPi M4.
One of the things that I find great about Raspbian is that the boot files are stored on a FAT32 partition which is readable from any OS, and there are some nice considerations for headless setups: drop in a wpa_supplicant.conf to configure Wi-Fi, `touch ssh` to get OpenSSH enabled at next boot. Do any of the alternative OSes, like Ubuntu, offer this?
I recently installed Arch Linux on my Pi Zero, it also uses FAT32 for the boot partition. I did the initial setup the way you described. Check out https://archlinuxarm.org/ , it's pretty great.
So that's how they came up with that. As someone who normally installs "normal" Linux systems I find it quite irritating that you have to put a file somewhere, especially the boot record, to "enable" ssh of all things. Up until now I considered it a weird decision. (and I still think it is not optimal) I deploy my RPi's in the field and don't put a monitor on them so I would expect ssh running as default. First time I found out about it was when reading the unit file when I was building a custom image based on raspbian, so I wouldn't consider it obvious :) When working on a Linux Device I just mount the main partition and do my customizations.
> I deploy my RPi's in the field and don't put a monitor on them so I would expect ssh running as default. First time I found out about it was when reading the unit file...
So I guess you weren't building those images? If you're building headless RPi images that's something you learn immediately.
So I guess you weren't building those images? If you're building headless RPi images that's something you learn immediately.
I have a tendency to learn stuff in reverse. I used to make an SD Card ready, connected monitor and keyboard and enabled everything as needed. Then I got the task to deploy n-RPi's and looked at the image first to customize it for the requirements we had. That's when I looked around and saw the unusual unit file and tried to understand why the service would look at the boot partition to start a service. In the end I think I added the symlink to start the service thorugh systemd.
I don’t know the details of how they implement this but it sounds like you want a systemd drop in file to override the ConditionPathExists (or similar) directive (I’m assuming you can’t or don’t want to modify the upstream unit directly given the use of symlinks).
It's at https://github.com/RPi-Distro/raspberrypi-sys-mods/blob/mast...
Or you can chroot to the mounted Raspbian root partition and do a normal `systemctl enable ssh` as part of your image customisation. Because, to be clear, you do not have to put a file in /boot to enable SSH, as it was claimed above. That is purely a helpful shortcut.
Or you can chroot to the mounted Raspbian root partition and do a normal `systemctl enable ssh` as part of your image customisation. Because, to be clear, you do not have to put a file in /boot to enable SSH, as it was claimed above. That is purely a helpful shortcut.
Raspbian did originally come with SSH enabled by default but the default credentials pi/raspberry made it trivial for misuse:
https://www.zdnet.com/article/linux-malware-enslaves-raspber...
Exactly. You can't have it both ways: make it easy to remember the user/pass and also make it easy to login remotely.
Probably the right decision then. As I don't put them on public networks and delete the pi user this is of little concern to me, but given the target group, it is a simple safety measure.
I think there is a better solution: On a new install on first login, over ssh or on the gui, a user/password must be created.
This way the initial login only works once. Both gui user/pass and ssh user/pass are tied by default.
This way the initial login only works once. Both gui user/pass and ssh user/pass are tied by default.
This is how it works in ARMbian. It forces a password change on first login. It can be annoying if you intend on deleting the alarm user right after that, but I can easily see why. "Default" passwords are always suboptimal.
Have you considered something like Yocto[0]? I've had good luck with it.
[0] https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-Yocto.h...
[0] https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-Yocto.h...
I'm interested in trying it. In the past I used a system based on Debian's Live Build (https://github.com/gumstix/live-build) and I was wondering how it differs, but I'm just now noticing even this suggests using Yocto instead.
it is a bit of work to get rolling, but IMHO worth the effort
Dietpi is pretty similar and easy to setup - https://dietpi.com/
One thing I really wish raspbian had is full disk encryption.
I'd go for vanilla Debian, but cannot since raspberry pi4 hardware is not open enough.
The Ubuntu releases for the Raspberry Pi utilize cloud-init with the cloud-init configuration files being stored on the FAT32 partition.
Although there are some benefits, I see it as a complication.
The requirement of multiple partitions, with different filesystem types causes a lot of needless complication in the configuration and boot process.
The requirement of multiple partitions, with different filesystem types causes a lot of needless complication in the configuration and boot process.
[deleted]
One downside: you will probably need a bit more memory to do the same work. An unmentioned benefit: better ASLR, for what that's worth.
> A 64 bit system means that RAM can be accessed in 8 byte read/writes per instruction.
I mean...kind of but that's probably not really what's happening in this benchmark. Firstly, in that test `memset()` is surely using NEON instructions internally on both ARMv7 and AArch64, which can load/store up to 32 bytes in a single instruction. Further that test is really just showing the bandwidth of the memory controller. I'm not sure why AArch64 would matter there. It's possible that `memset` / `memcmp` are using smarter prefetching instructions in AArch64.
> A 64 bit system means that RAM can be accessed in 8 byte read/writes per instruction.
I mean...kind of but that's probably not really what's happening in this benchmark. Firstly, in that test `memset()` is surely using NEON instructions internally on both ARMv7 and AArch64, which can load/store up to 32 bytes in a single instruction. Further that test is really just showing the bandwidth of the memory controller. I'm not sure why AArch64 would matter there. It's possible that `memset` / `memcmp` are using smarter prefetching instructions in AArch64.
I wonder if using zram or zswap can help "eat" those overly long pointers in RAM via compression ?
I've never understood why zswap decompresses the pages before writing them out to disk. Disk IO is the slowest part of the whole chain, you've already paid to compress the data, why on earth would you throw that away AND use up more disk IO?
My guess is that this is due to some inherent limitations of the Linux kernel swap mechanism ? Possibly so that you can safely disable zswap at runtime (you can) ?
Or quite possibly no one implemented it just yet & writing uncompressed pages is transparently compatible with what normal swapping does.
Or quite possibly no one implemented it just yet & writing uncompressed pages is transparently compatible with what normal swapping does.
Compressing improves disk's bandwidth
Noob question: What kind of optimizations are even responsible for those kinds of performance improvements?
32 bits is enough to utilize all 4 GB of the Raspberry Pi4, so I figured the only benefit of using a 64 bit OS would be to support 64 bit software. Why would a 64 bit build perform better than a 32 bit build on the same hardware?
32 bits is enough to utilize all 4 GB of the Raspberry Pi4, so I figured the only benefit of using a 64 bit OS would be to support 64 bit software. Why would a 64 bit build perform better than a 32 bit build on the same hardware?
64-bit ARM NEON (SIMD instruction set) is much more efficient vs 32-bit. And not just NEON, there are more registers as well, deeper pipeline, etc.
Also, 32-bit ARMs often (but not always) don't have hardware integer division.
Integer division is mandatory from v7VE on upwards (so roughly Cortex-A9, A15 and later), and it's definitely in the 32-bit support of any 64-bit capable core. If Raspbian is compiling its packages for the lowest common denominator arm v6 cpu some Pis have, then you'd get some slowdown from division being an out-of-line call (to library code that can use the hw insn), but you'd see that part of the speedup just from building for 32-bit but optimised for newer cores than v6, I think.
>If Raspbian is compiling its packages for the lowest common denominator arm v6 cpu some Pis have
It is definitely the case, the official Raspbian images support Raspberry Pi 1.
It is definitely the case, the official Raspbian images support Raspberry Pi 1.
> 32 bits is enough to utilize all 4 GB of the Raspberry Pi4
For 4 GB of memory, 32 bits of address space are not good enough, as discussed a couple of days ago (https://lwn.net/Articles/813201/ discussed here at https://news.ycombinator.com/item?id=22440733).
For 4 GB of memory, 32 bits of address space are not good enough, as discussed a couple of days ago (https://lwn.net/Articles/813201/ discussed here at https://news.ycombinator.com/item?id=22440733).
There's also 64-bit Ubuntu Server and Core.
→ https://www.raspberrypi.org/downloads/
→ https://www.raspberrypi.org/downloads/
Slightly off topic, but kind of on theme at least: does anyone know where to get an SBC (single board computer) with 6GB of RAM... or even 4.5GB (that doesn't cost like $~300)? The raspberry PI is almost perfect, I just need slightly more ram per unit.
ARM or X86 I really don't care...
ARM or X86 I really don't care...
You should get an old server rack at that price point unless space is important for you.
Udoo - https://shop.udoo.org/udoo-x86-ii-ultra.html
LattePanda - https://www.lattepanda.com/products/lattepanda-alpha-864s.ht...
There are a few more. You could also build your own. All you need is a micro ITX motherboard + ram and a processor and something to power. Get them used and it will be way more powerful.
Udoo - https://shop.udoo.org/udoo-x86-ii-ultra.html
LattePanda - https://www.lattepanda.com/products/lattepanda-alpha-864s.ht...
There are a few more. You could also build your own. All you need is a micro ITX motherboard + ram and a processor and something to power. Get them used and it will be way more powerful.
Yeah; I just want something smaller and easier to toss around than the giant ass server I have sitting next to me.
I've got a LattePanda v1 but I've been a bit disappointed with it. Udoo bolt is my next test but they're just a bit more expensive than I'd like.
I've got a LattePanda v1 but I've been a bit disappointed with it. Udoo bolt is my next test but they're just a bit more expensive than I'd like.
Just build a mini-itx.
An old asus chromebox.
Apparently in `config.txt` you can set `arm_64bit=1`
Then `sudo rpi-update`
Then `sudo rpi-update`
This results in a 64-bit kernel, but 32-bit userland, whereas the arm64 Debian is entirely 64-bit.
Do you need the `arm_64bit=1` flag to get 64-bit kernel, 64-bit userland or no? I would have thought 64-bit everything would have been the default.
The Raspbian images by design support all versions of the Pi, back to the original. The developers have stated they don't have the resources to maintain separate 32- and 64-bit userland images, but provide the config.txt option to at least enable the 64-bit kernel.
If you want a 64-bit userland, there are other options besides Raspbian. You can also try an in-place arch change from Raspbian to aarch64, but I imagine that would break some things, since the Raspbian-specific package repository only has packages for the armhf arch.
If you want a 64-bit userland, there are other options besides Raspbian. You can also try an in-place arch change from Raspbian to aarch64, but I imagine that would break some things, since the Raspbian-specific package repository only has packages for the armhf arch.
I just gave it a try and now apt is installing a bunch of new kernel .img files. So for me, yes, this seemed to be required before 64bit kernels were installed. I'm feeling surprised. I'm 40% of the way through the install and hoping it reboots okay in the end.
Edit: to be more clear, I also run 'apt update' afterwards
Edit: to be more clear, I also run 'apt update' afterwards
Update: oh yeah, that broke a lot of stuff
config.txt only affects the kernel and doesn't affect which userland is selected.
So running the Pi in 64-bit is really overall beneficial. I'm running a customised version of Ubuntu for the Pi4.
https://ubuntu.com/download/raspberry-pi
I've also customised the image by adding users, public keys and such. Removing some of the cloud cruft.
These instructions make it very, very easy and you can do this on an x86 machine. Just make sure to use /usr/bin/qemu-aarch64-static (64 bits) instead of qemu-arm-static (32 bits).
https://powersj.io/post/raspbian-edit-image/
https://ubuntu.com/download/raspberry-pi
I've also customised the image by adding users, public keys and such. Removing some of the cloud cruft.
These instructions make it very, very easy and you can do this on an x86 machine. Just make sure to use /usr/bin/qemu-aarch64-static (64 bits) instead of qemu-arm-static (32 bits).
https://powersj.io/post/raspbian-edit-image/
The flipside is-- if you have a 1GB pi, you're going to be wasting a whole lot of RAM on wider pointers, and you may be more constrained by RAM than CPU throughput.
(Also, 64 bit Pi is not nearly as well supported as 32 bit currently. Does hardware GL work yet? What about if you're going to do MIPI, etc?) IMO it's worth waiting for a little more maturity.
(Also, 64 bit Pi is not nearly as well supported as 32 bit currently. Does hardware GL work yet? What about if you're going to do MIPI, etc?) IMO it's worth waiting for a little more maturity.
Has nobody tried to implement an aarch64 equivalent of the "x32" ABI? That is, using the 64-bit instruction set but keeping pointers 32 bits long?
I have not used or had a need for hardware GL. Don't even know what MIPI is.
I use it as very cheap bare metal. Still plan to use 3 of them as Ceph monitors.
I use it as very cheap bare metal. Still plan to use 3 of them as Ceph monitors.
It used to be Mobile Industry Processor Interface [1] but no longer is used as an acronym.
High-speed serial links for driving displays and cameras in cell phones, originally. Proprietary.
[1] https://en.m.wikipedia.org/wiki/MIPI_Alliance
High-speed serial links for driving displays and cameras in cell phones, originally. Proprietary.
[1] https://en.m.wikipedia.org/wiki/MIPI_Alliance
Depends on the Pi. The A53 based Pis did a little worse in 64 bit mode on average IIRC (probs because of the cache pressure).
I realize I'm a little late to the party as this post is almost a day old, however as a Pi enthusiast I feel obligated to mention RackN's edgelab [1] project which leverages Pi4's PXE boot to rapidly build a 64-bit mini Pi lab and then a k3s cluster on top.
Full disclosure: I don't work for RankN but am a customer; my use case is zero-touch ESXi cluster and Linux builds but I like the tool and have way too many Pis.
[1] https://github.com/digitalrebar/edgelab
Full disclosure: I don't work for RankN but am a customer; my use case is zero-touch ESXi cluster and Linux builds but I like the tool and have way too many Pis.
[1] https://github.com/digitalrebar/edgelab
I’ve been running Ubuntu on most of mine for a while now, but I go back to Raspbian now and then for the hardware support (graphics, in particular, are a bit of a pain, but I’ve also had issues with the built-in Wi-Fi). My “lab” Pi 4 is on Raspbian also because I like noodling in Mathematica (which I don’t think you can get in ARM64 at all for any distro).
It would be awesome to have a decent ARM64 SBC with a good GPU (able to drive 2 4K monitors and run Firefox/VS Code). Any recommendations from the non-Pi crowd?
It would be awesome to have a decent ARM64 SBC with a good GPU (able to drive 2 4K monitors and run Firefox/VS Code). Any recommendations from the non-Pi crowd?
Have you explored the Nvidia Jetson Nano[0]? It runs Ubuntu Aarch64 as its default OS.
[0]: https://developer.nvidia.com/embedded/jetson-nano-developer-...
[0]: https://developer.nvidia.com/embedded/jetson-nano-developer-...
Last time I booted up my Jetson Nano, it was so slow. When I would ssh in it would take like two seconds to get a response after typing ls. Obviously this was sd card reads. It just doesn't have the polish/optimisations raspberry pi has.
Rock Pi 4, Nano Pi M4, Orange Pi 4B, and Pine HardROCK64 ( this one ETA April) are all RK3399, which I think should be able to handle that.
Then there's also Khadas VIM3.
Then there's also Khadas VIM3.
Has anyone gotten the aarch64 Raspberry Pi image of Alpine to work on the Pi 4? It seems like a good fit for a RAM constrained system but I could never get it past the colorful boot screen.
It's fairly easy to trim down what runs on Raspbian. so if your only concern is background tasks eating RAM, you don't need something like Alpine to fix that issue. We use Pis at my work for running slideshows, and while performing their duty they use about 123MB of RAM for all running software (mostly our software + monitoring tools).
Raspbian doesn't offer aarch64 userspace though.
https://dietpi.com/ might be of interest to you. I use it on my old pi, it just works. :)
Add a heatsink and fan. You'll get better performance.
Just run current firmware from last september or later and keep the pi4 standing up on its side naked, it will never throttle no matter what you throw at it. (unless maybe your room is 80F or more)
I've had my pi4 compile stuff for days on end and it does not throttle if the above setup is followed.
I've had my pi4 compile stuff for days on end and it does not throttle if the above setup is followed.
What did the firmware change to enable this kind of performance?
Better power management has reduced overall power consumption by almost a watt.
https://www.hackster.io/news/raspberry-pi-4-firmware-updates...
https://www.hackster.io/news/raspberry-pi-4-firmware-updates...
I'd still recommend a fan or something because the Pi 4 still gets quite hot, like around 50c idle without cooling solutions, and throttling only starts at 80c, which is crazy hot to touch.
I just put a copper and aluminium thing with large mass (about 100g) directly on the processor. Under light load it is about 20 over ambient. Despite CPU stress tests, I have never had it throttle.
Make sure it touches the usb chip.
Edit: my rock pi has a proper heat sink and it runs even cooler. 12 over ambient
Make sure it touches the usb chip.
Edit: my rock pi has a proper heat sink and it runs even cooler. 12 over ambient
Everyone's asking for the Debian image / instal instructions. I'm curious too, from when this post was shared originally and trended in early Jan.
sakaki- has a build guide here: https://github.com/sakaki-/gentoo-on-rpi-64bit/wiki/Build-an...
And weekly kernel builds here: https://github.com/sakaki-/bcm2711-kernel
She focuses more on Gentoo, but it's mostly the same - just pop it onto an existing Debian 64 image.
And weekly kernel builds here: https://github.com/sakaki-/bcm2711-kernel
She focuses more on Gentoo, but it's mostly the same - just pop it onto an existing Debian 64 image.
what would be ideal for raspi 3 and up are aarch64 binaries with a 32 bit memory model (aarch64-ilp32).
Dies debian support/use this?
As far as I'm aware, ilp32 support for aarch64 has been proposed and implemented, but the patches weren't accepted upstream in the kernel because nobody was able to make a sufficiently convincing case that there was enough benefit to justify adding a whole new ABI to the kernel (there are a few benchmarks where it's noticeable but mostly it just doesn't make enough difference to be worthwhile, AIUI). Possibly the situation has changed since I last heard about it.
[deleted]
I think the main issue keeping the pi with a 32b userspace is the lack of availability of 64bit GPU firmware
> I think the main issue keeping the pi with a 32b userspace is the lack of availability of 64bit GPU firmware
Uh, no. It's there because of the design goals and priorities of the Foundation. They want to be able to distribute a distribution that runs on any raspberry pi generation. Performance is of secondary concern to them. So Raspbian remains at 32-bit.
Uh, no. It's there because of the design goals and priorities of the Foundation. They want to be able to distribute a distribution that runs on any raspberry pi generation. Performance is of secondary concern to them. So Raspbian remains at 32-bit.
Can you cite a source for this claim?
Why yes, yes I can: https://www.raspberrypi.org/forums/viewtopic.php?t=252369#p1...
> this will be combined with a 32bit userland, for the reason mentioned above. - the amount of work needed to update all the libraries that talk to the GPU.
My eyes kinda glaze over when I see that because the vc4-fkms-v3d driver works fine enough with mesa on 2D and 3D shit using a full 64-bit system. Maybe it's not as fast as the original proprietary driver, I don't know as I've never used the original proprietary one nor have I seen anyone benchmark it.
The fully open source driver stack can't some soon enough. (yes I know firmware files still exist, but they don't prevent you from running a 64 bit kernel and userspace)
Up until the RPi 4 I thought AES instructions were a part of AArch64, but I was wrong. Such a weird omission to make on (I expect) Broadcom's side. All other 64-bit ARM SBCs just have it, even the low cost ones.