5kloc is about 10x larger than musl's existing (old) malloc in source lines. I suspect lots of that is low code density, comments, etc.
I have to lookup what exactly mimalloc is/does every time someone mentions it, because the readme/documentation isn't very descriptive except discussing extensions outside the normal API. I didn't have time to dig through this again today. But I did look at it in some depth on several occasions in the past and it really wasn't suitable for or comparable to what we're doing in musl.
We'd be happy to address specific problems on the mailing list. I believe it's a known issue that the Rust compiler is making really heavy use of rapid allocation/freeing cycles, and would benefit from linking a performance-oriented malloc replacement. Doing so is inherently a tradeoff between many factors including performance, memory overhead, safety against erroneous usage by programs, etc.
One statement in your post, which some readers pointed out was apparently added later, "Others have suggested that the performance problems in musl go deeper than that and that there are fundamental issues with threading in musl, potentially making it unsuitable for my use case," seems wrong unless they just meant that the malloc implementation is not thread-caching/thread-local-arena-based. The threads implementation in musl is the only one I'm aware of that doesn't still have significant bugs in some of the synchronization primitives or in cancellation. It's missing a few optional and somewhat obscure features like priority-ceiling mutexes, and Linux doesn't even admit a fully correct implementation in some regards like interaction of thread priorities with some synchronization primitives, but all the basic functionality is there and was written with extreme attention to correctness, and musl aims to be a very good choice in situations where this matters.
The justifications are partly the same as what Daniel Micay has written extensively on in the rational for hardened_malloc (https://github.com/GrapheneOS/hardened_malloc) - unsynchronized per-thread state inherently sacrifices global consistency for performance and makes it impossible to detect a lot of types of memory usage errors (DF/UAF, etc) that could otherwise be caught.
However musl has the additional constraint of being compatible with small/very-low-memory environments. Lack of global consistency inherently means you will end up using memory less efficiently and requesting significantly more from the system. The new malloc about to go upstream in musl is, to my knowledge, the first/only advanced hardened allocator using slab-type design rather than traditional dlmalloc type split/merge, but also designed for extremely low overhead/waste at low to moderate usage rather than extreme performance. And in the vast majority of applications, this is perfectly reasonable. Even Firefox for example does very well with it.
With that said, new malloc is expected to be somewhat faster than old on lots of workloads (and considerably faster than old would be if we fixed the flaws in old that motivated it), but it's not a performance-oriented allocator. If you really want/need that you should probably link jemalloc or similar (and accept all the tradeoffs that come with that). In Rust programs without "unsafe", it may make sense to do that by default.
To clarify, the numbers in my tweets are measured on Linux, comparing musl libc's posix_spawn (with CLONE_VM) to plain fork+exec (which should be independent of libc). I just posted the test program on our mailing list: http://www.openwall.com/lists/musl/2015/06/04/1
The biggest problem with Cygwin is that programs linked with Cygwin inherit global state from from a Cygwin installation on the system they're running on. If you want to produce a Windows program that just runs on any system you install it on using Cygwin, it will work right for most users, but if a power user who has Cygwin installed and has their own custom mounts, options (like different binary/text mode settings), etc. tries to run your program, it might break spectacularly. This makes Cygwin a really poor choice for making binaries you want to distribute as standalone programs.
Aside from that, Cygwin tries to hard to be a complete Unix environment on Windows, whereas midipix just gives you enough to use interfaces that were standardized in POSIX as a reasonable, uniform API for all operating systems to provide. Some functions go beyond that, but you don't have to use them. And even some things that are mandatory in POSIX are optional in midipix; as I understand it, you can choose at build time whether you want the overhead of being able to support tty devices (and the associated semantics like job control, signals from the controlling tty, etc.).
These are some good observations. Note that on modern POSIX, calling anything but async-signal-safe functions after forking in a multi-threaded process results in undefined behavior. I think it's totally reasonable to consider any programs using the WinAPI (not just POSIX functions provided through midipix) as being formally multi-threaded, and to consider the WinAPI non-AS-safe. I'm not 100% sure what midipix is doing in this regard, but my advice on the project (I'm the primary author/maintainer of musl libc, which midipix is using) has been not to worry about making arbitrary functions work after fork, but only supporting the things that POSIX requires to work. My view in general is that fork should be phased out, but posix_spawn is not sufficiently powerful yet to replace all uses of fork+exec -- it can't do advanced uid changes, setsid, resource limits, etc., and posix_spawn can't be used effectively from an async signal context because the API (attributes and file actions) inherently involves allocation.
No, fork is only one path that can lead to overcommit. Allocation of new memory as COW references to a zero page, and COW writable MAP_PRIVATE mappings of files (such as the writable LOAD segments of any executable or library file) also lead to overcommit unless you do proper commit accounting. Any system that does not need to do detailed commit accounting to avoid overcommit is basically wasting the fact that it has virtual memory/MMU.
No, OOM killer is completely orthogonal to this and is a consequence of not doing correct commit accounting. With strict commit accounting turned on (vm.overcommit_memory=2) fork will correctly fail when there is not sufficient physical backing. But you're correct that fork+exec is a bad model.
Yes, this is the really big deal that everyone focused on fork and mmap semantics and other details is overlooking. Having midipix as the means of producing Windows versions of cross-platform software like Firefox should make it possible to remove a lot of ugly #ifdeffery and/or whole "portability layers" that are avoiding the standard functions like fopen because of a lack of Unicode support on Windows.
Interix is a lot different because it requires installing a system component, which requires administrator privileges. Midipix produces applications that (at least as I understand it) run on basically any NT-based Windows with no special privileges.
There are still plenty of applications that use fork semantically, to keep the same process image in the child, which benefit somewhat from a fast fork. But most places where fork affects performance now are things that are already pessimized by using fork+exec instead of posix_spawn: the shell, make, cgi, etc. (GCC still uses vfork, but GNU make recently switched from vfork to fork because of vfork-related bugs.) Regardless of how fast or slow fork is on midipix (but I expect it to be fairly fast, much faster than cygwin), they would benefit a lot more from just switching to using posix_spawn.
The choice isn't between 30sec or 15sec boot times. It's between 30sec, 15sec, or 1sec boot times, the latter coming from dropping all of the crap and writing a flat linear /etc/rc file.
If your boot time was bad before and decent now, this is not thanks to the goodness of systemd but rather the badness of whatever hideous system your distro was using before, and/or because your distro is starting a bunch of useless junk that shouldn't be running in the first place. I've been using flat /etc/rc (all commands in one file, & at the end of anything non-essential) for years (decades?) now, and have always had a login prompt faster than the display can synchronize to the video mode change.
Both forking and double-forking are completely wrong behavior for supervised daemons run as part of any automated system. Forking makes sense for small systems which lack any automated supervision and where a human admin is responsible for all process starting and stopping, but it's a bug anywhere else. Modern Linux has various methods for working around this bug (like the method of assigning an alternate process for orphaned descendants to be reparented to, which systemd uses, AFAIK), but really any production-quality daemon should provide a non-forking way to start it.
There are plenty of viable alternatives: s6, runit, OpenRC, and so on. I'm not really convinced uselessd is a viable alternative - it keeps way too much of the badness of systemd, but I guess that makes it viable if you were willing to consider systemd to begin with.
A much better solution for the problem of user-facing applications (e.g. "desktop environment" software) depending on systemd's public dbus interfaces is to provide a fake service that gives them fake data - the same way you would sandbox Android apps for privacy by giving them a fake Contacts list, etc.
As for the other main "public interface" of systemd that things are starting to depend on, the systemd service file format, it would be easy to add support for this file format to any other process supervision system.
People actually _want_ the logging behavior of systemd? My impression is that it's the most widely hated part; I've heard endless stories of journald thrashing the filesystem forever, losing logs completely on corruption, etc. And even operating properly, its performance is comparable to grepping a flat text log, since despite having a "more efficient" format, it increased the actual data size by something like 4-10x.
Logs are essentially write-once, write-often, read-rarely data. As such, the optimal format is always going to be a flat, append-only file.
Systems which do not use systemd simply do not have these problems because there is no analogous component. If syslogd goes down, the worst that happens is you don't get logs. Init doesn't go down because it essentially has no inputs. Individual services can go down if they're poorly written, but they won't bring the system down with them. Traditional systems (the hideousness that is "sysvinit") have plenty of other different problems (e.g. race conditions in process supervision), but deadlock or bringing down the whole system is not one of them.
With systemd on the other hand, all of the components under the systemd banner are tightly interconnected and communicating. In particular pid 1 has ongoing communication with multiple other components, and misbehavior from them can, both in theory and in practice, deadlock the whole system. In case you missed it, this is roughly what "monolithic architecture" means: even though the components are modular, they're designed for use in a tightly interwoven manner that's fragile. It's completely the opposite type of "monolithic" from the kernel, which has everything running in one address space, but with architectural modularity, where interdependency between components is kept fairly low.
Whether it's all in pid 1 or not is irrelevant. What matters is that it has a monolithic architecture, whereby breakage in any one part or their communication channels can bring down the whole system. This is not just a theoretical concern; it has REPEATEDLY happened.
I have to lookup what exactly mimalloc is/does every time someone mentions it, because the readme/documentation isn't very descriptive except discussing extensions outside the normal API. I didn't have time to dig through this again today. But I did look at it in some depth on several occasions in the past and it really wasn't suitable for or comparable to what we're doing in musl.