ElectricFence is great, and I recall having a similar feeling to the OP when I first used it because many of the described issues were also new to me at the time. It's a great experience :)
If you're on Solaris or a Solaris-derived OS, you also have the libumem and watchmalloc libraries that can help you out. I have used libumem to great effect in the past.
It's been a while, but I anecdotally recall that Solaris is more stingy with its mallocs than Linux. I used to compile and run my C projects on Solaris as a first pass in my search for memory errors.
Since Git LFS support in GitLab is targeted at both CE and EE, do you expect to open up the EE-only git-annex support to CE as well since the two features serve similar purposes?
I see. You are correct, Erlang definitely falls on the shared-nothing side if you ignore some uses of ETS and ref-counted binaries (binaries above 64 bytes, I believe).
I disagree with the statement "Erlang requires multiple machines to shine," but my motivations, and my definition of "shine," are likely different from yours. Erlang only requires >1 machines if you are creating a system with proper fault-tolerance. In my opinion, it has good SMP support and it performs well for general-purpose applications. You wouldn't use it to implement matrix multiplication, though :)
Anyway, the gist of my post is that the lessons learned from writing Erlang programs might be useful in your work.
Good luck! I look forward to hearing about your progress.
I haven't written any Clojure, but I have written a good bit of Erlang. My understanding of the Erlang Virtual Machine tells me that it solves a lot of the problems you describe regarding immutable data. For example, per-process garbage collection due to immutability, message passing via copying data, etc.
I think you would be well-served to experiment with and learn some Erlang to help inform your design. In fact, Erlang often feels to me like an operating system due to the independence of processes (and the multitude of tools built on top of it, but that's not directly relevant here). I've often dreamt of an Erlang Machine like the Lisp Machines of the past.
As an aside, Erlang's primary goal is fault-tolerance. It's other properties, such as immutable data, message passing, and functional properties were all design decisions made to achieve this goal. My point is that your OS could be very well suited to fault-tolerant systems.
I've used it lightly and so far it has worked well.