Why not port the Linux kernel to Common Lisp? (2009)(stackoverflow.com)
stackoverflow.com
Why not port the Linux kernel to Common Lisp? (2009)
http://stackoverflow.com/questions/1848029/why-not-port-linux-kernel-to-common-lisp
6 comments
This seems like a perfectly reasonable reaction :-)
I would add: if someone really _did_ do all the work to write a whole modern operating system in Lisp, it would be criminally unimaginative to use the Linux ABI as your interface to the world.
You would have to write your own userspace too for the whole enterprise to be worth bothering with. And then you have basically written SqueakNOS (http://squeaknos.blogspot.ch). :-)
I would add: if someone really _did_ do all the work to write a whole modern operating system in Lisp, it would be criminally unimaginative to use the Linux ABI as your interface to the world.
You would have to write your own userspace too for the whole enterprise to be worth bothering with. And then you have basically written SqueakNOS (http://squeaknos.blogspot.ch). :-)
> Why port the Linux kernel to Lisp?
For many of the same reasons people started writing OS kernels in C rather than assembly: Less scut work, wrong code looks wrong more often, and more opportunity for code reuse.
Other reasons include being able to provide a higher-level API, which only kinda happened with the move to C. A Lisp API would enable things like system-level garbage collection and possibly things like a single-address-space OS where the kernel is responsible for fine-grained memory protection and multiprocessing becomes as cheap as multithreading without becoming as unsafe as multithreading.
But that wouldn't be Linux anymore, I suppose; it would be a Linux-compatible system with extended features beyond what the C kernel provides. On the other hand, is Linux 3.11 the same as Linux 1.0 or is it merely mostly-compatible with Linux 1.0?
For many of the same reasons people started writing OS kernels in C rather than assembly: Less scut work, wrong code looks wrong more often, and more opportunity for code reuse.
Other reasons include being able to provide a higher-level API, which only kinda happened with the move to C. A Lisp API would enable things like system-level garbage collection and possibly things like a single-address-space OS where the kernel is responsible for fine-grained memory protection and multiprocessing becomes as cheap as multithreading without becoming as unsafe as multithreading.
But that wouldn't be Linux anymore, I suppose; it would be a Linux-compatible system with extended features beyond what the C kernel provides. On the other hand, is Linux 3.11 the same as Linux 1.0 or is it merely mostly-compatible with Linux 1.0?
Less scut work is a good arguement when writing a kernel from scratch. The scut work has already been done in the linux kernel.
For those are interested, I am sure it has been brought up before, the inverse has been brought up before, an in-kernel Scheme for debugging and some more crazy testing, schemix. [0] Unfortunately, the project is long since deprecated.
[0] http://www.abstractnonsense.com/schemix/
[0] http://www.abstractnonsense.com/schemix/
I updated it long ago to sort-of work with 2.6, but lost the code in a hard drive crash (right after getting commit access but before check-in, sigh), then lost interest on redoing it.
Are there any operating systems that run on an x86 machine that are open source, well-written and written in Common Lisp or Scheme? It doesn't need to be a production OS, just a toy one to learn from.
Movitz was neat back in the day (last update 2008): http://common-lisp.net/project/movitz/
That's a one man effort to write a Lisp OS for x86 from scratch. He wrote his own compiler, basically with the chain of macroexpansions bottoming out in WITH-INLINE-ASSEMBLER.
I only played with it once. I was struck by how cool it was that I could play with the Lisp REPL from inside the garbage collector (also written in Lisp). This was surprising but then seemed obvious: the GC is simply a Lisp function that's moving objects from old-space into new-space (and then deleting old-space), and it doesn't mind if you wake up and interleave a few more allocations into new-space.
Movitz was current back in 2004 when Pascal Costanza organized the first real european Lisp workshop in Oslo and a whole bunch of Lisp hackers who only knew each other through the 'net got to meet up in person. fun times.
That's a one man effort to write a Lisp OS for x86 from scratch. He wrote his own compiler, basically with the chain of macroexpansions bottoming out in WITH-INLINE-ASSEMBLER.
I only played with it once. I was struck by how cool it was that I could play with the Lisp REPL from inside the garbage collector (also written in Lisp). This was surprising but then seemed obvious: the GC is simply a Lisp function that's moving objects from old-space into new-space (and then deleting old-space), and it doesn't mind if you wake up and interleave a few more allocations into new-space.
Movitz was current back in 2004 when Pascal Costanza organized the first real european Lisp workshop in Oslo and a whole bunch of Lisp hackers who only knew each other through the 'net got to meet up in person. fun times.
I can understand why the question was closed as not constructive, but it's still an interesting question. An OS written in Lisp targeted towards current needs (as opposed to the needs we had in the 70's) would be a worthy research goal.
But I wouldn't go for Common Lisp. I'd go for something smaller and cleaner.
But I wouldn't go for Common Lisp. I'd go for something smaller and cleaner.
Yeah, you'd need a more focused Lisp implementation specifically designed for writing low-level code. Among its features should be
1. The ability to manually manage memory instead of relying on a GC.
2. Low-level access to memory. As in, the ability to address an arbitrary location in physical memory
3. Inline assembly and the ability to link with assembly code
There have been operating systems written in Lisp in the past, such as the operating systems for the various Lisp Machines. https://en.wikipedia.org/wiki/Lisp_machine
1. The ability to manually manage memory instead of relying on a GC.
2. Low-level access to memory. As in, the ability to address an arbitrary location in physical memory
3. Inline assembly and the ability to link with assembly code
There have been operating systems written in Lisp in the past, such as the operating systems for the various Lisp Machines. https://en.wikipedia.org/wiki/Lisp_machine
All of those features are already in SBCL, in one form or another. I think the issue is a little deeper than that; for example, the difficulty of producing a small executable in SBCL, one which does not come with the entire CL runtime attached to it. There is also the difficulty of debugging low-level code in SBCL.
In other words, the problem is not with Lisp, nor with a lack of support for low-level features, but with tooling. The tool support for writing a kernel in Common Lisp is pretty bad at the moment, and there are not many people trying to improve on that.
In other words, the problem is not with Lisp, nor with a lack of support for low-level features, but with tooling. The tool support for writing a kernel in Common Lisp is pretty bad at the moment, and there are not many people trying to improve on that.
It does? Could you point to some documentation? I'd be really interested in that.
Those features are mostly in SBCL's FFI, which is (somewhat) documented here:
http://www.sbcl.org/manual/#Foreign-Function-Interface
You can disable the garbage collector with without-gcing or gc-off. To manually allocate and deallocate you need to go beyond SBCL, at least as far as I know; you can call malloc and free directly if you want (via the FFI) or you can write your own allocator using the low-level memory access facility.
Hiding in the low-level code in the FFI and in SBCL itself are "system area pointers:"
http://www.sbcl.org/manual/#index-int_002dsap
Finally, inline assembly. This is the least documented of the three features you mentioned, but here is an example of how to do it with the VOP facility:
http://pvk.ca/Blog/Lisp/hacking_SSE_intrinsics-part_1.html
Unfortunately, all of the above are extremely hard to use due to the lack of good tooling. Manual memory management is possible, but going from a pointer to a block of memory to a Lisp object is not at all straightforward (it is possible with a lot of low-level hacking using the sb-vm and sb-sys packages). You can define VOPs, but it involves a lot of steps; you are probably better off just using the FFI, but debugging across the FFI boundary is hard, and the FFI imposes a performance hit that might outweigh the benefit of your assembly language code.
Like I said, the problem here is tooling. I suspect that there is a lack of demand for these things among SBCL and CMUCL users, and in the Lisp community in general.
http://www.sbcl.org/manual/#Foreign-Function-Interface
You can disable the garbage collector with without-gcing or gc-off. To manually allocate and deallocate you need to go beyond SBCL, at least as far as I know; you can call malloc and free directly if you want (via the FFI) or you can write your own allocator using the low-level memory access facility.
Hiding in the low-level code in the FFI and in SBCL itself are "system area pointers:"
http://www.sbcl.org/manual/#index-int_002dsap
Finally, inline assembly. This is the least documented of the three features you mentioned, but here is an example of how to do it with the VOP facility:
http://pvk.ca/Blog/Lisp/hacking_SSE_intrinsics-part_1.html
Unfortunately, all of the above are extremely hard to use due to the lack of good tooling. Manual memory management is possible, but going from a pointer to a block of memory to a Lisp object is not at all straightforward (it is possible with a lot of low-level hacking using the sb-vm and sb-sys packages). You can define VOPs, but it involves a lot of steps; you are probably better off just using the FFI, but debugging across the FFI boundary is hard, and the FFI imposes a performance hit that might outweigh the benefit of your assembly language code.
Like I said, the problem here is tooling. I suspect that there is a lack of demand for these things among SBCL and CMUCL users, and in the Lisp community in general.
Ahh, yeah, hmm. That's not exactly what I meant. A language for writing an OS on a conventional CPU would need to allow straightforward control of the memory for its own data structures, not just foreign memory. In lieu of that, I suppose you could turn just off the GC when doing time-sensitive things (like responding to interrupts).
That's sort of besides the point, though, because the main problem with SBCL and other CL implementations is that they are written for use in userspace programs and depend on having access to kernel system calls and library functions like malloc(). So if you were to write a kernel in CL, you'd first have to write an implementation for CL that can run on bare metal. At that point, you might as well just implement a different Lisp dialect that only has the features you need in order to write the kernel.
That's sort of besides the point, though, because the main problem with SBCL and other CL implementations is that they are written for use in userspace programs and depend on having access to kernel system calls and library functions like malloc(). So if you were to write a kernel in CL, you'd first have to write an implementation for CL that can run on bare metal. At that point, you might as well just implement a different Lisp dialect that only has the features you need in order to write the kernel.
If you use a CPU that is specifically designed to run LISP, then you don't need any of that:
[http://dspace.mit.edu/handle/1721.1/5731]
[http://dspace.mit.edu/handle/1721.1/5731]
Did that ever run code?
I don't think it was ever built.
It's just interesting that in theory a CPU could be designed that didn't have the concept of memory as a sequential array of bytes.
In fact, in the future this will probably rear its head again. Managing storage at a higher level could make things like massively parallel computing, caching schemes and virtual memory more powerful.
Perhaps one day the elaborate hoops current CPUs and OSs have to jump through to maintain performance in the face of memory latency limitations will be surpassed by something like this, and just as C is best suited as a systems programming language for a flat memory model, so a language like lisp will be best suited to low-level programming for this new architecture.
It's just interesting that in theory a CPU could be designed that didn't have the concept of memory as a sequential array of bytes.
In fact, in the future this will probably rear its head again. Managing storage at a higher level could make things like massively parallel computing, caching schemes and virtual memory more powerful.
Perhaps one day the elaborate hoops current CPUs and OSs have to jump through to maintain performance in the face of memory latency limitations will be surpassed by something like this, and just as C is best suited as a systems programming language for a flat memory model, so a language like lisp will be best suited to low-level programming for this new architecture.
The Lisp Machines? They sold a few of them, but they ultimately weren't commercially successful.
https://en.wikipedia.org/wiki/Lisp_machine
It would be interesting to see how to implement a Lisp machine for the modern day. For it to be useful, it should support some sorts of parallel-processing capabilities that conventional CPUs don't have. Perhaps pmap and preduce as SIMD instructions.
https://en.wikipedia.org/wiki/Lisp_machine
It would be interesting to see how to implement a Lisp machine for the modern day. For it to be useful, it should support some sorts of parallel-processing capabilities that conventional CPUs don't have. Perhaps pmap and preduce as SIMD instructions.
I wouldn't say they were not commercially successful. In their market they must have generated sales of hard- and software of a billion dollars (dollars of the 80s). Was the Commodore C64 not successful, just because it is no longer used?
> For it to be useful, it should support some sorts of parallel-processing capabilities that conventional CPUs don't have. Perhaps pmap and preduce as SIMD instructions.
A Lisp Machine can be useful without that or with different architectures. In those days there were quite a few attempts to speed up Lisp. Thinking Machines sold the ultimate Lisp accelerator: the Connection Machine. Stuff like that was mind-blowing and mind-blowing expensive.
Remember, a useful Lisp Machine in the early 90s had a 40 Mhz cpu with 40bit architecture, a graphics accelerator for a second screen, 40 or more megabytes of memory, a 400 MB disk, 10Mbit/sec Ethernet, 5 Mbit/sec SCSI, ...
Today we get already multiples of those numbers for tiny money and a 64bit processor. Lisp runs nicely when implemented natively.
> For it to be useful, it should support some sorts of parallel-processing capabilities that conventional CPUs don't have. Perhaps pmap and preduce as SIMD instructions.
A Lisp Machine can be useful without that or with different architectures. In those days there were quite a few attempts to speed up Lisp. Thinking Machines sold the ultimate Lisp accelerator: the Connection Machine. Stuff like that was mind-blowing and mind-blowing expensive.
Remember, a useful Lisp Machine in the early 90s had a 40 Mhz cpu with 40bit architecture, a graphics accelerator for a second screen, 40 or more megabytes of memory, a 400 MB disk, 10Mbit/sec Ethernet, 5 Mbit/sec SCSI, ...
Today we get already multiples of those numbers for tiny money and a 64bit processor. Lisp runs nicely when implemented natively.
I grant you that this would be an interesting research project, but Linus isn't going to rip out the internals of the linux kernel, start from scratch, and re-implement everything in Lisp just for kicks. Too much work for too little return, and Iron Lisp is still not as performant as C. Also(I would confidently wager) more kernel programmers are proficient in C than in Lisp, at least for systems programming.
It's difficult to take the question seriously rather than as an attention-grab.
It's difficult to take the question seriously rather than as an attention-grab.
Not to mention that one of the main arguments for using Lisp was "Lisp programs are less buggy than C programs". That's assuming that you trust your Lisp implementation. All implementing it in Common Lisp does is move some of the potential bugs from the kernel to the Lisp implementation.
Common Lisp can be separated into core of 25 or so core operators and rest could be considered as library.
I think CL is in a way more operating system like than most other Lisps and would make good base for operating system. You can load and unload stuff, clean the system, etc.
I think CL is in a way more operating system like than most other Lisps and would make good base for operating system. You can load and unload stuff, clean the system, etc.
It's turtles all the way down.
That idiom doesn't apply to this situation.
Doesn't implementing a kernel in LISP really just move the interesting bits into the LISP GC, compiler, and runtime?
What makes you think that those bits cannot be written in Lisp? Keep in mind that entire OSes were written in Lisp in the 70s and 80s.
No more so than writing a kernel in C moves the interesting bits into the compiler. LISP OS's have been made before, and there is no reason one couldn't be written again that looks UNIX-like.
The networking stack, device drivers, bootloader, etc, all need to be written in much the same way they would in a C-written kernel.
The networking stack, device drivers, bootloader, etc, all need to be written in much the same way they would in a C-written kernel.
[deleted]
Anybody who claims that this has sense, just do it yourself and let me know when you succeed.
Linux took many years to get to the point where it had the stability and performance to be able to compete with commercial operating systems. Users of Linux will not care what language the kernel is written in. They're only going to care whether it will run all of their existing applications without changes, as fast and reliably as Linux does. If the new system doesn't, it will never have any users. (People were willing to put up with the early versions of Linux because it was free and its competitors were not. Any new OS would have to compete with an existing free OS.)
Who would write and maintain it? Linus and most of his followers would stick with the C version, which they know and love, so where are we going to suddenly find the hundreds or thousands of contributors who are both expert Lisp programmers and expert kernel hackers?