Lua in the NetBSD Kernel [pdf](fosdem.org)
fosdem.org
Lua in the NetBSD Kernel [pdf]
https://fosdem.org/2013/schedule/event/lua_in_the_netbsd_kernel/attachments/slides/278/export/events/attachments/lua_in_the_netbsd_kernel/slides/278/kernel_mode_lua.pdf
9 comments
Do you know if that talk was recorded, by any chance? I checked here http://video.fosdem.org/2013 but couldn't find it there
It wasn't recorded. At least not officially. Fosdem has so many rooms, I think they just give up at some point. It's a shame though.
Edit: Just checked and there are videos for only one devroom. And there were around 30...
Edit: Just checked and there are videos for only one devroom. And there were around 30...
The graph processing room was recorded, as were the lightning talks, but I don't know if those videos are available yet.
As someone who once threw together a Lua kernel module for FreeBSD 4 just for fun, I approve of the idea -- although I don't know how useful it is.
Being able to rapidly prototype kernelspace things is probably the main advantage. I'd think twice about actually running garbage-collected code in the kernel in production. And the slide that touts that it's easy for users to modify -- yes, well -- you could easily argue that it shouldn't be easy for users to modify what's in the kernel :)
Being able to rapidly prototype kernelspace things is probably the main advantage. I'd think twice about actually running garbage-collected code in the kernel in production. And the slide that touts that it's easy for users to modify -- yes, well -- you could easily argue that it shouldn't be easy for users to modify what's in the kernel :)
Indeed, prototyping functionality is the main point here. The whole system isn't even part of the default install and can be turned on/off.
Easier is always good. It's still far from easy though.
Easier is always good. It's still far from easy though.
I've seen this on an embedded product that has been sold for years.
The license is liberal, it's remarkably easy to work with Lua, is can basically call any of your C functions. Great for debugging stuff, prototyping, certain kinds of monkey patches. Great for doing unit test like experiments too; you can have some Lua put the system in to a particular state and then do your experiment.
The license is liberal, it's remarkably easy to work with Lua, is can basically call any of your C functions. Great for debugging stuff, prototyping, certain kinds of monkey patches. Great for doing unit test like experiments too; you can have some Lua put the system in to a particular state and then do your experiment.
How would you compare it with python for things like unit testing c code? Here lua is chosen because of it's small footprint, but for testing that doesn't really matter.
Lua is multi-threading friendly. It is allowed and easy to run multiple VMs in different threads in the same process.
I believe MicroMint, (aka Steve Ciarcia's hardware company), offers Lua on a few ARM boards.
Yea, I agree that it's neat, but probably of limited practical value.
If you need something like this, it's probably best to either host a kernel in userspace (like UML), or else have a lightweight kernel interface that allows you to do your heavy lifting in userspace. I've never done any Plan9 driver development, but my understanding is that Plan9 has userspace equivalents of most of the internal driver API calls, so that a driver module can almost seamlessly be developed as a userspace driver and then (if it turns out to be performance critical) run as an in-kernel module in production.
Once you've got good userspace APIs for the kernel APIs you need to use, then all kinds of rapid prototyping possibilities open up.
If you need something like this, it's probably best to either host a kernel in userspace (like UML), or else have a lightweight kernel interface that allows you to do your heavy lifting in userspace. I've never done any Plan9 driver development, but my understanding is that Plan9 has userspace equivalents of most of the internal driver API calls, so that a driver module can almost seamlessly be developed as a userspace driver and then (if it turns out to be performance critical) run as an in-kernel module in production.
Once you've got good userspace APIs for the kernel APIs you need to use, then all kinds of rapid prototyping possibilities open up.
Lua sometimes drives me nuts, but it's such a humble little language that you can't hate it for too long. There are alternatives I would have preferred, like the Lua-inspired Squirrel (http://http://squirrel-lang.org ), but Lua is the most well-known and is written in plain C, which is likely appealing to a BSD project. I'm interested to see if this gains any popularity and if the other BSDs show interest.
How does it drive you nuts?
Arrays based on 1. The odd and slightly under-documented stack that you have to use when interfacing it with C. Global variable scope by default. Tables (.. are both good and bad). Not standard named operators like '=~' and ':'.
Having said that, it's not a bad language. It has an excellent community, who really went to great lengths to help me writing the Lua libguestfs bindings.
http://libguestfs.org/guestfs-lua.3.html
Having said that, it's not a bad language. It has an excellent community, who really went to great lengths to help me writing the Lua libguestfs bindings.
http://libguestfs.org/guestfs-lua.3.html
The 1-based arrays are theoretically annoying (I'm a C guy), but in practice I find they simply aren't an issue, I guess due to the differences in typical programs for each language. "~=" is unusual (I too wish it used "!=") but other than that, Lua's pretty much like everybody else (":" doesn't really count because it's an operation that doesn't exist in most languages).
"Global by default" can be annoying, but at least it's better than "local by default" like python etc. [As a programmer, who'd rather find bugs easily than save some typing, I think better than either would be "nothing by default, you gotta declare your variables", but that might not be popular for "dash off a quick script" usage; maybe make it configurable based on the load context or something?]
But this:
> The odd and slightly under-documented stack that you have to use when interfacing it with C
... seems almost bizarre. Lua's C interface is probably the most elegant and usable I've ever seen. A stack is the classic abstraction to use for this application, Lua's implementation is well-designed, and Lua's API documentation is both comprehensive and very well written.
"Global by default" can be annoying, but at least it's better than "local by default" like python etc. [As a programmer, who'd rather find bugs easily than save some typing, I think better than either would be "nothing by default, you gotta declare your variables", but that might not be popular for "dash off a quick script" usage; maybe make it configurable based on the load context or something?]
But this:
> The odd and slightly under-documented stack that you have to use when interfacing it with C
... seems almost bizarre. Lua's C interface is probably the most elegant and usable I've ever seen. A stack is the classic abstraction to use for this application, Lua's implementation is well-designed, and Lua's API documentation is both comprehensive and very well written.
It's annoying building a class system by hand. I've got C functions I wrote to do it for me, but I wish I didn't have to.
There are certain behaviors I dislike even more. One example is a historical quirk in which all non-nil or non-false expressions are true. This means inconsistent behavior like:
while 0 do print("Loops forever") end while not 1 do print("Does nothing") end
and:
while 1 do print("Loops forever") end while not 0 do print("Does nothing") end
I also don't like how undefined variables return nil. This can cause a host of subtle errors, including accidentally creating sparse arrays if you make a typo in a variable name, which will mess up the behavior of the length operator.
The syntax, which is what most people complain about in Lua, really doesn't bother me. It's the sometimes unconventional behavior of the language and its minimal feature set.
There are certain behaviors I dislike even more. One example is a historical quirk in which all non-nil or non-false expressions are true. This means inconsistent behavior like:
while 0 do print("Loops forever") end while not 1 do print("Does nothing") end
and:
while 1 do print("Loops forever") end while not 0 do print("Does nothing") end
I also don't like how undefined variables return nil. This can cause a host of subtle errors, including accidentally creating sparse arrays if you make a typo in a variable name, which will mess up the behavior of the length operator.
The syntax, which is what most people complain about in Lua, really doesn't bother me. It's the sometimes unconventional behavior of the language and its minimal feature set.
Hmm, I understand how 0 can evaluate to true (it's non-nil, non-false), but how is "not 1" false too, if 1 is true?
The only things that evaluate to false are nil and false, so applying the "not" operator to a true value (non-nil, non-false) will evaluate to false. Hence, "not 1" is false, as is "not 0".
Your link is borked. It should be
http://squirrel-lang.org
P.S. No criticism meant. Just a friendly correction. :-)
http://squirrel-lang.org
P.S. No criticism meant. Just a friendly correction. :-)
How does this really work, is it possible to write such stuff as drivers with a garbage collected language?
Yes.
There are operating systems written in system programming languages with GC. Check Native Oberon and A2 for desktop operating systems used at Zurich's university.
Smalltalk was originally developed as OS. Only the so called primitives were done in Assembly, with the remaining parts in Smalltalk itself.
Sun played around with Java in Solaris for device drivers and Squawk is a VM that runs directly on hardware with device drivers written in Java, as an example.
There are operating systems written in system programming languages with GC. Check Native Oberon and A2 for desktop operating systems used at Zurich's university.
Smalltalk was originally developed as OS. Only the so called primitives were done in Assembly, with the remaining parts in Smalltalk itself.
Sun played around with Java in Solaris for device drivers and Squawk is a VM that runs directly on hardware with device drivers written in Java, as an example.
Also OpenMirage, written in OCaml straight to the "hardware" (where "hardware" is a Xen virtual machine).
http://openmirage.org/
Could anyone explain, why Lua is so special?
It's very small, fast, and uses little memory. It's easy to learn and easy to interop with C. It's got two great implementations, with one (LuaJIT) probably holding the top spot among all dynamic, garbage-collected language currently in existence, measured by performance (although the creator -- `mikep` on HN -- is very careful to avoid making this sort of claim).
It's also got one of the coolest GUI toolkits around in IUP. If you've ever wondered by a cross-platform GUI toolkit that uses native widgets has to be so huge and complex, IUP's what you're looking for. Unfortunately, IUP doesn't yet have a native OSX driver, but you can use motif graphics driver to build a program that will run on OSX. Like Lua, IUP's in portable C, so a number of other languages have built bindings, notably go and perl. Like Lua, IUP has a very liberal license license (MIT), so you can use it for commercial projects without worrying about dynamic or static linking.
It's also got one of the coolest GUI toolkits around in IUP. If you've ever wondered by a cross-platform GUI toolkit that uses native widgets has to be so huge and complex, IUP's what you're looking for. Unfortunately, IUP doesn't yet have a native OSX driver, but you can use motif graphics driver to build a program that will run on OSX. Like Lua, IUP's in portable C, so a number of other languages have built bindings, notably go and perl. Like Lua, IUP has a very liberal license license (MIT), so you can use it for commercial projects without worrying about dynamic or static linking.
Correction: LuaJIT's creator Mike Pall is http://news.ycombinator.com/user?id=mikemike .
Thanks for the correction. I should have verified the HN username for Mike; not sure where I pulled `mikep` out of since I always enjoy reading his comments here and elsewhere. It's too late for me to correct it in my original comment, so thanks again.
He's mikemike on reddit as well.
IUP seems cool in theory, but in practice, I've never even been able to compile/install it, despite trying multiple times over the years! Unfortunately the build system seems to be mostly of the "Works great on the author's system!" sort... ><
I always try a bit to hack around the issues, but the problems have always exceeded my "oh this is too annoying, I'll try again next year" threshold (part of the problem being that it's an unusual build system, not just a few Makefiles, so it's less familiar).
I always try a bit to hack around the issues, but the problems have always exceeded my "oh this is too annoying, I'll try again next year" threshold (part of the problem being that it's an unusual build system, not just a few Makefiles, so it's less familiar).
I just noticed your comment, so sorry for the belated reply. I just wanted to mention that the tecgraf, makers of IUP, has thankfully simplified their build system -- no more custom build system. Also, they've started releasing pre-built binaries for all major OSs, including Linux and OSX. So now it's as easy and downloading a file and running sudo ./install. And if you want to build it yourself, it's as simple as running a few make files (make; make install each for the IUP, CD, and IM libraries).
But absolutely, the first time I tried to use IUP a few years ago, I couldn't get it build due to the complexity of the build system. Fortunately, that's all changed now.
But absolutely, the first time I tried to use IUP a few years ago, I couldn't get it build due to the complexity of the build system. Fortunately, that's all changed now.
This is a tweaked/updated version of a stackoverflow answer I wrote (see end for link) to someone wanting to know what language they should use for embedding:
Lua, and Lua's standard implementation(s), capture a wonderful balance of features/smallness/speed/usability, particularly for embedded use.
Lua:
1. Is very small, both source and binary, an order of magnitude or more smaller than many more popular languages (Python etc). Because the Lua source code is so small and simple, it's perfectly reasonable to just include the entire Lua implementation in your source tree, if you want to avoid adding an external dependency.
2. Is very fast. The Lua interpreter is much faster than most scripting languages (again, an order of magnitude is not uncommon), and LuaJIT2 is a very good JIT compiler for some popular CPU architectures (x86, arm, mips, ppc). Using LuaJIT can often speed things up by another order of magnitude, and in many cases, the result approaches the speed of C. LuaJIT is also a "drop-in" replacement for standard Lua: no application or user code changes are required to use it.
3. Has LPEG. LPEG is a "Parsing Expression Grammar" library for Lua, which allows very easy, powerful, and fast parsing, suitable for both large and small tasks; it's a great replacement for yacc/lex/hairy-regexps. [I wrote a parser using LPEG and LuaJIT, which is much faster than the yacc/lex parser I was trying emulate, and was very easy and straight-forward to create.] LPEG is an add-on package for Lua, but is well-worth getting (it's one source file).
4. Has a great C-interface, which makes it a pleasure to call Lua from C, or call C from Lua. For interfacing large/complex C++ libraries, one can use SWIG, or any one of a number of interface generators (one can also just use Lua's simple C interface with C++ of course).
5. Has liberal licensing ("BSD-like"), which means Lua can be embedded in proprietary projects if you wish, and is GPL-compatible for FOSS projects.
6. Is very, very elegant. It's not lisp, in that it's not based around cons-cells, but it shows clear influences from languages like scheme, with a straight-forward and attractive syntax. Like scheme (at least in it's earlier incarnations), it tends towards "minimal" but does a good job of balancing that with usability. For somebody with a lisp background (like me!), a lot about Lua will seem familiar, and "make sense", despite the differences.
7. Is very flexible, and such features as metatables allow easily integrating domain-specific types and operations.
8. Has a simple, attractive, and approachable syntax, which can be important if you intend to have end-users write scripts.
9. Is designed for embedding, and besides its small size and fast speed, has various features such as an incremental GC that make using a scripting language more viable in such contexts.
10. Has a long history, and responsible and professional developers, who have shown good judgment in how they've evolved the language over the last 2 decades.
11. Has a vibrant and friendly user-community.
Stackoverflow link: http://stackoverflow.com/questions/7919763/lua-vs-embedded-l...
Lua, and Lua's standard implementation(s), capture a wonderful balance of features/smallness/speed/usability, particularly for embedded use.
Lua:
1. Is very small, both source and binary, an order of magnitude or more smaller than many more popular languages (Python etc). Because the Lua source code is so small and simple, it's perfectly reasonable to just include the entire Lua implementation in your source tree, if you want to avoid adding an external dependency.
2. Is very fast. The Lua interpreter is much faster than most scripting languages (again, an order of magnitude is not uncommon), and LuaJIT2 is a very good JIT compiler for some popular CPU architectures (x86, arm, mips, ppc). Using LuaJIT can often speed things up by another order of magnitude, and in many cases, the result approaches the speed of C. LuaJIT is also a "drop-in" replacement for standard Lua: no application or user code changes are required to use it.
3. Has LPEG. LPEG is a "Parsing Expression Grammar" library for Lua, which allows very easy, powerful, and fast parsing, suitable for both large and small tasks; it's a great replacement for yacc/lex/hairy-regexps. [I wrote a parser using LPEG and LuaJIT, which is much faster than the yacc/lex parser I was trying emulate, and was very easy and straight-forward to create.] LPEG is an add-on package for Lua, but is well-worth getting (it's one source file).
4. Has a great C-interface, which makes it a pleasure to call Lua from C, or call C from Lua. For interfacing large/complex C++ libraries, one can use SWIG, or any one of a number of interface generators (one can also just use Lua's simple C interface with C++ of course).
5. Has liberal licensing ("BSD-like"), which means Lua can be embedded in proprietary projects if you wish, and is GPL-compatible for FOSS projects.
6. Is very, very elegant. It's not lisp, in that it's not based around cons-cells, but it shows clear influences from languages like scheme, with a straight-forward and attractive syntax. Like scheme (at least in it's earlier incarnations), it tends towards "minimal" but does a good job of balancing that with usability. For somebody with a lisp background (like me!), a lot about Lua will seem familiar, and "make sense", despite the differences.
7. Is very flexible, and such features as metatables allow easily integrating domain-specific types and operations.
8. Has a simple, attractive, and approachable syntax, which can be important if you intend to have end-users write scripts.
9. Is designed for embedding, and besides its small size and fast speed, has various features such as an incremental GC that make using a scripting language more viable in such contexts.
10. Has a long history, and responsible and professional developers, who have shown good judgment in how they've evolved the language over the last 2 decades.
11. Has a vibrant and friendly user-community.
Stackoverflow link: http://stackoverflow.com/questions/7919763/lua-vs-embedded-l...
Let me add my favorite hobby horse to your point 4: LuaJIT's C FFI is a tremendous and undervalued advantage compared to any other language/implementation I've used. Its core value proposition:
0) It can parse C header files and automatically construct bindings.
1) You can write Lua code that works directly on C data types with C memory layout and is JIT compiled. It makes it very easy to achieve performance within ~2x of C while still maintaining the expressiveness and extensibility of Lua. And if you miss writing OO C++, you can easily add methods to your C structs and they will be JIT compiled. See here http://wiki.luajit.org/Allocation-Sinking-Optimization and search for "Point Class with FFI cdata Struct" and then "Point Class Benchmarks".
2) Because the memory layout matches C data types, it is trivial to swap out a Lua implementation of a core operation with a C implementation if that becomes necessary.
3) It is easy to write cross-platform systems code without conditional compilation because the native calls through FFI can all be constructed dynamically through Lua (so instead of #ifdef WIN32 all over your socket code, you just have a single line checking for ffi.os == "windows" and then if necessary call the winsock initializer as ffi.C.WSAStartup(foo,bar)).
0) It can parse C header files and automatically construct bindings.
1) You can write Lua code that works directly on C data types with C memory layout and is JIT compiled. It makes it very easy to achieve performance within ~2x of C while still maintaining the expressiveness and extensibility of Lua. And if you miss writing OO C++, you can easily add methods to your C structs and they will be JIT compiled. See here http://wiki.luajit.org/Allocation-Sinking-Optimization and search for "Point Class with FFI cdata Struct" and then "Point Class Benchmarks".
2) Because the memory layout matches C data types, it is trivial to swap out a Lua implementation of a core operation with a C implementation if that becomes necessary.
3) It is easy to write cross-platform systems code without conditional compilation because the native calls through FFI can all be constructed dynamically through Lua (so instead of #ifdef WIN32 all over your socket code, you just have a single line checking for ffi.os == "windows" and then if necessary call the winsock initializer as ffi.C.WSAStartup(foo,bar)).
> LuaJIT's C FFI is ...
Indeed LuaJIT's FFI is a very cool and usable, but it does one huge drawback: it basically ties you to LuaJIT.
I'm not trying to diss LuaJIT—it's a wonderful and impressive JIT implementation, and for some applications, irreplaceable—but compared to the standard Lua implementation, LuaJIT is (1) less portable, (2) more complicated, and (3) has various implementation drawbacks.† In addition, of course, there are other alternate Lua implementations which have their own set of tradeoffs.
Essentially, it's very nice to be able to switch Lua implementations depending on the circumstance. With normal usage, LuaJIT is carefully designed to make this possible (indeed, easy: one need only change the library one links against, it's not even necessary to recompile one's app), but once you start using things like FFI, that ability is lost.
There's a port of LuaJIT's FFI interface to standard Lua, but it's also not portable, and because it will have very different performance characteristics than FFI in LuaJIT (which can "compile out" FFI accesses), it's not really a drop-in replacement for it, and I think not nearly as compelling.
† The one that's bitten me in the past is that LuaJIT has a much smaller limit on addressable memory than standard Lua because of the particular design of its nan-encoded object representation. There are some input files for my Lua-using app that will only work when I compile with standard Lua, because using LuaJIT will run out of memory; those are times when I'm very glad I didn't commit to a LuaJIT-only design...
Indeed LuaJIT's FFI is a very cool and usable, but it does one huge drawback: it basically ties you to LuaJIT.
I'm not trying to diss LuaJIT—it's a wonderful and impressive JIT implementation, and for some applications, irreplaceable—but compared to the standard Lua implementation, LuaJIT is (1) less portable, (2) more complicated, and (3) has various implementation drawbacks.† In addition, of course, there are other alternate Lua implementations which have their own set of tradeoffs.
Essentially, it's very nice to be able to switch Lua implementations depending on the circumstance. With normal usage, LuaJIT is carefully designed to make this possible (indeed, easy: one need only change the library one links against, it's not even necessary to recompile one's app), but once you start using things like FFI, that ability is lost.
There's a port of LuaJIT's FFI interface to standard Lua, but it's also not portable, and because it will have very different performance characteristics than FFI in LuaJIT (which can "compile out" FFI accesses), it's not really a drop-in replacement for it, and I think not nearly as compelling.
† The one that's bitten me in the past is that LuaJIT has a much smaller limit on addressable memory than standard Lua because of the particular design of its nan-encoded object representation. There are some input files for my Lua-using app that will only work when I compile with standard Lua, because using LuaJIT will run out of memory; those are times when I'm very glad I didn't commit to a LuaJIT-only design...
Small, fast, and easily embeddable in C programs. It's been popular in videogames for years but hasn't seen much usage outside that industry, so it's always interesting to see it used in different ways.
It's not the language itself. It's its implementation. Embedding a Lua interpreter in a C program is straightforward. See the 'Python' and 'Java' slides in the OP: it's all about integration.
Take JavaScript, remove all the wonky stuff, and now you've got Lua. Then ask yourself, why has JavaScript become popular outside the browser?
Javascript is popular outside the browser not because it's a great programming language, but because it gets us back to where we were before the browser became the dominant app delivery platform: it's now again possible to write your entire client-server app in one language.
If you're talking Node.js, sure, but that was a stepping stone. It's now being used for all sorts of weird things, like WinRT, and as GNOME's flagship desktop app language. These need not have any server-side components.
A large part is obviously that JavaScript is now well known and many developers have been exposed to it, but also because a minimal prototype language with first class functions is often all you need or want, just like Lua. (I assume this has been realized after exposure to JavaScript, or maybe I'm giving JS devs too much credit and they're not fans of minimal and clean languages.)
A large part is obviously that JavaScript is now well known and many developers have been exposed to it, but also because a minimal prototype language with first class functions is often all you need or want, just like Lua. (I assume this has been realized after exposure to JavaScript, or maybe I'm giving JS devs too much credit and they're not fans of minimal and clean languages.)
JavaScript is this generations Perl.
It's very small and uses very little memory. It's designed to be embedded. And has a liberal license (MIT).
Similar to this SnabbSwitch is a virtual Ethernet switch for Linux written mostly in Lua (using LuaJIT).
https://github.com/SnabbCo/snabbswitch/wiki
https://github.com/SnabbCo/snabbswitch/wiki
Oh my god, I may have fallen in love with that PowerPoint theme...
I think it's one of the standard themes for Beamer (LaTeX package for creating presentations):
http://www.hartwork.org/beamer-theme-matrix/
http://www.hartwork.org/beamer-theme-matrix/
will the backward compatibility of Lua be a matter?
He said he considered java (there have been attempts to write drivers in java) and python. Python didn't make it because of high ram usage (java wasn't any better). He didn't look into ruby, but "not for any particular reason". Probably because embedding ruby is quite hard compared to python or lua.
Very interesting talk!