A unikernel experiment: A VM for every URL(skjegstad.com)
skjegstad.com
A unikernel experiment: A VM for every URL
http://www.skjegstad.com/blog/2015/03/25/mirageos-vm-per-url-experiment/
14 comments
Note that both of the setups are quite different in how they boot unikernels. The EoX Zerg demo boots one unikernel per connection by having an HTTP proxy in the frontend spin up the VM.
Magnus' version builds a static set of unikernels whose total resource usage is known ahead of time, and boots them based on DNS lookups as necessary. Because there is a race between a DNS response, the first TCP SYN from the client, and the VM booting up to receive it, there is a helper unikernel called "Synjitsu" which terminates TCP connections and replays them to the MirageOS unikernel as soon as it boots up. This is very easy due to the OCaml TCP/IP stack in Synjitsu sharing its state via serialising it into an s-expression. More gory details in http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
Now, imagine a version of Synjitsu that did this handover with TLS connections rather than just TCP. The recipient unikernel wouldn't ever even have to know the private key, just the ephemeral session key...
Magnus' version builds a static set of unikernels whose total resource usage is known ahead of time, and boots them based on DNS lookups as necessary. Because there is a race between a DNS response, the first TCP SYN from the client, and the VM booting up to receive it, there is a helper unikernel called "Synjitsu" which terminates TCP connections and replays them to the MirageOS unikernel as soon as it boots up. This is very easy due to the OCaml TCP/IP stack in Synjitsu sharing its state via serialising it into an s-expression. More gory details in http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
Now, imagine a version of Synjitsu that did this handover with TLS connections rather than just TCP. The recipient unikernel wouldn't ever even have to know the private key, just the ephemeral session key...
That is just a demo to show startup times. Presumably (just by looking at it from a high level) a proxy instance for handling initial connection + a pre-allocated, or maybe booted but paused, set of worker images would work in either architecture.
I suppose work like this is demonstrating that, contrary to what e.g. the creators of Docker say, a virtual machine image is not too heavyweight to be our industry's equivalent of a shipping container. What is too heavyweight is a VM image of a conventional operating system distribution with all of the baggage that it brings.
Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)? Or is Xen uniquely suited to this kind of thing?
Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)? Or is Xen uniquely suited to this kind of thing?
> "What is too heavyweight is a VM image of a conventional operating system distribution with all of the baggage that it brings."
Indeed. I didn't realise that Docker/Container folks had ever said VMs were too heavyweight. Is this linked anywhere?
> "Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)?"
We've been asked this before and for KVM at least, it just boils down to someone suitably motivated to write the boot and VirtIO code [1, 2]. If anyone's interested in this, we'd love to hear about it [3]!
[1] http://lists.xenproject.org/archives/html/mirageos-devel/201...
[2] http://lists.xenproject.org/archives/html/mirageos-devel/201...
[3] http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageo...
Indeed. I didn't realise that Docker/Container folks had ever said VMs were too heavyweight. Is this linked anywhere?
> "Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)?"
We've been asked this before and for KVM at least, it just boils down to someone suitably motivated to write the boot and VirtIO code [1, 2]. If anyone's interested in this, we'd love to hear about it [3]!
[1] http://lists.xenproject.org/archives/html/mirageos-devel/201...
[2] http://lists.xenproject.org/archives/html/mirageos-devel/201...
[3] http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageo...
> Indeed. I didn't realise that Docker/Container folks had ever said VMs were too heavyweight. Is this linked anywhere?
https://github.com/docker/docker#user-content-better-than-vm...
https://github.com/docker/docker#user-content-better-than-vm...
Thanks. In hindsight that should have been an obvious place for me to look.
The next step is to compile it all to JS and just serve static files that boot in the browser! :) Something like http://bellard.org/jslinux/
I've always thought that something really cool could be done with unikernel for SaaS: you let each of your customer have a VM on their own, spinning the unikernel up when the requests came in, and let it process any request come in, after a while, if the specific user stop requesting then you kill the VM. There seems to be a lot of benefits from that: single user authentication, horizontally scaling is simple ("vertical" scaling also get easier as you can just move VM away), app design will easily let it become an on premise installation. Code upgrade is also easier as it won't disturb everyone at once, and you can upgrade when the VM is not running for a specific user (no hotswap needed). There is definitely operational overhead in a DNS server, but besides that it seems like the benefits would outweight the cons.
I know that's what DabbleDB was doing back then (in Squeak Smalltalk, each of their users has a smalltalk VM serving them). Is there anyone else doing anything similar nowadays?
I know that's what DabbleDB was doing back then (in Squeak Smalltalk, each of their users has a smalltalk VM serving them). Is there anyone else doing anything similar nowadays?
That sort of design is what motivated Magnus' post that has been linked here. We've been laying the uniSaaS groundwork with a new Xen toolstack daemon called Jitsu ("Just in Time Summoning of Unikernels").
Here's the camera ready paper that we'll be presenting at USENIX NSDI in May: http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
There's a lot more besides just the spawning to build a robust SaaS of course, but you can see the direction we're heading in...
Here's the camera ready paper that we'll be presenting at USENIX NSDI in May: http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
There's a lot more besides just the spawning to build a robust SaaS of course, but you can see the direction we're heading in...
I think the obstacle is that most folk are not prepared to give up their languages for this, so containerising PaaSes are still likely to dominate the scene.
Disclaimer: I worked on Cloud Foundry.
Disclaimer: I worked on Cloud Foundry.
> I think the obstacle is that most folk are not prepared to give up their languages
s/languages/frameworks, and I agree. One step at a time -- this work is a foundational block that we're building on, not the ultimate solution.
s/languages/frameworks, and I agree. One step at a time -- this work is a foundational block that we're building on, not the ultimate solution.
Hm. Got me thinking about how to fit unikernels into the CF architecture, which is largely taking containers (Garden or Docker, take your pick) as the unit of currency at the low level.
It looks like the place to make the change would be in Diego in the Executor[0][1] component, which currently implements behaviour in terms of containers (using Linux-Garden as the container system).
So long as you can conceptually fit into the Receptor Actions API[2] it looks as though unikernels would be a plausible addition to Executor. Once you have that, you can either wire into Lattice[3] or a full Cloud Foundry installation as a first-class citizen.
Disclaimer: I'm not as familiar with Diego as other CFers are, so it's possible I'm just making stuff up.
[0] https://github.com/cloudfoundry-incubator/executor
[1] https://github.com/cloudfoundry-incubator/diego-design-notes
[2] https://github.com/cloudfoundry-incubator/receptor/blob/mast...
[3] http://lattice.cf/
It looks like the place to make the change would be in Diego in the Executor[0][1] component, which currently implements behaviour in terms of containers (using Linux-Garden as the container system).
So long as you can conceptually fit into the Receptor Actions API[2] it looks as though unikernels would be a plausible addition to Executor. Once you have that, you can either wire into Lattice[3] or a full Cloud Foundry installation as a first-class citizen.
Disclaimer: I'm not as familiar with Diego as other CFers are, so it's possible I'm just making stuff up.
[0] https://github.com/cloudfoundry-incubator/executor
[1] https://github.com/cloudfoundry-incubator/diego-design-notes
[2] https://github.com/cloudfoundry-incubator/receptor/blob/mast...
[3] http://lattice.cf/
Thanks, this are all really helpful pointers! It's particularly useful to find Lattice, which seems to be a simpler entry point than a full CF setup.
The shiny website was new to me. I wonder if we've officially announced it yet?
In any case, we started the Lattice effort last year to make it easier for people to play with Diego without having to stand up a whole Cloud Foundry installation.
My only Lattice-related claim to fame is not knowing what is was: https://github.com/pivotal-cf-experimental/lattice-app/issue...
In any case, we started the Lattice effort last year to make it easier for people to play with Diego without having to stand up a whole Cloud Foundry installation.
My only Lattice-related claim to fame is not knowing what is was: https://github.com/pivotal-cf-experimental/lattice-app/issue...
I did something similar during my PhD. Maybe you want to have a look: http://se.inf.tu-dresden.de/pubs/papers/knauth2014dreamserve...
It was published at SYSTOR last year (2014).
It was published at SYSTOR last year (2014).
Erlang on Xen is probably closest to what you're asking: http://erlangonxen.org/
The fun thing about this is that you can try it out locally and apparently it took (about) an afternoon to put together (most of the work is in rewriting URLs).
This kind of fast experimentation is going to lead to some really cool things happening with unikernels.
This kind of fast experimentation is going to lead to some really cool things happening with unikernels.
I seem to remember something similar that was an erlang VM running (I think) directly on Xen. The underlying idea in that case was each request causes a new VM to boot, service that request, then terminate.
Mentioned in another top-level comment: http://erlangonxen.org/
This is an interesting experiment in moving the computation to the endpoints of a namespace. I would suggest a variant where instead of a VM you place a Fossil file
https://www.fossil-scm.org/index.html/doc/trunk/www/fileform...
Using that you can store a whole filesystem at the URL instead of one file, and you also get synchronization mechanisms built in, so your endpoints can communicate with each other in a structured way that you control.. vs the undefined communication you will get between VM's if you try this with the VM architecture.
https://www.fossil-scm.org/index.html/doc/trunk/www/fileform...
Using that you can store a whole filesystem at the URL instead of one file, and you also get synchronization mechanisms built in, so your endpoints can communicate with each other in a structured way that you control.. vs the undefined communication you will get between VM's if you try this with the VM architecture.
I'm not sure what the applications/market for this is or could be, but I love the idea. Everything that goes "micro" I love dearly.
I'm no DNS or security expert, but couldn't one application be increased security of connections? E.g. it's harder to do that 0day kernel hack with a buffer overflow if every URL request is sandboxed in a VM?
I probably got it wrong but I'd love to hear a more expert opinion on why/why not this could be used for security..
I probably got it wrong but I'd love to hear a more expert opinion on why/why not this could be used for security..
" I have not been able to find a hosting provider that has a minimum configuration which would be appropriate for large or small unikernels (e.g. 8MB vs 64MB RAM)." This is the biggest problem with unikernels right now.
Looks like an arbitrage opportunity! Buy 64MB server, sell 8MB servers.
Similar to ZeroVM [0], where a VM is spawned and destroyed for each individual request.
[0] http://www.zerovm.org/
[0] http://www.zerovm.org/
This is something I've been dreaming of. It is lovely to see it coming into the world. Death to the website! Long live the UX in a container!
This is better than a "container." A Mirage unikernel literally compiles in the pieces of an "OS" as libraries. There is no monolithic operating system with code sitting around waiting to be exploited. The web server in this example is the kernel.
When do we upgrade from IPv6?
http://erlangonxen.org/
Site is running on such a VM as well, press escape for a console to see stats.
Here is an example output from running their demo:
---
300 sec : is how long it takes to launch a Linux instance to availability on Amazon EC2.
50 sec : is the time between power on and the lock screen for an Android phone.
0.4 sec : ago is when we received your request. Within this time we managed to create a new Xen instance, boot it, and run the application that rendered the page you are viewing. By the time you are done reading this, the instance will be gone.
---