Docker, Mesos, Marathon, and the End of Pets(blog.factual.com)
blog.factual.com
Docker, Mesos, Marathon, and the End of Pets
http://blog.factual.com/docker-mesos-marathon-and-the-end-of-pets
35 comments
> Mesos (sic) is just over kill for everything.
That's definitely an opinion. :)
I have seen Mesosphere deployed with great success.
Insofar as state, this is one reason I'm not crazy about CoreOS - I feel more comfortable containerizing the application tier than the data tier, though both are certainly possible.
I'm really not eager to replace a highly tuned MySQL or Postgres machine with a container environment experiencing several levels of abstraction and redirection. I get frustrated enough trying to align partitions with block boundaries through RAID controllers.
But if you have 20 front-end app servers and 5 machines that run cron jobs, container services can help you to utilize your capacity much better. I can't say how many times I've worked somewhere that we desperately needed capacity, but didn't have the budget to expand until we cleaned up a bunch of machines that were vastly underutilized.
Anyway, Mesosphere isn't perfect, I have only even used it moderately, but there's a lot of tooling out there which we can use.
Def agree on the wierd theology of fleet, but also generally that it just doesn't do enough for me. It's way too much fucking trouble to say, "Run an http proxy on each physical machine".
That's definitely an opinion. :)
I have seen Mesosphere deployed with great success.
Insofar as state, this is one reason I'm not crazy about CoreOS - I feel more comfortable containerizing the application tier than the data tier, though both are certainly possible.
I'm really not eager to replace a highly tuned MySQL or Postgres machine with a container environment experiencing several levels of abstraction and redirection. I get frustrated enough trying to align partitions with block boundaries through RAID controllers.
But if you have 20 front-end app servers and 5 machines that run cron jobs, container services can help you to utilize your capacity much better. I can't say how many times I've worked somewhere that we desperately needed capacity, but didn't have the budget to expand until we cleaned up a bunch of machines that were vastly underutilized.
Anyway, Mesosphere isn't perfect, I have only even used it moderately, but there's a lot of tooling out there which we can use.
Def agree on the wierd theology of fleet, but also generally that it just doesn't do enough for me. It's way too much fucking trouble to say, "Run an http proxy on each physical machine".
The thing I don't get about the storage issue is this: Does volume mounting into a Docker container impact performance meaningfully at all? No? Can't you just punt on the migratable storage question until then? You'd not be any worse off than you are and doing upgrades to the db engine would be pretty easy, right? I dunno, maybe there's something I'm missing, block storage, what are you gonna do?
Docker is great also to hide complexity during implementation. Discourse.org is doing a great work "enveloping" their complex rails app in containers to easy the install process. And is not stateless.
"hide complexity"
You mean hide ugly sprawling uninstallable messes of rube goldberg code? :)
I'm referring to things that require PHP+MySQL, Node, Redis, and an old JVM process running Struts/Spring all managed by nginx except for that one situation where Apache2 .htaccess semantics are required for rewrite rules in which case it runs Apache proxied by nginx.
You mean hide ugly sprawling uninstallable messes of rube goldberg code? :)
I'm referring to things that require PHP+MySQL, Node, Redis, and an old JVM process running Struts/Spring all managed by nginx except for that one situation where Apache2 .htaccess semantics are required for rewrite rules in which case it runs Apache proxied by nginx.
By shipping only in a Docker container you will limit the audience for your app. That is the reason we ship GitLab as Omnibus packages (deb/rpm).
While deb/rpm packages are better than a monolithic container, having used one of your rpms, I'd say they're only just barely better.
What didn't you like about the Omnibus rpms?
Omnibus packages are a horrible idea. They are security issues waiting to happen.
A great example is the Chef server rpm. It is a 500mb mini distribution in one package. It has copies of perl, python, Ruby, and Erlang in it. If any of these has a security vulnerability, I have to wait on the maintainer to release a new version, and hope it included the security fixes.
They also tend to include things like python header files for no reason. You wouldn't compile against an Omnibus package, but they are there anyway. Examples of this are Sumologic's and Datadog's agents.
A great example is the Chef server rpm. It is a 500mb mini distribution in one package. It has copies of perl, python, Ruby, and Erlang in it. If any of these has a security vulnerability, I have to wait on the maintainer to release a new version, and hope it included the security fixes.
They also tend to include things like python header files for no reason. You wouldn't compile against an Omnibus package, but they are there anyway. Examples of this are Sumologic's and Datadog's agents.
We are aware that we'll have to patch any security issues and have done so reliably. I agree it is not ideal and we'll always be slower than the distribution packages. On the other hand the installation is much faster to perform (2 minutes instead of 10 pages of copy pasting) and we're able to ship with very secure settings for the integration points (sockets, etc.). But we recognize that some people will prefer native packages and are sponsoring work to make native Debian packages.
I disagree that they're doing great work, to me it looks like they've basically painted themselves in a corner by including too many dependencies, and the only way to get out of this corner was Docker.
The result is that their forum software requires another operating system to run. Had they been more disciplined in their development approach, Docker would have been merely a convenient way to test Discourse, and not the only supported option.
The result is that their forum software requires another operating system to run. Had they been more disciplined in their development approach, Docker would have been merely a convenient way to test Discourse, and not the only supported option.
There are some surprising places where you can enforce state where it seems impossible. Once you have that, the benefits coalesce.
That's why I built ShutIt, which we've used to encapsulate complex legacy environments to produce stateless builds:
http://ianmiell.github.io/shutit/
For example, teams can have a development environment (with _everything_ in it) rebuilt daily. As everyone uses it, everyone curates it, and they're all talking about the same thing - one pet if you like, rather than n, where n is the number of developer/development envs.
That's why I built ShutIt, which we've used to encapsulate complex legacy environments to produce stateless builds:
http://ianmiell.github.io/shutit/
For example, teams can have a development environment (with _everything_ in it) rebuilt daily. As everyone uses it, everyone curates it, and they're all talking about the same thing - one pet if you like, rather than n, where n is the number of developer/development envs.
You might like Lattice[0], which is extracted from Cloud Foundry.
Basically, everyone is racing back to PaaSes. Heroku pioneered it and are still out there. Red Hat have OpenShift and are making noises about turning it into a Docker+Kubernetes thing in version 3. Cloud Foundry has been around for a few years now. There are other also-rans.
The thing is that apart from Heroku, you've not heard of installable PaaSes because they're being pitched to the Fortune 500s.
I've worked on Cloud Foundry and I work for the company which donates the most effort to the Foundation. It's been surreal to watch other people introduce pieces of a PaaS and see the excitement about the pieces. Meanwhile, we literally have an entire turnkey system already. If you need a full PaaS -- push your app or service and have it running in seconds, with health management, centralised logging, auto-placement, service injection, the works -- we built it already. Free and opensource, owned by an independent Cloud Foundry Foundation.
Anyhow, I'm obviously biased, YMMV etc etc. But I'd play with Lattice, to get the hang of things.
[0] http://lattice.cf/
Basically, everyone is racing back to PaaSes. Heroku pioneered it and are still out there. Red Hat have OpenShift and are making noises about turning it into a Docker+Kubernetes thing in version 3. Cloud Foundry has been around for a few years now. There are other also-rans.
The thing is that apart from Heroku, you've not heard of installable PaaSes because they're being pitched to the Fortune 500s.
I've worked on Cloud Foundry and I work for the company which donates the most effort to the Foundation. It's been surreal to watch other people introduce pieces of a PaaS and see the excitement about the pieces. Meanwhile, we literally have an entire turnkey system already. If you need a full PaaS -- push your app or service and have it running in seconds, with health management, centralised logging, auto-placement, service injection, the works -- we built it already. Free and opensource, owned by an independent Cloud Foundry Foundation.
Anyhow, I'm obviously biased, YMMV etc etc. But I'd play with Lattice, to get the hang of things.
[0] http://lattice.cf/
"Red Hat have OpenShift and are making noises about turning it into a Docker+Kubernetes thing in version 3"
They've already decided that, and have even reached code-freeze. Their conference is later this month, so that's when it's going to all be announced and rollout plans detailed.
https://github.com/openshift/origin
They've already decided that, and have even reached code-freeze. Their conference is later this month, so that's when it's going to all be announced and rollout plans detailed.
https://github.com/openshift/origin
I remember reading the design docs last year. Interesting times ahead.
The real problem is going from tutorial to something you would use in production. Throw in logging, security and service discovery and you can have a few engineers hacking away for months.
So I want to plug a project I've been contributing to: https://github.com/CiscoCloud/microservices-infrastructure
We're trying to make it super easy to deploy these tools. For example every time you launch a docker container, it will register with consul and be added to haproxy. The nice thing about using Mesos is we can support data like workloads Cassandra, HDFS, and Kafka on the same cluster your run Docker images on.
We use terraform to deploy to multiple clouds so you don't get locked in to something like cloudformation.
So I want to plug a project I've been contributing to: https://github.com/CiscoCloud/microservices-infrastructure
We're trying to make it super easy to deploy these tools. For example every time you launch a docker container, it will register with consul and be added to haproxy. The nice thing about using Mesos is we can support data like workloads Cassandra, HDFS, and Kafka on the same cluster your run Docker images on.
We use terraform to deploy to multiple clouds so you don't get locked in to something like cloudformation.
This is basically why Kubernetes exists: for all the plumbing, discovery, etc required on top of bare containers.
It still requires work to go from zero to production-quality stack, of course.
It still requires work to go from zero to production-quality stack, of course.
We like Kubernetes (and are looking to add it to our project), but our goal is to integrate building blocks that allows us to run many different types of workloads. Think of our project more like an Ubuntu for distributed systems.
Kubernetes may eventually spread out beyond Docker, but for today we need to support things like Kafka and Spark.
As others have noted, we've had things like CloudFoundry, OpenShift and Heroku, and these all-in-one frameworks tend not to extend outside their original domain.
Kubernetes may eventually spread out beyond Docker, but for today we need to support things like Kafka and Spark.
As others have noted, we've had things like CloudFoundry, OpenShift and Heroku, and these all-in-one frameworks tend not to extend outside their original domain.
You should look at Cloud Foundry again, particularly with the introduction of Lattice. It used to be tied to apps, now it basically thinks about tasks and processes in a completely generic way.
The Circle CI post http://blog.circleci.com/its-the-future/ reads as a parody on this one.
Wow. That was hilarious! So much truth in that.
Is anyone running Marathon in production? Real production. The kind where any downtime means lost money.
I see a lot of intro-level tutorials, but almost nothing on the more advanced side.
My (completely casual) experience with Marathon is pretty bad, with the main process crashing quite regularly even under no load, so I'm wondering if people who write about these systems have actually used them for non-trivial tasks. And for something as critical as Marathon, which is supposed to handle... well... all my services, I'd rather be sure that the system is rock solid.
(This is specifically about Marathon. Mesos itself has proven more reliable)
I see a lot of intro-level tutorials, but almost nothing on the more advanced side.
My (completely casual) experience with Marathon is pretty bad, with the main process crashing quite regularly even under no load, so I'm wondering if people who write about these systems have actually used them for non-trivial tasks. And for something as critical as Marathon, which is supposed to handle... well... all my services, I'd rather be sure that the system is rock solid.
(This is specifically about Marathon. Mesos itself has proven more reliable)
I've been running Marathon in production (real production) to power more than 100 applications for the past six months. I chose it because it seemed like the most stable thing at the time; however, quickly found it was not production ready. While many of the original issues I encountered in 0.7.x were resolved with the 0.8.x release, 0.8.x brought new issues such as stuck deployments, etc. Additionally, I have found the upgrade path to be obtrusive and frankly scary. I am actively moving away from Marathon because of these issues.
Marathon does not make using Docker or building microservices simple. There are many important pieces that Marathon does not provide. Sure your operations team can tie in Mesos-DNS / Bamboo / Consul / whatever else, but it's going to take time, requires a specialized team, and leaves you feeling nervous about what happens if everything crashes in the middle of the night. Even when tying in these third party tools, it is likely you will have to make significant code updates to utilize features such as service-discovery / SRV records. You will inevitably end up with a hobbled-together system that needs serious support from your operations team.
I am fairly frustrated as a whole with Mesosphere, and expected more from a company who raised so much capital.
Marathon does not make using Docker or building microservices simple. There are many important pieces that Marathon does not provide. Sure your operations team can tie in Mesos-DNS / Bamboo / Consul / whatever else, but it's going to take time, requires a specialized team, and leaves you feeling nervous about what happens if everything crashes in the middle of the night. Even when tying in these third party tools, it is likely you will have to make significant code updates to utilize features such as service-discovery / SRV records. You will inevitably end up with a hobbled-together system that needs serious support from your operations team.
I am fairly frustrated as a whole with Mesosphere, and expected more from a company who raised so much capital.
I wouldn't expect Marathon to do service discovery for you, as I believe that is better left to something like Mesos-dns/Consul which marathon can supervise for you, and docker integration has been fairly simple.
In any case I found my marathon was not without issues, like failover causing every application to restart (I think this was fixed in 0.8.2), or the fact that marathon tends to use 2x as much RAM as Zookeeper or Mesos-Master (I run the 3 on the same node).
Have you seen the aurora apache project? It solves the same problems as marathon, and its creators claim it was built to handle stability. I originally chose marathon as JSON configuration over REST was easier to wrap my head around, but was this something you tried and how did it work for you?
In any case I found my marathon was not without issues, like failover causing every application to restart (I think this was fixed in 0.8.2), or the fact that marathon tends to use 2x as much RAM as Zookeeper or Mesos-Master (I run the 3 on the same node).
Have you seen the aurora apache project? It solves the same problems as marathon, and its creators claim it was built to handle stability. I originally chose marathon as JSON configuration over REST was easier to wrap my head around, but was this something you tried and how did it work for you?
Might I ask: If you're moving away from Marathon, where are you moving to? Kubernetes?
Running a complete hosted telephony service using nearly the exact stack defined in this article - https://developers.corvisa.com/
It has worked remarkably well and allowed us to scale up/down during peak hours or unexpected high traffic peaks.
It has worked remarkably well and allowed us to scale up/down during peak hours or unexpected high traffic peaks.
Yes, many people have in fact run it in "real production". Go ahead and Google my name for credibility.
There were indeed issues with the 0.7.x series of Marathon, but we've made a big effort to focus on stability and performance in 0.8.x, and onward. As with any new software project, there tend to be issues in early releases.
There were indeed issues with the 0.7.x series of Marathon, but we've made a big effort to focus on stability and performance in 0.8.x, and onward. As with any new software project, there tend to be issues in early releases.
Factual have done what lots of people do, which is invent the first 20% of a PaaS.
PaaSes are awesome. They also, once you go past the basics, require enormous engineering effort. And that's the problem: engineering effort spent on curating your own homegrown PaaS is engineering effort not available for creating user value.
5 years ago rolling your own was a source of competitive advantage. Today you can get an installable PaaS (Cloud Foundry or OpenShift) off the shelf and run it. In 2 years Docker, Mesos and CoreOS will probably all have PaaSes of their own.
Interesting times.
PaaSes are awesome. They also, once you go past the basics, require enormous engineering effort. And that's the problem: engineering effort spent on curating your own homegrown PaaS is engineering effort not available for creating user value.
5 years ago rolling your own was a source of competitive advantage. Today you can get an installable PaaS (Cloud Foundry or OpenShift) off the shelf and run it. In 2 years Docker, Mesos and CoreOS will probably all have PaaSes of their own.
Interesting times.
I know two companies running running openshift on Mesos. The reason modern Paas aren't enough, is the factat some scale you're not just running web services. But also rather complex data pipelines involving distributed data systems like kafka, spark, cassandra, etc .. and a simple paas has issues handling those workloads. That's where Mesos shines.
As for the user value, I consider having an uptime system, providing reliable and intelligent service based on data processing, by efficiently using your resources is has an important business value.
I find Mesos the most interesting because it's a kind of mirror image of Lattice[1], particularly Diego[2][2][3]. Both of them push intelligence out to the edges, but in different ways.
Diego pushes the scheduling problem out to the executors themselves through an auction mechanism. Mesos delegates it back to the requestors, as I understand it.
The way it was described to me by Onsi Fakhouri is that Diego is "demand-driven", and Mesos is "supply-driven". Diego grew from the lessons learnt on Cloud Foundry v2, so it favours fast placement of requests over perfect fit. But in the fact of the network fallacies, it probably turns out to be a good approach anyhow.
By way of warning, it has been a few months since I read the relevant papers and my memory is fuzzy.
Edit: and it looks as though the auction mechanism was moved away from. Hm.
[1] http://lattice.cf/
[2] https://github.com/cloudfoundry-incubator/diego-design-notes
[2] https://www.youtube.com/watch?v=1OkmVTFhfLY (an excellent overview)
[3] https://www.youtube.com/watch?v=SSxI9eonBVs (an excellent update)
Diego pushes the scheduling problem out to the executors themselves through an auction mechanism. Mesos delegates it back to the requestors, as I understand it.
The way it was described to me by Onsi Fakhouri is that Diego is "demand-driven", and Mesos is "supply-driven". Diego grew from the lessons learnt on Cloud Foundry v2, so it favours fast placement of requests over perfect fit. But in the fact of the network fallacies, it probably turns out to be a good approach anyhow.
By way of warning, it has been a few months since I read the relevant papers and my memory is fuzzy.
Edit: and it looks as though the auction mechanism was moved away from. Hm.
[1] http://lattice.cf/
[2] https://github.com/cloudfoundry-incubator/diego-design-notes
[2] https://www.youtube.com/watch?v=1OkmVTFhfLY (an excellent overview)
[3] https://www.youtube.com/watch?v=SSxI9eonBVs (an excellent update)
To be honest, I still have yet to see one of these systems that beats simply using Bash. They're all trying to make a scripting problem into a configuration problem. That's sometimes a reasonable idea, when what you're doing is common enough that only a few things here and there need to diverge from the defaults. But every image I've ever had to create contained far more edge cases than default cases, and half of those edge cases are things nobody thought of and therefore their system doesn't handle it. Rather than trying to fight with one of these systems to get them to do something I could easily do with a few Bash commands, I find it easier to just script the setup in Bash to begin with.
Obviously that doesn't work for Windows systems.
Obviously that doesn't work for Windows systems.
In that case you might want to look at ansible: it does exactly what such a system should (logs in with SSH and runs some scripts) but does in a smart way (eg. you can configure certain boxes to be web servers and have it run a script on all web server boxes). It does have some weird config format but it does also allow you to run scripts.
Quite curious about if whether we had more robust shell languages, like Inferno's es, which uses functional expressions, though it's some 20+ years old by now, if kept alive by some enthusiasts... would much of the rationale behind the more basic configuration management and provisioning go away?
The chief rationale for embarking on the CM route is idempotence. Though it seems to me having a basic, lexer-only language simply for chain loading commands to exec() without much of the state and environment baggage behind full command shells could work as an alternative. execline works like this: http://skarnet.org/software/execline/
The chief rationale for embarking on the CM route is idempotence. Though it seems to me having a basic, lexer-only language simply for chain loading commands to exec() without much of the state and environment baggage behind full command shells could work as an alternative. execline works like this: http://skarnet.org/software/execline/
I've used Ansible, and of the systems I've used to solve this and similar problems (Chef, Docker, Puppet) Ansible has been the cleanest, most well-thought-out, and easiest to use. But ultimately it's included in my review of the above tools. It misses 10% of cases instead of 50%, but I still end up spending an inordinate amount of time working around the system instead of running it.
This is poetic:
"Kubernetes has a Clintonesque inevitability to it"
Docker, its great if you have no state. But then if you have no state shit is easy. Mapping images to highspeed storage securely and reliably is genuinely hard. (unless you use NFSv4 and kerberos)
Mesos is just over kill for everything. How many people actually need shared image programs bigger than a 64 core machine with 512gigs of ram? (and now good are you at juggling NUMA or NUMA like interfaces)
I can't help thinking that what people would really like is just a nice easy to use distributed CPU scheduler. Fleet basically, just without the theology that comes with it.
Seriously, mainframes look super sexy right now. Easy resource management, highly scalable real UNIX. (no need to spin up/down, just spawn a new process)