Marelle: logic programming for devops(quietlyamused.org)
quietlyamused.org
Marelle: logic programming for devops
http://quietlyamused.org/blog/2013/11/09/marelle-for-devops/
8 comments
That's the main reason we've switched over from Puppet to Ansible. It has less abstraction layers, but at least its execution order is predictable. The easy learning curve for new colleagues and the compact syntax are also nice benefits.
That's what I don't understand about Ansible. It's basically -- literally, actually -- Puppet reinvented with a new syntax based on YAML.
Puppet works so well, Ansible seems like a typical "not invented here" tragedy that frequently occurs in open source. Rather than coalesce around something that works and has years of experience and knowledge and bug fixing sunk into it, people create something from scratch.
To be specific, this is Ansible:
Puppet has very little abstraction. But what you do get, which Ansible has not really understood, I think, is a data model. The above results in the graph getting an object File['/etc/memcached.conf'] which can be referred to. I can do this:
Likewise, the graph model is extensible. I can invent a new object like so:
Ansible may look simpler on the surface, but I think it's all surface. When you have learned to use Puppet, it becomes simple. Ansible looks to me like the product of a person who looked at Puppet, said something like "ugh Ruby, ew syntax" and made his own thing, without looking deeper. Simialrly, Ansible appeals to people who look at Puppet with the same reaction, again without looking deeper. The irony is that they are buying into something that is arguably poorer.
That said, I would argue that part of the problem is Puppet's documentation, which surely must drive people to "poor man's Puppet" solutions. (The cynic in me wonders if Puppet's poor documentation is a deliberate ploy on part of Puppet Labs to boost their consulting business.)
Puppet works so well, Ansible seems like a typical "not invented here" tragedy that frequently occurs in open source. Rather than coalesce around something that works and has years of experience and knowledge and bug fixing sunk into it, people create something from scratch.
To be specific, this is Ansible:
- name: Memcached
template:
src=$repository_basedir/memcached/templates/memcached.conf.j2
dest=/etc/memcached.conf
owner=root group=root mode=0644
notify:
- memcached-restart
file {
'/etc/memcached.conf':
content => template('memcached/memcached.conf'),
owner => root,
group => root,
mode => '0644',
notify => Service['memcached'];
}
Puppet has very little abstraction. But what you do get, which Ansible has not really understood, I think, is a data model. The above results in the graph getting an object File['/etc/memcached.conf'] which can be referred to. I can do this:
service {
'memcached':
require => File['/etc/memcached.conf'];
}
Stage[main]/Node[myserver]/Service['memcached']
Likewise, the graph model is extensible. I can invent a new object like so:
define public_key() {
file {
"/home/$name/.ssh/authorized_keys":
content => "$key";
}
}
public_key {
'someuser':
key => 'some public key';
}
account {
'someuser':
require => PublicKey['someuser'];
}
Ansible may look simpler on the surface, but I think it's all surface. When you have learned to use Puppet, it becomes simple. Ansible looks to me like the product of a person who looked at Puppet, said something like "ugh Ruby, ew syntax" and made his own thing, without looking deeper. Simialrly, Ansible appeals to people who look at Puppet with the same reaction, again without looking deeper. The irony is that they are buying into something that is arguably poorer.
That said, I would argue that part of the problem is Puppet's documentation, which surely must drive people to "poor man's Puppet" solutions. (The cynic in me wonders if Puppet's poor documentation is a deliberate ploy on part of Puppet Labs to boost their consulting business.)
With all due respect to lobster_johnson but I think that you are just in love and blinded by the complexity of Puppet.
I have used Puppet in production in my 2 previous jobs and once I found Ansible that was like why puppet could not be as simple as Ansible. With Ansible, you do not have to draw or understand any freaking graphs of dependency and the only logic is the order of tasks you list them in yaml files and the order of how you include yaml files in runbooks. I think that a first grade student will easily be able to understand the order of how tasks are going to be executed. In comparison to Puppet, in order to achieve the right order of executions you will need to write all these require and what not. At the end of the days, a lot of people are going to be so mad at themselves because puppet fails them in the best possible ways.
Don't dismiss something too quickly as poor man's puppet solution where Ansible users are laughing to the banks because we save tons of times with Ansible and finish our works in PREDICTABLE fashions 100% of the times with little sweat. You can go ahead and spend all your times with your fancy graphs of dependency and write puppet modules that you and only you will understand. We, Ansible users, will not face with such problems because a new or veteran Ansible user will be able to understand the exact same things as how the automation of our tasks will be.
And I do not know if puppet's documentation is the problem or not and I do not think that is the case. There have been plenty books and articles written about puppet and people should be able to read them and become better at that. I personally bought 2 books on Puppet, read ways to many articles written on the web about puppet and at the end of the day it is not only because of the DSL, dependency nightmares, complexity in getting a master/slave environment up but because puppet has tried to make something better but on the other hands make other things way more too complicated than needed.
Poor you my friend for only seeing one tree in a forest. I know that a lot of people are reluctant to change because they have invested heavily in the times, money, efforts in something like puppet and they simply do not want to give puppet up. It all boils down to whether you ask yourself if you could do something better and question the norms. It does not mean that something has been around for many years is the best thing and it does not mean because you use it I have to use it too. We all have choices and we all want better things than that shiny thing on the pedestal.
I have used Puppet in production in my 2 previous jobs and once I found Ansible that was like why puppet could not be as simple as Ansible. With Ansible, you do not have to draw or understand any freaking graphs of dependency and the only logic is the order of tasks you list them in yaml files and the order of how you include yaml files in runbooks. I think that a first grade student will easily be able to understand the order of how tasks are going to be executed. In comparison to Puppet, in order to achieve the right order of executions you will need to write all these require and what not. At the end of the days, a lot of people are going to be so mad at themselves because puppet fails them in the best possible ways.
Don't dismiss something too quickly as poor man's puppet solution where Ansible users are laughing to the banks because we save tons of times with Ansible and finish our works in PREDICTABLE fashions 100% of the times with little sweat. You can go ahead and spend all your times with your fancy graphs of dependency and write puppet modules that you and only you will understand. We, Ansible users, will not face with such problems because a new or veteran Ansible user will be able to understand the exact same things as how the automation of our tasks will be.
And I do not know if puppet's documentation is the problem or not and I do not think that is the case. There have been plenty books and articles written about puppet and people should be able to read them and become better at that. I personally bought 2 books on Puppet, read ways to many articles written on the web about puppet and at the end of the day it is not only because of the DSL, dependency nightmares, complexity in getting a master/slave environment up but because puppet has tried to make something better but on the other hands make other things way more too complicated than needed.
Poor you my friend for only seeing one tree in a forest. I know that a lot of people are reluctant to change because they have invested heavily in the times, money, efforts in something like puppet and they simply do not want to give puppet up. It all boils down to whether you ask yourself if you could do something better and question the norms. It does not mean that something has been around for many years is the best thing and it does not mean because you use it I have to use it too. We all have choices and we all want better things than that shiny thing on the pedestal.
That implicit vs explicit ordering becomes very important in bigger teams (or even when sharing deployment between multiple teams). Everything is fine when you're writing your bit - it works as you wanted. But then someone else takes over, or bugfixes something - and they can do one of two mistakes: silly one - assume some dependency where there is none and write some more-complicated-than-necessary workaround; or a very critical one - fail to spot a dependency and assume they can reorder some steps, or split them into separate modules.
I've seen this a lot of times when working with chef already. Someone moves a very trivial thing to a different place, or adds/moves a notification - current system works, clean deployment works, ... but current system will fail on the next reboot or package update that cascades in an unpredictable series of events. Unless you explicitly list all existing dependencies, the system is very fragile and sensitive to modifications. I agree that most people deploying a single-server webapp will never run into this, notice the problem, or even care about it. When deploying a multi-host, minimal-downtime infrastructure it becomes a problem though.
I can appreciate the simple chef/ansible way of provisioning things, but starting with some size I think that explicit dependencies are the only way to go. Puppet/salt allow me to do things that are quite hard to do in chef.
I've seen this a lot of times when working with chef already. Someone moves a very trivial thing to a different place, or adds/moves a notification - current system works, clean deployment works, ... but current system will fail on the next reboot or package update that cascades in an unpredictable series of events. Unless you explicitly list all existing dependencies, the system is very fragile and sensitive to modifications. I agree that most people deploying a single-server webapp will never run into this, notice the problem, or even care about it. When deploying a multi-host, minimal-downtime infrastructure it becomes a problem though.
I can appreciate the simple chef/ansible way of provisioning things, but starting with some size I think that explicit dependencies are the only way to go. Puppet/salt allow me to do things that are quite hard to do in chef.
So Ansible has the disadvantage of unclear, implicit dependencies.
The undeterministic declarative system (Puppet) has the disadvantage of unclear, missing dependencies.
I would prefer a deterministic system, so that I can validate the output by automated acceptance tests. Preferably, these tests also take into account the various upgrade paths (new->head and head^->head)
The undeterministic declarative system (Puppet) has the disadvantage of unclear, missing dependencies.
I would prefer a deterministic system, so that I can validate the output by automated acceptance tests. Preferably, these tests also take into account the various upgrade paths (new->head and head^->head)
> I know that a lot of people are reluctant to change because they have invested heavily in the times, money, efforts in something like puppet and they simply do not want to give puppet up
That's not me. Just recently I switched companies and needed to set up a new devops environment from scratch, and before starting I evaluated Ansible, Chef and Salt to determine whether they were superior to Puppet. I decided they were not.
I am not "in love" with Puppet. People who are in love with technology tend to make decisions for the wrong reasons. I have very little loyalty when it comes to choosing software, and will gladly replace any part of my stack if there is a better solution; but I greatly appreciate maturity and stability. Freshly minted, grandly-hyped technology is by definition immature.
I am truly tired of software being unnecessarily reinvented. The "not invented here" (or perhaps "not invented by me") treadmill where young, inexperienced developers throw away older, mature technology simply because it's not fresh and new is something I try to stay away from. Sometimes (Rails, Node.js, Nginx, Go) it brings something genuinely new to the stable, but generally it doesn't.
Lastly, I don't appreciate your condescending tone. And really, I'm not your friend.
That's not me. Just recently I switched companies and needed to set up a new devops environment from scratch, and before starting I evaluated Ansible, Chef and Salt to determine whether they were superior to Puppet. I decided they were not.
I am not "in love" with Puppet. People who are in love with technology tend to make decisions for the wrong reasons. I have very little loyalty when it comes to choosing software, and will gladly replace any part of my stack if there is a better solution; but I greatly appreciate maturity and stability. Freshly minted, grandly-hyped technology is by definition immature.
I am truly tired of software being unnecessarily reinvented. The "not invented here" (or perhaps "not invented by me") treadmill where young, inexperienced developers throw away older, mature technology simply because it's not fresh and new is something I try to stay away from. Sometimes (Rails, Node.js, Nginx, Go) it brings something genuinely new to the stable, but generally it doesn't.
Lastly, I don't appreciate your condescending tone. And really, I'm not your friend.
You do know where the author of Ansible came from right? ;-)
Ansible keeps the task and orchestration layer simple so you stay focused on the problem you're solving vs. focusing some complexity of the tool itself, like a hierarchical data model.
If you need complex logic, use the programming language of your choice and write a module, it's simple and your code can be as complex as you need.
If you need more complex data structures, then simply have Ansible query it from a real database or other datasource, with as complex a data structure as you need.
But at the layer you're expressing what needs to be done, it should be simple and easy to read.
Ansible keeps the task and orchestration layer simple so you stay focused on the problem you're solving vs. focusing some complexity of the tool itself, like a hierarchical data model.
If you need complex logic, use the programming language of your choice and write a module, it's simple and your code can be as complex as you need.
If you need more complex data structures, then simply have Ansible query it from a real database or other datasource, with as complex a data structure as you need.
But at the layer you're expressing what needs to be done, it should be simple and easy to read.
> Puppet works so well, Ansible seems like a typical "not invented here" tragedy that frequently occurs in open source.
That is a bit kind of what I get from Ansible.
Ok at first the best thing it had going for it was 'it works over plain SSH and uses YAML'. Basically for those who did everything via SSH and shell scripting. This was an upgrade.
Now once they get into complicated playbooks and transitive dependencies, 0mq daemons, yes it looks like it is just another Puppet but newer (some kids just like newer things because they want to feel like they are at the cutting edge of technology).
There is also Salt. That had the "look really fast and responsive configuration" because it has the 0mq based distribution mechanism. But then Ansible added that too as a feature. Salt looked at and said "ok fine, we'll add SSH only option too". So now they are both basically solving a similar problem along with the older tools (Puppet, Chef etc)
That is a bit kind of what I get from Ansible.
Ok at first the best thing it had going for it was 'it works over plain SSH and uses YAML'. Basically for those who did everything via SSH and shell scripting. This was an upgrade.
Now once they get into complicated playbooks and transitive dependencies, 0mq daemons, yes it looks like it is just another Puppet but newer (some kids just like newer things because they want to feel like they are at the cutting edge of technology).
There is also Salt. That had the "look really fast and responsive configuration" because it has the 0mq based distribution mechanism. But then Ansible added that too as a feature. Salt looked at and said "ok fine, we'll add SSH only option too". So now they are both basically solving a similar problem along with the older tools (Puppet, Chef etc)
Haven't used Salt much, as I understand it they do take Puppets model. Even if you want that (and I don't) there are still a lot of ways you could do a better job, so I'm somewhat open to Salt.
However, Ansible is significantly different from Puppet. For starters it is push-based rather than pull-based as a lot of Puppet setups are.
The other important difference is that Ansible has top-to-bottom ordering. That's what first attracted me.
However, Ansible is significantly different from Puppet. For starters it is push-based rather than pull-based as a lot of Puppet setups are.
The other important difference is that Ansible has top-to-bottom ordering. That's what first attracted me.
Push based SSH models fail pretty badly when you starting talking hundreds or thousands of servers under management. Those models have existed for a log time in tools like rdist, and sitescope tried to sell 'agentless' monitoring which required SSH in order to monitor servers. I've seen deployments of sitescope like that hit scalability walls at a few hundred servers as the ssh client processes on the sitescope server were chewing all the CPU doing SSH session negotiation. And the point here is that 'agentless' is really a lie. You're not agentless, you're using sshd as your agent. And when you dig into that protocol its expensive and not designed for RPC, its designed for interactive logins, and its chatty and chews up resources. It also isn't terribly reliable. It seems reliable when you're using it on your workstation and logging into a few dozen servers a day. You don't remember that time a few months ago when it had a 'derp' and you had to login again. When you're hitting 30,000 servers multiple times a day, you start to see the unreliability of the underlying protocol.
If Ansible is going to survive you're going to see some kind of 'persistent fireball mode' where you ssh into the box and leave the zeromq daemon running and it starts looking a whole lot more like push.
And really you want nodes registering back to your server. It makes discovery so much easier since you just configure your chef/puppet/cfengine/whatever server in whatever bootstrapping scheme you're using (kickstart, custom amis, whatever) and then whenever a server comes up, it registers and starts working. The push method where you have to edit a file on a server which is used to push code just leads to crufty servers getting untracked. The idea isn't really new, its been around for decades, and its always the first baby steps on the road to doing config management.
If Ansible is going to survive you're going to see some kind of 'persistent fireball mode' where you ssh into the box and leave the zeromq daemon running and it starts looking a whole lot more like push.
And really you want nodes registering back to your server. It makes discovery so much easier since you just configure your chef/puppet/cfengine/whatever server in whatever bootstrapping scheme you're using (kickstart, custom amis, whatever) and then whenever a server comes up, it registers and starts working. The push method where you have to edit a file on a server which is used to push code just leads to crufty servers getting untracked. The idea isn't really new, its been around for decades, and its always the first baby steps on the road to doing config management.
What Ansible really got that it sells is 1) It is new. Everyone likes a fresh new thing. 2) It suckers people in who just didn't know or wanted to care about configuration management. Tell them about Puppet and declarative syntax and their eyes start rolling. "But I just have 2 servers... I don't need declarative syntax". And, of course, they are doing configuration management, they just do it by hand, one step at a time, via SSH and shell scripting. Ansible comes in and says "we can make that a little better for you", at least that how I got to know it.
It is funny the Salt vs Ansible interplay. As people complained about each other, they implemented each others' feature in response. Salt got SSH mode and Ansible got fireball mode.
It is funny the Salt vs Ansible interplay. As people complained about each other, they implemented each others' feature in response. Salt got SSH mode and Ansible got fireball mode.
Rackspace recently mentioned having tens of thousands of hosts managed with Ansible actually, and it works fine over SSH. Here's some slides on issues they ran into and resolved: http://prezi.com/tyasc3-vj_id/ansible-turbo/
Though one of the core technology decisions behind Ansible is that you don't need much to get it running. You can scale things up as needed, such as fireball mode. And you can boot strap that process using Ansible. You don't even need DNS.
Hosts can register back with Ansible on start up, this works today with their AWX product and cloud-init's phone home feature. Also your inventory of hosts can simply be queried from sources, such as AWS's inventory list, so in that case you can't have a host that's unaccounted for.
Though one of the core technology decisions behind Ansible is that you don't need much to get it running. You can scale things up as needed, such as fireball mode. And you can boot strap that process using Ansible. You don't even need DNS.
Hosts can register back with Ansible on start up, this works today with their AWX product and cloud-init's phone home feature. Also your inventory of hosts can simply be queried from sources, such as AWS's inventory list, so in that case you can't have a host that's unaccounted for.
> If Ansible is going to survive you're going to see some kind of 'persistent fireball mode' where you ssh into the box and leave the zeromq daemon running and it starts looking a whole lot more like push.
It has this (the previous incarnation was even called "fireball mode"):
http://www.ansibleworks.com/docs/playbooks_acceleration.html
It has this (the previous incarnation was even called "fireball mode"):
http://www.ansibleworks.com/docs/playbooks_acceleration.html
Ansible looks to me like the product of a person who looked
at Puppet, said something like "ugh Ruby, ew syntax" and
made his own thing, without looking deeper.
I said looks, I was not being literal. I don't know anything about Mr. Michael Dehaan and I'm not going to speculate.
He was the product manager some years ago and very briefly.
I second everything said here.
In my internship last summer, I arbitrarily chose Puppet to automate the configuration of the company's developer environments. Puppet's unpredictable ordering should be a write-and-forget benefit, but in reality it's very difficult to reason about. In the end, my script imposed a top-to-bottom ordering with requires.
Also, developing configuration scripts against Vagrant is a huge pain. While `puppet parser validate` checks for syntax errors, most of the logical, dependency-based errors reveal themselves after Vagrant takes a good minute to fire up.
Anyway, hopefully my Puppet-Vagrant scripts aren't causing any maintenance nightmares...
In my internship last summer, I arbitrarily chose Puppet to automate the configuration of the company's developer environments. Puppet's unpredictable ordering should be a write-and-forget benefit, but in reality it's very difficult to reason about. In the end, my script imposed a top-to-bottom ordering with requires.
Also, developing configuration scripts against Vagrant is a huge pain. While `puppet parser validate` checks for syntax errors, most of the logical, dependency-based errors reveal themselves after Vagrant takes a good minute to fire up.
Anyway, hopefully my Puppet-Vagrant scripts aren't causing any maintenance nightmares...
Puppet has implicit ordering (for example, a file will depend on its parent directory) as well as explicit (arrow syntax, new in 3.0, I think).
But really, I don't understand the complaint. Writing explicit dependencies literally what Puppet -- or any other config management system -- is about. Puppet cannot possibly know what your intent is. So you have to tell it. Just like you have to tell the OP's system. It's simple enough.
Having used Puppet for a few years, my biggest complaint is that it's not quite modular enough. For example, in one system I have a way of declaring an "app". An app may have some dependencies (Postgres, Memcached), settings, users, etc. I have two choices, I think; either have one large, monolithic define allowing me to instantiate the app from a bunch of config:
Puppet is very modular, but it's a bit awkward when it comes to this level of modularity -- or I haven't found a better way.
But really, I don't understand the complaint. Writing explicit dependencies literally what Puppet -- or any other config management system -- is about. Puppet cannot possibly know what your intent is. So you have to tell it. Just like you have to tell the OP's system. It's simple enough.
Having used Puppet for a few years, my biggest complaint is that it's not quite modular enough. For example, in one system I have a way of declaring an "app". An app may have some dependencies (Postgres, Memcached), settings, users, etc. I have two choices, I think; either have one large, monolithic define allowing me to instantiate the app from a bunch of config:
apps::app {
"someapp":
environment => 'production',
uses_unicorn => true,
db_name => 'someapp_production',
db_user => 'someapp',
memcached_servers => [...],
# etc.
apps::app {
"someapp":
environment => 'production';
}
apps::memcached_config {
"someapp":
servers => [...];
}
apps::db_config {
"someapp":
host => ...,
db_name => ...,
db_user => ...;
}
apps::unicorn_config {
"someapp":
;
}
Puppet is very modular, but it's a bit awkward when it comes to this level of modularity -- or I haven't found a better way.
> Writing explicit dependencies literally what Puppet -- or any other config management system -- is about. Puppet cannot possibly know what your intent is. So you have to tell it.
Other tools just execute manifests top to bottom, which is a much less error-prone approach.
I agree that finding the right abstraction level for modules (and for reuse) is not well solved.
Other tools just execute manifests top to bottom, which is a much less error-prone approach.
I agree that finding the right abstraction level for modules (and for reuse) is not well solved.
I'm a puppet newbie, but I think that you might use Augeas rather than concat bits from conf.d.
If reasoning about explicit order of resources is too much for you, just enable the new resource ordering feature: http://docs.puppetlabs.com/references/latest/configuration.h...
Then assign classes like: class{'ntp':}-> class{'ssh':}-> class{'other_stuff':}
problem solved?
Then assign classes like: class{'ntp':}-> class{'ssh':}-> class{'other_stuff':}
problem solved?
Sure, except that once you're doing stuff like that, you might as well switch to a manifest-ordered tool like Ansible. Ordering isn't the only problem with Puppet, just the most serious.
What are the other problems with Puppet?
Puppet's unpredictable ordering should be a write-and-forget benefit, but in reality it's very difficult to reason about.
But this is Prolog and ordering in Prolog is deterministic. Clause goals are proven in order. When there are multiple clauses with the same predicate, they are executed in order. If this weren't the case extra-logical facilities such as cuts wouldn't work in Prolog
But this is Prolog and ordering in Prolog is deterministic. Clause goals are proven in order. When there are multiple clauses with the same predicate, they are executed in order. If this weren't the case extra-logical facilities such as cuts wouldn't work in Prolog
:D I said Puppet, not Prolog. Though I did learn Prolog for a programming languages class in the time between the end of my internship and today. The connection didn't appear to me until reading calpaterson's post.
I have to respectfully disagree. I try not to take sides in these issues because the software I make all works well and co-exists with Puppet, Chef, etc. But I feel I need to voice my opinion here.
I _love_ the explicit ordering of Puppet. It is one of my favorite features over other options. I'll explain why I think it is a good thing.
For credentials to add some sort of credibility to this post: I worked for 4 years as an ops person managing not small (but not big) infrastructures for multiple companies ranging from 10 nodes up to 200 nodes. Some organizations used Chef, some used Puppet, and I even used Salt once. I created and maintain Vagrant and Packer, both of which integrate deeply with all major configuration management systems, so I have to know how they work decently well to be able to maintain that.
Let's use Chef for the remainder of this post, but please don't take this post as saying "Chef is bad. Puppet is good." That is false. I'm just using Chef as an example where ordering is implicit (based on the order resources appear in a recipe and run list).
The explicit ordering of Puppet makes it very clear exactly what a resource depends on. When training junior ops, I was frequently asked "so I use requires/notify/etc. to tell Puppet what order to do things?" And I would say NO! That is the _result_, but the purpose of requires is to list what the resource has as a logical _dependency_. For example, a common problem with junior puppet devs is to say "Oh, I'm making resource C which depends on resource B and A, but since B depends on A, I'll just put 'B' in the requires list." That is _wrong_. Logically, C depends on B and A, so put both there! Don't think about ordering, think about requirements/dependencies.
Given this, Puppet is so great. You can look at any resource and say "oh hey, this depends on ONLY these list of things." Sure, the dependencies may have their own dependencies, but the resource in question doesn't care about that. Example: this file requires that this service is installed. That service might require, say, RubyGems, but the file doesn't care! Its not an immediate dependency of the file.
With Chef, on the other hand, I would see "okay, C happens after B and A, but does C _really_ need to happen after A?" I don't know. I can't answer that. It requires much deeper domain knowledge of that infrastructure to answer that question. This leads to more of a nightmare [for me] months down the road, when I don't quite remember if something needs to happen after something else.
With Puppet, as a style rule, I require all ops engineers list _all_ requirements (exhaustive require clauses) whenever they're working. Then, months down the road, you can move around anything you want. You know if you broke something because Puppet will tell you (before it runs, during the 'compilation' phase).
Personally, I prefer this sort of explicitness.
I understand that some don't like this, but I like it. And people can like it. I don't believe either way is STRICTLY better/worse. I felt my opinion needed to be shared.
I _love_ the explicit ordering of Puppet. It is one of my favorite features over other options. I'll explain why I think it is a good thing.
For credentials to add some sort of credibility to this post: I worked for 4 years as an ops person managing not small (but not big) infrastructures for multiple companies ranging from 10 nodes up to 200 nodes. Some organizations used Chef, some used Puppet, and I even used Salt once. I created and maintain Vagrant and Packer, both of which integrate deeply with all major configuration management systems, so I have to know how they work decently well to be able to maintain that.
Let's use Chef for the remainder of this post, but please don't take this post as saying "Chef is bad. Puppet is good." That is false. I'm just using Chef as an example where ordering is implicit (based on the order resources appear in a recipe and run list).
The explicit ordering of Puppet makes it very clear exactly what a resource depends on. When training junior ops, I was frequently asked "so I use requires/notify/etc. to tell Puppet what order to do things?" And I would say NO! That is the _result_, but the purpose of requires is to list what the resource has as a logical _dependency_. For example, a common problem with junior puppet devs is to say "Oh, I'm making resource C which depends on resource B and A, but since B depends on A, I'll just put 'B' in the requires list." That is _wrong_. Logically, C depends on B and A, so put both there! Don't think about ordering, think about requirements/dependencies.
Given this, Puppet is so great. You can look at any resource and say "oh hey, this depends on ONLY these list of things." Sure, the dependencies may have their own dependencies, but the resource in question doesn't care about that. Example: this file requires that this service is installed. That service might require, say, RubyGems, but the file doesn't care! Its not an immediate dependency of the file.
With Chef, on the other hand, I would see "okay, C happens after B and A, but does C _really_ need to happen after A?" I don't know. I can't answer that. It requires much deeper domain knowledge of that infrastructure to answer that question. This leads to more of a nightmare [for me] months down the road, when I don't quite remember if something needs to happen after something else.
With Puppet, as a style rule, I require all ops engineers list _all_ requirements (exhaustive require clauses) whenever they're working. Then, months down the road, you can move around anything you want. You know if you broke something because Puppet will tell you (before it runs, during the 'compilation' phase).
Personally, I prefer this sort of explicitness.
I understand that some don't like this, but I like it. And people can like it. I don't believe either way is STRICTLY better/worse. I felt my opinion needed to be shared.
This sounds like a question of how declarative your deps are, with Chef and Puppet picking different points on the scale.
I can relate that with Babushka now, we commonly rely on ordering of requirements (A <- B, C in that order!) at times when we should really have one of those requirements require the other (A <- B, B <- C). This works fine for the original purpose, but leaves minefields floating around when you want to reuse deps for another project.
I can relate that with Babushka now, we commonly rely on ordering of requirements (A <- B, C in that order!) at times when we should really have one of those requirements require the other (A <- B, B <- C). This works fine for the original purpose, but leaves minefields floating around when you want to reuse deps for another project.
Hm, part of my consulting agreement when I did Puppet was that I could re-use my own set of Puppet modules (which I never did open source). I shared the same set of Puppet modules among many clients and never had issues with "minefields" because of my explicit requirements.
Therefore, I think this comes down to how you design your Puppet modules.
Personally, I design my Puppet modules much in the way that Jamie Windsor designs Chef cookbooks and talks about in the ChefConf keynote: "The Berkshelf Way".
Therefore, I think this comes down to how you design your Puppet modules.
Personally, I design my Puppet modules much in the way that Jamie Windsor designs Chef cookbooks and talks about in the ChefConf keynote: "The Berkshelf Way".
One nit to pick here is that I'd say that Chef is 'explicit' about ordering (I explicitly say exactly what is going to happen), while Puppet I would call 'implicit' based on solving a dependency graph and a bit of random hash order evaluation.
This is actually one of the founding principles of Chef in that Adam Jacobs got tired of, in large systems, when you get one link in your dependency tree wrong you wind up with code which will pass tests and get deployed to prod then accidentally get broken the next day as the random ordering changes.
Also, Chef does have dependencies between cookbooks and we do depsolve at that level. You can even construct things so that your base role that does most of your user account creation, ntp, dns, mail config, etc -- which is most of the time orthogonal to your app -- does not need to be a dependency of your application cookbooks. If you're deploying a rails app you can make that cookbook depend mostly just on installing ruby and deploying from github and you can test that, tightly, in isolation on a vagrant VM with test-kitchen. You can also setup more complicated integration scenarios where your nginx load balancer cookbooks don't depend on your apps, and your apps don't depend on your nginx config, but you can have integration tests for your nginx loadbalancers that brings up both in a virt and makes sure that you can load balance your app -- again, though, without sucking in every person's ssh keys in the org to set it up.
And even though you're expressing dependencies, you're doing so in a array with an ordering to it, so that the dependencies reliably get solved in the correct direction, so that even if you don't specify the deps right, if you get the ordering correct it all works.
And for credentials, I started using CFEngine in 1999, I did most of the "base role" of Amazon's CM infrastructure from 2001-2006 (growing from doing config pushes to 400 servers in my department to 30,000 servers across the company), then I built a few more small CFEngine deployments of 1,000-3,000 servers before moving on to working at Opscode.
And my background with CFEngine -- which really invented the Prolog-style-Config-Mgmt that this whole article is about some 20+ years ago. And with the frameworks that I built on CFEngine I always imposed ordering so that you could read scriptlets from top to bottom and see what happened and then composed those scriptlets in runs that would execute in order. When we started out not doing that it was hugely confusing to newbies since actions would be taken in a seemingly random order in the file and that is not a natural way for people who walk into config management -- so we went out of our way to break the dependency-solving that CFEngine did and create something that ran almost exactly like Chef does (technically even more extreme, since there was no dependency resolution between "cookbooks" and while there wasn't an explicit run_list the order that files were included simply determined the order of execution).
This is actually one of the founding principles of Chef in that Adam Jacobs got tired of, in large systems, when you get one link in your dependency tree wrong you wind up with code which will pass tests and get deployed to prod then accidentally get broken the next day as the random ordering changes.
Also, Chef does have dependencies between cookbooks and we do depsolve at that level. You can even construct things so that your base role that does most of your user account creation, ntp, dns, mail config, etc -- which is most of the time orthogonal to your app -- does not need to be a dependency of your application cookbooks. If you're deploying a rails app you can make that cookbook depend mostly just on installing ruby and deploying from github and you can test that, tightly, in isolation on a vagrant VM with test-kitchen. You can also setup more complicated integration scenarios where your nginx load balancer cookbooks don't depend on your apps, and your apps don't depend on your nginx config, but you can have integration tests for your nginx loadbalancers that brings up both in a virt and makes sure that you can load balance your app -- again, though, without sucking in every person's ssh keys in the org to set it up.
And even though you're expressing dependencies, you're doing so in a array with an ordering to it, so that the dependencies reliably get solved in the correct direction, so that even if you don't specify the deps right, if you get the ordering correct it all works.
And for credentials, I started using CFEngine in 1999, I did most of the "base role" of Amazon's CM infrastructure from 2001-2006 (growing from doing config pushes to 400 servers in my department to 30,000 servers across the company), then I built a few more small CFEngine deployments of 1,000-3,000 servers before moving on to working at Opscode.
And my background with CFEngine -- which really invented the Prolog-style-Config-Mgmt that this whole article is about some 20+ years ago. And with the frameworks that I built on CFEngine I always imposed ordering so that you could read scriptlets from top to bottom and see what happened and then composed those scriptlets in runs that would execute in order. When we started out not doing that it was hugely confusing to newbies since actions would be taken in a seemingly random order in the file and that is not a natural way for people who walk into config management -- so we went out of our way to break the dependency-solving that CFEngine did and create something that ran almost exactly like Chef does (technically even more extreme, since there was no dependency resolution between "cookbooks" and while there wasn't an explicit run_list the order that files were included simply determined the order of execution).
Even in "non-deterministic" style prolog, the execution order is still fixed, and dependencies are traversed in the order you wrote them. There will be no surprises.
The missing full-stop is no different to a missing semi-colon, comma, or backslash, issues we still have in many modern languages.
The missing full-stop is no different to a missing semi-colon, comma, or backslash, issues we still have in many modern languages.
> The missing full-stop is no different to a missing semi-colon
Yeah, it's an annoyance. The difference is that the feedback loop on operations codebases is longer, so they're more annoying.
Yeah, it's an annoyance. The difference is that the feedback loop on operations codebases is longer, so they're more annoying.
For Puppet you can use a validation script and a git commit hook to shorten the feedback loop on syntax errors: http://spoop.net/devops/2013/11/12/validate-puppet.html
It's been a while since I've studied Prolog, but it's good to see it getting some press. I learned Prolog before ever dabbling in Scheme or ML, and it was just as much of a paradigm shift from imperative style. It's kind of like FP-style pattern matching, but taken to a whole new level, such that the pattern matching itself is pretty much the entire computational engine.
Rather than using functions, which take a number of arguments and return a result, you use predicates on a number arguments. You build your program out of clauses, which are conjunctions of predicates. A process called unification [1] finds the argument values (if any) that satisfy an entire clause. By sharing constants and variables between predicates, the valid results that unification can produce are restricted to what the logic of your program allows.
So your algorithms end up being, in many ways, description of what a valid results look like, and the Prolog engine goes out and finds them. This paradigm clearly maps nicely to domains with lots of rules, like decision engines, parsers, and such. Sounds awesome, right? One caveat is that, in practice, you as a programmer have to have a pretty good understanding of how unification works in order to write performant algorithms.
This is all from several years-old recollections and a little bit of Wikipedia to refresh my memory, so someone please chime in if I'm incorrect someplace or leaving out key concepts.
[1] http://en.wikipedia.org/wiki/Unification_%28computer_science...
Rather than using functions, which take a number of arguments and return a result, you use predicates on a number arguments. You build your program out of clauses, which are conjunctions of predicates. A process called unification [1] finds the argument values (if any) that satisfy an entire clause. By sharing constants and variables between predicates, the valid results that unification can produce are restricted to what the logic of your program allows.
So your algorithms end up being, in many ways, description of what a valid results look like, and the Prolog engine goes out and finds them. This paradigm clearly maps nicely to domains with lots of rules, like decision engines, parsers, and such. Sounds awesome, right? One caveat is that, in practice, you as a programmer have to have a pretty good understanding of how unification works in order to write performant algorithms.
This is all from several years-old recollections and a little bit of Wikipedia to refresh my memory, so someone please chime in if I'm incorrect someplace or leaving out key concepts.
[1] http://en.wikipedia.org/wiki/Unification_%28computer_science...
Prolog's a surprisingly good fit for many every-day tasks where people otherwise invent DSLs or poorly-defined state machines and pattern matching algorithms.
So how come we don't use it more often? Well, the reason is actually quite simple:
Prolog's incredibly difficult to reason about once you grow past trivial facts and clauses like in the OP's article -- the examples given by him would've been introduced to you in the first lesson on Prolog in, say, University. The complexities of green/red cuts, backtracking and the procedural nature of Prolog makes it a really difficult language to truly learn -- much less one to specify complex programs in.
I've never programmed in a language where 5-6 lines of errant Prolog code could stump myself, a post grad and a professor before until I did Prolog.
So how come we don't use it more often? Well, the reason is actually quite simple:
Prolog's incredibly difficult to reason about once you grow past trivial facts and clauses like in the OP's article -- the examples given by him would've been introduced to you in the first lesson on Prolog in, say, University. The complexities of green/red cuts, backtracking and the procedural nature of Prolog makes it a really difficult language to truly learn -- much less one to specify complex programs in.
I've never programmed in a language where 5-6 lines of errant Prolog code could stump myself, a post grad and a professor before until I did Prolog.
So I'm not experienced with Prolog, but the thing that scared me off is that it doesn't appear to have any computational complexity guarantees. It seems like will scale very poorly as your data gets bigger. It's just a notation for exhaustive search as far as I know, and you have to add ad hoc cuts outside the declarative model to get reasonable performance.
I have experience with other abstractions which are supposed to be "declarative", but you really have to peek under the hood at the implementation to get them to work in production settings, and I don't like that.
I have experience with other abstractions which are supposed to be "declarative", but you really have to peek under the hood at the implementation to get them to work in production settings, and I don't like that.
It is a shame really. Declarative programming is so much simpler to think about and extremely powerful. I've been working on trying to make declarative programming more accessible and while I personally have seen 60x improvements in ability, I have been unable to convince any of my teammates to give it a whirl.
Also like the bootstrap.sh - very nicely structured and easier to verify than many others I've seen. I'd still prefer these scripts to output missing dependencies that calling sudo -- but otherwise very nice.
That is, rather than:
That is, rather than:
function install_git() {
echo 'Trying to install git'
case $(uname -s) in
#...
Linux)
if has_exec apt-get; then apt_update
sudo apt-get install -y git
#...
else bail "Unknown linux variant"
fi
;;
*)
bail "Unknown operating system $(uname -s)"
;;
esac
}
echo Please install missing packages:
echo sudo apt-get install ${missing-deps}
Thanks! :)
It's a little aggressive in calling sudo mainly because I'm using it to bootstrap new systems non-interactively. I don't want it to ask any questions at all, just to go.
It's a little aggressive in calling sudo mainly because I'm using it to bootstrap new systems non-interactively. I don't want it to ask any questions at all, just to go.
Prolog does seem a reasonable fit for the problem space.
That said, even CFEngine's long winded syntax (born of its commitment to "promise theory") isn't so bad. In my experience I don't feel expressiveness of the available languages is the problem most I need of solving.
The real problems not being tackled by the existing configuration management tools are testing bundles/(play|cook)books/modules, cleanly injecting metadata to facilitate reuse and the bundle debugging workflow.
That said, even CFEngine's long winded syntax (born of its commitment to "promise theory") isn't so bad. In my experience I don't feel expressiveness of the available languages is the problem most I need of solving.
The real problems not being tackled by the existing configuration management tools are testing bundles/(play|cook)books/modules, cleanly injecting metadata to facilitate reuse and the bundle debugging workflow.
What kind of metadata would facilitate reuse?
Have you seen the latest verbose mode in CFEngine (as of 3.5, June 2013)? It's more compact to facilitate debugging.
How else could debugging workflow be improved?
Have you seen the latest verbose mode in CFEngine (as of 3.5, June 2013)? It's more compact to facilitate debugging.
How else could debugging workflow be improved?
Say I have a simple bundle for inserting lines in /etc/security/limits.conf, it'd be good to have the bundle agent / edit_line which describes how to make the change, completely separate from the data that describes user1 gets a hard nofile of 64k.
There's lots of ways to achieve that from just doing selective usebundles from various bundle files containing only vars to more flexible approaches; In our deployment we've provided this as part of our bundle framework (we inject the parameter data via modules protocol - so a box in location A applies different data than a box under line of business B), they both reuse the exact same (comprehensivly tested) code.
This means no code release for a data change. No need to train everyone in CFE syntax just to allow an app developer to express that user c on box d is allowed more open file handles.
The point though is that It'd be good to have as a first class language feature instead of each cfe site reinventing a separate framework for the same functionality.
Haven't had any hands on with 3.5 so not familiar with that, but better print debugging isnt where I was going with the idea. I want to be able to step through a bundle, I want to be able to break when global class X is set. I want to then be able to set class X while stepping through. Kinda like making debug logging level interative and with higher signal to noise ratio on screen.
If there was language support for unit testing my bundles I could use that to reason about what I expect to happen. Diego Zambonis book proposed one way to do that with the existing facilities but its just a proof of concept, e.g. it cant guarantee environment cleanup after a show stopping error in a test.
There's lots of ways to achieve that from just doing selective usebundles from various bundle files containing only vars to more flexible approaches; In our deployment we've provided this as part of our bundle framework (we inject the parameter data via modules protocol - so a box in location A applies different data than a box under line of business B), they both reuse the exact same (comprehensivly tested) code.
This means no code release for a data change. No need to train everyone in CFE syntax just to allow an app developer to express that user c on box d is allowed more open file handles.
The point though is that It'd be good to have as a first class language feature instead of each cfe site reinventing a separate framework for the same functionality.
Haven't had any hands on with 3.5 so not familiar with that, but better print debugging isnt where I was going with the idea. I want to be able to step through a bundle, I want to be able to break when global class X is set. I want to then be able to set class X while stepping through. Kinda like making debug logging level interative and with higher signal to noise ratio on screen.
If there was language support for unit testing my bundles I could use that to reason about what I expect to happen. Diego Zambonis book proposed one way to do that with the existing facilities but its just a proof of concept, e.g. it cant guarantee environment cleanup after a show stopping error in a test.
The meet/met notation instantly made me think of babushka: http://babushka.me/
The base principle seems to be the same except that babushka is a ruby DSL and that the dependency solving is part of the library instead of the language.
The base principle seems to be the same except that babushka is a ruby DSL and that the dependency solving is part of the library instead of the language.
Babushka seems to be one of the inspirations. The article says "Still, Babushka made me uneasy: all this ceremony and complex templating, just to describe a few facts and simple rules?"
Very interesting. I wonder if it would be viable to build an entire package management system (like apt) in a similar manner -- and if there are any (semi-recent) improvements to prolog-like systems that might improve dependency resolution "for free"?
These are the package management systems that I use. A new package management systems just does this http://xkcd.com/927/ What we need is a cross-platform cross-applicaton meta package management system which comes as configuration management that evolves to a general purpose package manager without relying on any external package manager.
I'm not fanboy enough and would happily replace:
emerge, equo, apt-get, aptitude, ipkg, dpkg, zypp rpm, yum, tazpkg, pacman, homebrew
tar/gzip, slackpkg (Yeah, you, Slackware, we were friends and you helped me for long enough. You shouldn't do all my work anymore, you're old. Take a rest)
pip, cabal, bundler, gem, composer, pear, cpan, ctan, cran, maven, vundle – haven't you also a love hate relationship with these?
I'm not fanboy enough and would happily replace:
emerge, equo, apt-get, aptitude, ipkg, dpkg, zypp rpm, yum, tazpkg, pacman, homebrew
tar/gzip, slackpkg (Yeah, you, Slackware, we were friends and you helped me for long enough. You shouldn't do all my work anymore, you're old. Take a rest)
pip, cabal, bundler, gem, composer, pear, cpan, ctan, cran, maven, vundle – haven't you also a love hate relationship with these?
[deleted]
The secret of configuration management is that ordering NEEDS to be a first class feature and ordered manifests should be the default, not the option. Puppet suffers hugely from the influence from the fact that ordering is unpredictable and that you have to use arrows or requires to add order. People accidentally forget to add ordering, the resulting code works most of the time, and then one day you start running into an issue because some implicit dependency either didn't happen, or happened in the wrong order. In as much as all logic programming involves not explicitly controlling flow, I think they're inappropriate for configuration management. Sadly, I'm still stuck with using Puppet because of the organisation I work in.
Having worked with Prolog in the past, I would also suggest that an important feature of a configuration management language is that the syntax make it hard to make typos and other minor errors because the feedback loop of deploying your code against a local vagrant machine is already long enough. I vividly remember the annoyance of having missed the full stop to end a block. I'm not desperate to repeat that experience.