Testing with Jenkins, Ansible and Docker(blog.mist.io)
blog.mist.io
Testing with Jenkins, Ansible and Docker
http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with
5 comments
This limit is now higher(somewhere in the range of 100), and will eventually go away. Once this limit is gone, best practices will almost surely be one command per run.
> Once this limit is gone, best practices will almost surely be one command per run.
Agreed, I thought the same, but then I hit the 127 (I think) limit.
Agreed, I thought the same, but then I hit the 127 (I think) limit.
In my humble opinion, each layer should have a clear purpose:
http://www.extellisys.com/articles/docker-layer-overload
You make good points in this post. We are working on addressing some of them already. I would love to discuss this directly, you can ping me anytime, shykes on #docker / freenode.
At the same time, we need to move fast with development and deliver updates as soon as possible. We want to be able to easily deploy several times per day.
Genuine question, not intended as any sort of troll: what benefits would people with this philosophy say their organisation gains from routinely deploying multiple times per day?
I have nothing against better testing tools or more efficient development processes, of course, and if you have a serious bug then being able to fix it as quickly as possible is obviously beneficial. I just don't understand where this recent emphasis on always trying to move fast has come from, or what kind of management strategy someone might use to take advantage of such agility.
Genuine question, not intended as any sort of troll: what benefits would people with this philosophy say their organisation gains from routinely deploying multiple times per day?
I have nothing against better testing tools or more efficient development processes, of course, and if you have a serious bug then being able to fix it as quickly as possible is obviously beneficial. I just don't understand where this recent emphasis on always trying to move fast has come from, or what kind of management strategy someone might use to take advantage of such agility.
I found this article helpful in understanding the 'multiple times per day' motive.
http://martinfowler.com/articles/continuousIntegration.html
http://martinfowler.com/articles/continuousIntegration.html
The motivation for continuous integration I can certainly understand. I'm sure many of us have worked on projects where someone was working on a branch for an extended period to implement some big new feature, and then ran into the mother of all merge conflicts when they finally wanted to release their changes for general use. I don't think CI is appropriate in all cases, but I find it a better strategy than mega-merges for most projects, most of the time.
It's specifically the motivation for continuous deployment that I'm curious about here. Fixing bugs quickly is obviously a plus, and I found the point 'twic mentioned about rapidly instrumenting a production system interesting. These are good arguments for being able to release an update on demand, and indeed for continuous integration as a technique to support that goal.
However, to me neither seems like a strong argument for routinely releasing multiple updates per day. I've always been a little surprised that so many web sites and SaaS applications seem to take pride in making such frequent changes, even though doing so surely incurs some risk of breaking things and typically also causes some degree of instability in the UI. I was just wondering whether anyone had experiences and/or actual data they could share about whether the practice made a measurable improvement to, for example, revenues or reported customer satisfaction -- something concrete that might outweigh the risks -- or whether it's more of a subjective preference for that way of working.
It's specifically the motivation for continuous deployment that I'm curious about here. Fixing bugs quickly is obviously a plus, and I found the point 'twic mentioned about rapidly instrumenting a production system interesting. These are good arguments for being able to release an update on demand, and indeed for continuous integration as a technique to support that goal.
However, to me neither seems like a strong argument for routinely releasing multiple updates per day. I've always been a little surprised that so many web sites and SaaS applications seem to take pride in making such frequent changes, even though doing so surely incurs some risk of breaking things and typically also causes some degree of instability in the UI. I was just wondering whether anyone had experiences and/or actual data they could share about whether the practice made a measurable improvement to, for example, revenues or reported customer satisfaction -- something concrete that might outweigh the risks -- or whether it's more of a subjective preference for that way of working.
> However, to me neither seems like a strong argument for routinely releasing multiple updates per day.
I don't understand this line of thinking. If there's a bug that you can fix in short order, why not fix it and deploy said fix? If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.
I've always been a big proponent of small, atomic commits. Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens), and we can respond to feedback sometimes within minutes.
The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
I don't understand this line of thinking. If there's a bug that you can fix in short order, why not fix it and deploy said fix? If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.
I've always been a big proponent of small, atomic commits. Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens), and we can respond to feedback sometimes within minutes.
The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
If there's a bug that you can fix in short order, why not fix it and deploy said fix?
If you've got an important bug to fix, sure, of course you want it dealt with as quickly as possible.
If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.
Not if it's for a bug fix or security patch, but presumably most of your development work is adding new features? In that case, isn't the whole point that it's something the user will notice?
I've always been a big proponent of small, atomic commits.
And again, no argument from me there. While I don't think every development project is suitable for that approach, I favour it most of the time myself.
Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens)
This is where my experience and background seem to be very different to the deploy-hourly kind of philosophy. I've noticed that proponents of the latter often seem to assume that stuff is going to be breaking all the time, and therefore that being able to deploy bug fixes or diagnostics very quickly will be a significant advantage.
However, if you have so many significant bugs that you need to deploy multiple times per day just to keep up, to me that seems like a clear demonstration of insufficient QC/QA in the development process. And so I can't help wondering whether the pressure to release so often, and the consequent almost exclusive reliance on some sort of automated test suite for quality checking, is a causal factor in having so many bugs in the first place.
The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
I guess this is the paradoxical part that I don't understand. If your project enjoys excellent stability and your deployments are all soundly tested before they go live, then by definition you don't need the rapid deployments just to rush out bug fixes or diagnostics.
So that brings me back to where I came in: do you experience any concrete, measurable benefits from the "excellent development velocity" you mentioned, or is it simply your team's preferred development style?
If you've got an important bug to fix, sure, of course you want it dealt with as quickly as possible.
If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.
Not if it's for a bug fix or security patch, but presumably most of your development work is adding new features? In that case, isn't the whole point that it's something the user will notice?
I've always been a big proponent of small, atomic commits.
And again, no argument from me there. While I don't think every development project is suitable for that approach, I favour it most of the time myself.
Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens)
This is where my experience and background seem to be very different to the deploy-hourly kind of philosophy. I've noticed that proponents of the latter often seem to assume that stuff is going to be breaking all the time, and therefore that being able to deploy bug fixes or diagnostics very quickly will be a significant advantage.
However, if you have so many significant bugs that you need to deploy multiple times per day just to keep up, to me that seems like a clear demonstration of insufficient QC/QA in the development process. And so I can't help wondering whether the pressure to release so often, and the consequent almost exclusive reliance on some sort of automated test suite for quality checking, is a causal factor in having so many bugs in the first place.
The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
I guess this is the paradoxical part that I don't understand. If your project enjoys excellent stability and your deployments are all soundly tested before they go live, then by definition you don't need the rapid deployments just to rush out bug fixes or diagnostics.
So that brings me back to where I came in: do you experience any concrete, measurable benefits from the "excellent development velocity" you mentioned, or is it simply your team's preferred development style?
> So that brings me back to where I came in: do you experience any concrete, measurable benefits from the "excellent development velocity" you mentioned, or is it simply your team's preferred development style?
Yes, see the following:
> The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
We're not just deploying bug fixes quickly (which you seem to be fixating on a bit), we are constantly making small iterations. If we're handling support and see something that could be improved, we improve it and roll it out quickly. If we have a moderate sized feature branch, it is merged, tested, and rolled out immediately as soon as it passes QA. No need to set a rigid "We only deploy on Wednesdays" schedule. Pass QA, deploy immediately.
> Not if it's for a bug fix or security patch, but presumably most of your development work is adding new features? In that case, isn't the whole point that it's something the user will notice?
No, a huge chunk of dev time is spent iterating on and improving existing features. There gets to be a point in a product's life cycle where you are more or less feature complete, where the emphasis shifts to refining and improving what you already have. Sure, there are new features, but if you never iterate, you've got older stuff accumulating dust and ageing badly.
Yes, see the following:
> The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.
We're not just deploying bug fixes quickly (which you seem to be fixating on a bit), we are constantly making small iterations. If we're handling support and see something that could be improved, we improve it and roll it out quickly. If we have a moderate sized feature branch, it is merged, tested, and rolled out immediately as soon as it passes QA. No need to set a rigid "We only deploy on Wednesdays" schedule. Pass QA, deploy immediately.
> Not if it's for a bug fix or security patch, but presumably most of your development work is adding new features? In that case, isn't the whole point that it's something the user will notice?
No, a huge chunk of dev time is spent iterating on and improving existing features. There gets to be a point in a product's life cycle where you are more or less feature complete, where the emphasis shifts to refining and improving what you already have. Sure, there are new features, but if you never iterate, you've got older stuff accumulating dust and ageing badly.
I think the point is that deployments should be streamlined and we should not make such a big deal out of them as many of us used to.
If the code for a new feature is ready and all the tests pass, being able to deploy with the click of a button allows you to close the ticket right away and move on to the next task in line. Also, there's no need to coordinate with other developers that are about to finish their thing.
Overall, it's good for productivity and peace of mind, as streamlined deploys are less error prone.
If the code for a new feature is ready and all the tests pass, being able to deploy with the click of a button allows you to close the ticket right away and move on to the next task in line. Also, there's no need to coordinate with other developers that are about to finish their thing.
Overall, it's good for productivity and peace of mind, as streamlined deploys are less error prone.
It is all about feedback loops. If you finish a feature and no one sees it for a month because your on a monthly roll, you don't know if it really works for them. And if it comes back and needs an improvement, you have to figure out what you were doing again. Two month iterations are hard to learn from.
benefits
Over what? Are you seeking a comparison with releasing one a day, a week, a month, a year?
One of the responses to your question contrasts multiple-times-daily releases with once-every-two-months releases. There are certainly clear benefits there. Personally, i am less convinced of the net benefits of multiple-times-daily releases over a once-a-day or even once-a-week releases.
I work on a few apps which have fast release cycles, where we often release several times a week, and occasionally release several times a day.
One big benefit is the ability to fix bugs fast. When a customer service rep comes over and tells us about a mistake in the data we are showing our users, or a sysadmin comes up and tells us about some alerts we're causing on his dashboard, we can quite often fix the bug and release the fix the same day. That really helps keep customers and sysadmins happy.
Another benefit is the ability to get metrics into production fast. If we're making plans, sometimes we realise that we really need more information about the behaviour of the system or users to make a good decision. If that's information that can be captured in metrics or log events, we can add the metric release, and then gather information quickly enough not to completely derail our planning process.
The ability to add metrics fast also helps with tracking down bugs, because we can essentially instrument the production system on demand. For example, at one point, we had a suspicion that some service that took two overlapping date ranges as parameters was being called with date ranges that did not overlap. So we added a check for that which logged an event if they didn't, and pushed it to production. The next morning, we knew exactly how much this was happening, and which other application the was making the calls.
The flip side of this is that our frequently-released apps get essentially no meaningful manual QA, and precious little automatic testing. We have huge suites of tests which we run in CI, of course, but we don't have time to put every release through soak or performance tests.
If you wanted to make a general handwavy statement, i'd say that having really fast release cycles helps break down the wall between development and production. Which you might or might not find was a good thing.
Over what? Are you seeking a comparison with releasing one a day, a week, a month, a year?
One of the responses to your question contrasts multiple-times-daily releases with once-every-two-months releases. There are certainly clear benefits there. Personally, i am less convinced of the net benefits of multiple-times-daily releases over a once-a-day or even once-a-week releases.
I work on a few apps which have fast release cycles, where we often release several times a week, and occasionally release several times a day.
One big benefit is the ability to fix bugs fast. When a customer service rep comes over and tells us about a mistake in the data we are showing our users, or a sysadmin comes up and tells us about some alerts we're causing on his dashboard, we can quite often fix the bug and release the fix the same day. That really helps keep customers and sysadmins happy.
Another benefit is the ability to get metrics into production fast. If we're making plans, sometimes we realise that we really need more information about the behaviour of the system or users to make a good decision. If that's information that can be captured in metrics or log events, we can add the metric release, and then gather information quickly enough not to completely derail our planning process.
The ability to add metrics fast also helps with tracking down bugs, because we can essentially instrument the production system on demand. For example, at one point, we had a suspicion that some service that took two overlapping date ranges as parameters was being called with date ranges that did not overlap. So we added a check for that which logged an event if they didn't, and pushed it to production. The next morning, we knew exactly how much this was happening, and which other application the was making the calls.
The flip side of this is that our frequently-released apps get essentially no meaningful manual QA, and precious little automatic testing. We have huge suites of tests which we run in CI, of course, but we don't have time to put every release through soak or performance tests.
If you wanted to make a general handwavy statement, i'd say that having really fast release cycles helps break down the wall between development and production. Which you might or might not find was a good thing.
One benefit is that more releases means fewer changes in each release, which means when something starts to go wrong, it's easier to figure out what happened.
But of course there's a limit; you don't want releases that are so quick that you don't collect enough data about the effects of each release.
But of course there's a limit; you don't want releases that are so quick that you don't collect enough data about the effects of each release.
This book may be good read to you: http://www.thoughtworks.com/continuous-delivery
I've been using Ansible for a few production-related tasks lately, and think it's great. It provides the right level of abstraction, IMHO: you can crack open a playbook, read through it, and know exactly what it is doing. There's also a growing number of playbooks if you google around.
That said, the biggest downside I've seen with Ansible is reusable components. They have something called Galaxy in beta [1], which should help, although it feels a bit rough yet...
[1] https://galaxy.ansible.com/
That said, the biggest downside I've seen with Ansible is reusable components. They have something called Galaxy in beta [1], which should help, although it feels a bit rough yet...
[1] https://galaxy.ansible.com/
Galaxy is slowly, but surely, becoming a more helpful tool. At this point, I think most people are using ad-hoc playbooks and homegrown roles. I think more people will start using (and improving) Galaxy roles as the tool becomes more mature (most of the glaring bugs are gone in Ansible 1.6, which is coming soon), and more roles are added/shared.
It's nice being able to put together a plethora of servers within seconds, and start deploying them to DO, AWS, or a local box for playing around within a few minutes! See, for example: https://github.com/geerlingguy/ansible-vagrant-examples
It's nice being able to put together a plethora of servers within seconds, and start deploying them to DO, AWS, or a local box for playing around within a few minutes! See, for example: https://github.com/geerlingguy/ansible-vagrant-examples
>That said, the biggest downside I've seen with Ansible is reusable components
This was a major, major issue with puppet, too - although somewhat worse, in that the manifests were more unreadable.
One issue that ansible still hasn't really solved is the horrible if/then/elses you have to put in to accommodate RHEL/ubuntu/centos/whatever else differences.
I like that galaxy at least lets you specify which environment the playbook will run on, though.
This was a major, major issue with puppet, too - although somewhat worse, in that the manifests were more unreadable.
One issue that ansible still hasn't really solved is the horrible if/then/elses you have to put in to accommodate RHEL/ubuntu/centos/whatever else differences.
I like that galaxy at least lets you specify which environment the playbook will run on, though.
Check out the group_by module, as this can make those totally go away.
Was quite surprised to see a lack of playbooks in Galaxy that have anything to do with installing/setting up the Java RE .. anyone know any reason why that may be the case?
Cool stuff! We actually use a similar setup [1], but with the additional hassle of figuring out how to handle hardware connections back to the app. Docker + Jenkins has definitely been a big win.
[1] http://www.appneta.com/blog/automated-testing-with-docker/
[1] http://www.appneta.com/blog/automated-testing-with-docker/
Take a look at github.com/drone/drone for an upcoming CI platform built around Docker. It's totally open source, but you can use the hosted version at drone.io as well.
Instead of doing:
You could do: