In praise of continuous deployment: The WordPress.com story(toni.org)
toni.org
In praise of continuous deployment: The WordPress.com story
http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/
14 comments
I seriously would like more people to explain their experience if they tried this, but unit test and integration tests are no substitute for QA. Especially for a young startup trying to get traction, I seriously can't see how this is beneficial to have possibly a buggy to low quality app with possibly several bad deployments. How do you avoid the problem of continuously putting out small fires, rather than looking at big picture type problems with this continuous deployment cycles ? Am I misunderstanding the process ?
Continuous Deployment has been a tremendous boon to us at CollegeJobConnect. We make it work by:
- Being maniacal about integration tests
- Manually testing key flows (user registration in particular)
- Hoptoad exception tracking coupled with push email
- cap deploy:rollback
are you manually testing, on your production environments ?
When we make significant changes to elements that affect certain critical aspects of our site (a very limited number of specific places, like user registration) we sometimes walk through manually just so we can sleep a bit easier. To date, our test coverage has been sound and we haven't caught anything this way. It may be redundant but we like to see it for ourselves just to be sure.
They forgot: Optimize for security.
I think it's worth pointing out that while I suspect you may have intended that as pure snark, there is a deeper point lying under it. Assuming this continuous deployment is something like "developer picks up bug, hacks out whatever solution works, deploys it five minutes later", then you are punting on any issue that may require a bit more introspection. This most notably includes security, but there's other things that this procedure will fry too, like longterm maintainability and such. Not to mention offloading QA onto your customers, which may work for a free webservice holding nothing of particular value but hardly works for everybody. (And I hope they have good backups.)
With this logic, it's perfectly fair to point out that WordPress's security record is absolute shit (tptacek: "There are unforced errors in Wordpress. Every web application will have a cross-site scripting mistake. It takes a special one to have "anonymous commenter" -> "admin" privilege escalation, or executable style templates." - http://news.ycombinator.com/item?id=1328583) and it's fair to conclude that this "We systematically and deliberately fail to check our code before deploying it!" approach to development may be a contributing factor.
Now, nothing stops you from implementing "take bug, code solution, run it through code review and QA, deploy" on a patch-by-patch basis, but I have a hard time imagining making 20 releases a day with that procedure, so I'm assuming that's not what they are doing with some reason.
Basically, what I'm saying here is that hearing WordPress advocate this model is a strong argument against it. If they honestly think it's not costing them anything, or even just a net benefit, then I call their judgment into question. Or perhaps rather, further into question, since I think their security track record was already calling their judgment into question.
With this logic, it's perfectly fair to point out that WordPress's security record is absolute shit (tptacek: "There are unforced errors in Wordpress. Every web application will have a cross-site scripting mistake. It takes a special one to have "anonymous commenter" -> "admin" privilege escalation, or executable style templates." - http://news.ycombinator.com/item?id=1328583) and it's fair to conclude that this "We systematically and deliberately fail to check our code before deploying it!" approach to development may be a contributing factor.
Now, nothing stops you from implementing "take bug, code solution, run it through code review and QA, deploy" on a patch-by-patch basis, but I have a hard time imagining making 20 releases a day with that procedure, so I'm assuming that's not what they are doing with some reason.
Basically, what I'm saying here is that hearing WordPress advocate this model is a strong argument against it. If they honestly think it's not costing them anything, or even just a net benefit, then I call their judgment into question. Or perhaps rather, further into question, since I think their security track record was already calling their judgment into question.
> Not to mention offloading QA onto your customers, which may work for a free webservice holding nothing of particular value but hardly works for everybody.
Funny, when I worked at ESPN.com this is exactly what we did. When I first started, I was horrified.
But you know what? Traffic and revenue still grew, and users didn't seem to mind.
Deploying quickly gave us an incredible speed advantage that allowed us to change and respond to problems and new opportunities.
Funny, when I worked at ESPN.com this is exactly what we did. When I first started, I was horrified.
But you know what? Traffic and revenue still grew, and users didn't seem to mind.
Deploying quickly gave us an incredible speed advantage that allowed us to change and respond to problems and new opportunities.
Not surprisingly, I disagree. First of all, you are lumping WordPress the open source software and WordPress.com the hosted service together. My post is about WordPress.com which uses the continuous deployment model and has a very solid security track record (as good or better than any other large scale web service like Facebook, GMail, etc). Also, our fast deployment model helps to very quickly fix whatever problems there may. Finally, though I can't speak for WordPress.org because it's separate from Automattic, they have developed a very effective model of responding to security issues and auto-notifying and -updating people.
"First of all, you are lumping WordPress the open source software and WordPress.com the hosted service together."
Yes I am, and I apologize. I did not realize there was much of a distinction. I would edit away much of my message now if I could.
But...
"Also, our fast deployment model helps to very quickly fix whatever problems there may."
Security problems do not work that way. Rapid deployment does not recover your user database that the hacker got, just as one example. Rapid deployment may close XSS holes, but doesn't undo the damage they did while open. And you don't need rapid deployment to close XSS anyhow, historically lots of people have managed that without rapid deployment. I still see it as potentially beneficial on some fronts but an enormous risk on other fronts I care about a lot.
Yes I am, and I apologize. I did not realize there was much of a distinction. I would edit away much of my message now if I could.
But...
"Also, our fast deployment model helps to very quickly fix whatever problems there may."
Security problems do not work that way. Rapid deployment does not recover your user database that the hacker got, just as one example. Rapid deployment may close XSS holes, but doesn't undo the damage they did while open. And you don't need rapid deployment to close XSS anyhow, historically lots of people have managed that without rapid deployment. I still see it as potentially beneficial on some fronts but an enormous risk on other fronts I care about a lot.
Is wordpress.org a different code base?
That was my first thought: they are so many security flaws in Woedpress, how can it be a good example? Check the latest advisories on SecurityFocus.
Like many a member here, I am building something. In my case in PHP.
I have lovingly crafted ( wasted ) a system which is still in its infancy to handle a form of automated test driven development and from what I can see so far, the times I break it or discover a bug ( logic bug ) in my code later it is usually due to week tests.
Clearly I am not a large site like wordpress.com, a part of me thinks its the right thing to do. It certainly makes deploying a breeze and gives a little more "confidence" that the system didn't just fall over.
The sad part, at least in the PHP world, there are no libraries with an explanation on how best to use it with your site / application structure.
The database part is the biggest hiccup. My solution is a duplicate database with no data. First it confirms that the "fake db" matches the "real db" and warns if the table structure is different.
Then with my blank "fake db" using functions in my tests to "setup" and "destroy" I purposly build data to test with.
Once my site is operational, I will look to using live data in the "fake db" to simulate with real data. But so far it has been an interesting journey.
Clearly, TDD would become a bigger issue if you have to test sphinx/couchdb/mongodb etc setups but like with all creations its starts with a blank.php. ( in my case )
Not that this covers "Optimise for security".
But my test suite, after the first "test time" the test has been run, will warn if the "library","model", "controller" it is linked to has been altered without the "test" file changing, which at least warns me to the idea that maybe I need to refine my test.
All very padantic, but reading from the side lines of patio11, I can't help but see logic in "automating" things to make your life easier, more efficent and less likely to add human error into the equation.
I leave with what I consider a valid point.
Once upon a time, people looked at the "MVC" approach as time consuming and wasteful. At least from my understanding of watching the web evolve from the earlier days.
I have lovingly crafted ( wasted ) a system which is still in its infancy to handle a form of automated test driven development and from what I can see so far, the times I break it or discover a bug ( logic bug ) in my code later it is usually due to week tests.
Clearly I am not a large site like wordpress.com, a part of me thinks its the right thing to do. It certainly makes deploying a breeze and gives a little more "confidence" that the system didn't just fall over.
The sad part, at least in the PHP world, there are no libraries with an explanation on how best to use it with your site / application structure.
The database part is the biggest hiccup. My solution is a duplicate database with no data. First it confirms that the "fake db" matches the "real db" and warns if the table structure is different.
Then with my blank "fake db" using functions in my tests to "setup" and "destroy" I purposly build data to test with.
Once my site is operational, I will look to using live data in the "fake db" to simulate with real data. But so far it has been an interesting journey.
Clearly, TDD would become a bigger issue if you have to test sphinx/couchdb/mongodb etc setups but like with all creations its starts with a blank.php. ( in my case )
Not that this covers "Optimise for security".
But my test suite, after the first "test time" the test has been run, will warn if the "library","model", "controller" it is linked to has been altered without the "test" file changing, which at least warns me to the idea that maybe I need to refine my test.
All very padantic, but reading from the side lines of patio11, I can't help but see logic in "automating" things to make your life easier, more efficent and less likely to add human error into the equation.
I leave with what I consider a valid point.
Once upon a time, people looked at the "MVC" approach as time consuming and wasteful. At least from my understanding of watching the web evolve from the earlier days.
I can't really see this working in my current line of employment, warehouse automation. Once a system is in production we're lucky if we can deploy a bug fix once a fortnight.
Sounds alot like what I like about App Engine. Hit the upload button and you've got a new version uploaded and live within minutes.