Releasing software to the fleet far too quickly broke stuff(rachelbythebay.com)
rachelbythebay.com
Releasing software to the fleet far too quickly broke stuff
http://rachelbythebay.com/w/2020/04/23/rel/
13 comments
The only major changes I've seen in IT organizations of companies was where the existing IT department had declined or was let go over time and any development was in slumber mode. Then an excited manager comes around and erects a new IT organization from nothing (usually leaning heavily on consultants and freelancers), having sold a Digital Transformation to the company. The new IT / developer department spends millions a year in conjuring up a new server stack (cloud native) and a new software architecture (microservices woo), often building on top of existing ancient software (probably running on a mainframe three further abstraction levels down), with the intent of eventually replacing them. Eventually. Maybe.
Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features.
[/rant]
Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features.
[/rant]
That got a little to real for me, I started thinking about work.
If only CloudFormation followed this philosophy. Unfortunately, doing things "The Right Way (tm)" within AWS means torturously long feedback loops, tearing down stacks and rebuilding them again over and over again (and when you're working with RDS databases, "tear down" is 10 mins and "rebuild" is 15 mins), then some stacks fail to delete because an IAM role that was just created is now somehow unable to be deleted...
Our company has restricted it so CloudFormation is the only way to provision infrastructure because doing things the right way makes our lives significantly harder.
And so the "native cloud" tax continues to accumulate and grow until you spend more and more time and money on just building and maintaining infrastructure...
Our company has restricted it so CloudFormation is the only way to provision infrastructure because doing things the right way makes our lives significantly harder.
And so the "native cloud" tax continues to accumulate and grow until you spend more and more time and money on just building and maintaining infrastructure...
Why are you bringing down the RDS stack to make cloudformation udpates? Can you share an example of something that makes you need to do this over and over again?
I'm creating a database with parameters, roles, backup buckets and so on. Usually each time I run the stack I either get a template error, or the stack partly runs then hits some other error. Then the stack rolls back, and re-running it will fail because it's in rolled_back state. So then I need to delete the whole thing and start again.
When I can simply re-run and update the stack without deleting it I do (for example if I just change a property of one of the resources). But I spend a lot of time with stacks that are in broken states for various reasons.
When I can simply re-run and update the stack without deleting it I do (for example if I just change a property of one of the resources). But I spend a lot of time with stacks that are in broken states for various reasons.
I share this pain - it’s a major reason why we switched to Terraform, along with not having to wait 6-12 months for support for new AWS features - but I would recommend using CDK to generate your CF stacks. The cfn-python-lint tool was helpful for catching many things without a 20 minute delay but CDK goes even further on the boilerplate reduction and a real programming language beats YAML in every way.
Maybe changesets would be useful here. It’s like terraform plan but better because it’s aws native.
$aws cloudformation deploy. .. —no-execute-changes etc
https://docs.aws.amazon.com/cli/latest/reference/cloudformat...
$aws cloudformation deploy. .. —no-execute-changes etc
https://docs.aws.amazon.com/cli/latest/reference/cloudformat...
>>> ... when the change was introduced and made it far easier to do the right thing.
Correction: When the change was introduced and made it far easier for people to do the thing they are trying to do.
Nobody cares about doing the right thing, employees just do whatever they have to do now then move on to the next task/project. More often than not, changes are making it harder for people to do their job and it's no surprise that it's not welcome.
Correction: When the change was introduced and made it far easier for people to do the thing they are trying to do.
Nobody cares about doing the right thing, employees just do whatever they have to do now then move on to the next task/project. More often than not, changes are making it harder for people to do their job and it's no surprise that it's not welcome.
Exactly, from my personal experience that is the hardest ball to juggle: there are changes that will be painful, that won't have an immediate impact but that over time will compound to it. How do you sell this to people that are going to go through the pain but very likely won't be there to see its benefits?
A lot of people fail to realize there are large multiplicative factors when the infrastructure team makes other parts of the organization implement a “simple” change. In large companies, there might be dozens or hundreds of teams that rely on the infrastructure. If the change requires each team to learn the tool of the month, and make a one line configuration change, it will cost the organization hundreds of developer days to implement. If engineering and release management have to schedule that work to align with business constraints, the change is costing years of people time.
If you’re even further down the rabbit hole, you’ve adopted devops, and are forcing people that actively hate operations stuff to learn obscure in-house operations tools. That, combined with preventing them from doing the work they signed up for, leads to burn out and resignations.
It can get even worse than that. I routinely see all of the above compounded with infra bikeshedding that delays rollouts by months, and results in weeks of work per team. At that point, the organizational damage ends up being many times the budget of the IT department.
So, yeah, IT needs to figure out how to help people be more productive while simultaneously implementing best practices, and it’s hard. I deeply appreciate people that are good at that (and have worked with some).
If you’re even further down the rabbit hole, you’ve adopted devops, and are forcing people that actively hate operations stuff to learn obscure in-house operations tools. That, combined with preventing them from doing the work they signed up for, leads to burn out and resignations.
It can get even worse than that. I routinely see all of the above compounded with infra bikeshedding that delays rollouts by months, and results in weeks of work per team. At that point, the organizational damage ends up being many times the budget of the IT department.
So, yeah, IT needs to figure out how to help people be more productive while simultaneously implementing best practices, and it’s hard. I deeply appreciate people that are good at that (and have worked with some).
This is quite the representative case between Infra teams (or PaaS or SRE - based on where you're located) and application teams.
Being part of the platform team, I've been in the exact same situation before at an ex employer. We assumed that _infra problems_ were _everyone's problems_ and how can anyone who cares about their application _not_ adopt to the latest-and-greatest-platform (or tool) we released. A lot of time was spent on figuring out the 'best approach' to getting everyone onboard quickly - we tried the carrot and the stick. We failed miserably at both.
The trouble is, as someone else mentioned here - we didn't really go and _talk to the customer_. Internal customers are customers too - and 'product' thinking should be equally applied to all internal platforms. The general rule applies - talk to the customer and understand what they're going through. They have their own sets of problems and priorities.
(a) Is the latest way actually _saving them a lot of time_
(b) Can we take another 2 weeks and make it even easier for them to come on board?
(c) If they're really busy and/or lack the expertise, can we carve out some time to give enough training sessions?
Yet, this never happens and we invariably end up blaming the application teams for _not doing this seemingly simple_ change. It may seem downright stupid from the platform engineer's PoV, but we are partly to blame.
This also leads me to think that the cloud wars will be won by the <cloud-provider> that does this job better - of understanding the customer needs on a day to day basis and building platforms for them.
Being part of the platform team, I've been in the exact same situation before at an ex employer. We assumed that _infra problems_ were _everyone's problems_ and how can anyone who cares about their application _not_ adopt to the latest-and-greatest-platform (or tool) we released. A lot of time was spent on figuring out the 'best approach' to getting everyone onboard quickly - we tried the carrot and the stick. We failed miserably at both.
The trouble is, as someone else mentioned here - we didn't really go and _talk to the customer_. Internal customers are customers too - and 'product' thinking should be equally applied to all internal platforms. The general rule applies - talk to the customer and understand what they're going through. They have their own sets of problems and priorities.
(a) Is the latest way actually _saving them a lot of time_
(b) Can we take another 2 weeks and make it even easier for them to come on board?
(c) If they're really busy and/or lack the expertise, can we carve out some time to give enough training sessions?
Yet, this never happens and we invariably end up blaming the application teams for _not doing this seemingly simple_ change. It may seem downright stupid from the platform engineer's PoV, but we are partly to blame.
This also leads me to think that the cloud wars will be won by the <cloud-provider> that does this job better - of understanding the customer needs on a day to day basis and building platforms for them.
Really good post - "making it hard to do the wrong thing" is more or less my entire design ethos when it comes to software, particularly developer tools.
Somewhat on-topic, instead of re-inventing the wheel on this one, I've recently been using Flagger (https://flagger.app/) and have been exceptionally happy with it. Automated rollback on error rates, metrics or integration tests. It's so nice to finally be able to stop writing this code at every company I work for!
Somewhat on-topic, instead of re-inventing the wheel on this one, I've recently been using Flagger (https://flagger.app/) and have been exceptionally happy with it. Automated rollback on error rates, metrics or integration tests. It's so nice to finally be able to stop writing this code at every company I work for!
> making it hard to do the wrong thing" is more or less my entire design ethos when it comes to software, particularly developer tools.
See also poka-yoke: https://en.m.wikipedia.org/wiki/Poka-yoke
See also poka-yoke: https://en.m.wikipedia.org/wiki/Poka-yoke
I found it strange that that page doesn’t mention the example I immediately thought of after reading the first sentence: large dangerous machinery such as presses or lasers that require two hands to operate to prevent the case where the operator, in a hurry, still has one hand in the danger area when he presses the “do it” button with the other.
I'm going to add "poka-yoke" to the list of things that I make students commit to memory in my law school contract-drafting course; it'll be a synonym for "R.O.O.F" — Root Out Opportunities for [Foul]-ups.
(This is another example of how legal-contract drafters can learn from the software world, including concepts such as, for example, orthogonality; modularity; R.O.O.F.; D.R.Y. [Don't Repeat Yourself]; graceful failure; and the benefit of sometimes explaining why things are done the way they are, to help future readers understand.)
(This is another example of how legal-contract drafters can learn from the software world, including concepts such as, for example, orthogonality; modularity; R.O.O.F.; D.R.Y. [Don't Repeat Yourself]; graceful failure; and the benefit of sometimes explaining why things are done the way they are, to help future readers understand.)
That's lovely, thank you!
How does Flagger compare to Spinnaker in your experience?
We're Armory (YC W17) and we bring Spinnaker into complex enterprise environments. Happy to answer any ?s about it. Or you can learn more at https://www.armory.io/learn/what-is-spinnaker/ and docs at http://docs.armory.io/
I haven't used Spinnaker outside of a few tutorials and experiments, but Flagger is far simpler. It's just a "Canary" CRD you install that controls nginx-ingress (in our case). Watches prometheus, nginx and does things to a deployment.
Spinnaker invents a lot of new terms and complexities. Flagger just does one thing, and it does it pretty darn well :)
Spinnaker invents a lot of new terms and complexities. Flagger just does one thing, and it does it pretty darn well :)
Large-scale deploy systems for a distributed environment are ridiculously hard to build. We had to set one up quickly and on a budget, so in lieu of fancy engineering we set up three rules:
- Any change, whether it's a one-line config update or an entirely new backend, can only go out to a maximum of x% of hosts at a time
- Hosts are taken fully out of rotation before getting an update
- Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions
The first two are easy to enforce programmatically, and the third one relies on the team. While the trade-off is that it's harder to make relatively simple changes, it has served us well so far.
- Any change, whether it's a one-line config update or an entirely new backend, can only go out to a maximum of x% of hosts at a time
- Hosts are taken fully out of rotation before getting an update
- Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions
The first two are easy to enforce programmatically, and the third one relies on the team. While the trade-off is that it's harder to make relatively simple changes, it has served us well so far.
Downgrade should “just work” (with help from the team) for stateless services, since the old and new have to coexist during upgrade. However, if you’re pushing out new storage systems, supporting downgrade ends up lowering overall reliability in practice. It takes many times more effort to implement and test the downgrade of every disk layout change than it does to ensure that upgrades will always succeed in practice.
> Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions
How does that rule work out in practice? If you're doing staged rollouts of new builds because you think there's a decent chance those new builds have bugs in them, shouldn't you realistically expect that rollbacks could have unexpected side-effects?
How does that rule work out in practice? If you're doing staged rollouts of new builds because you think there's a decent chance those new builds have bugs in them, shouldn't you realistically expect that rollbacks could have unexpected side-effects?
> expect that rollbacks could have unexpected
Expect the unexpected? Sure anything can happen.
But paxys is specifically referring to expected (known) side-effects of rollbacks. For example if Foo and Bar are on versions Foo.3 and Bar.3, and we know rolling back to Bar.2 breaks Foo.3, then you broke backwards compatibility. Generally all combinations of possible rollbacks must continue to work, e.g., {Foo.2, Foo.3} x {Bar.2, Bar.3}. The number of rollbacks is limited to 1 or 2 versions, e.g., Foo.1 and Bar.1 cannot be used anymore.
Expect the unexpected? Sure anything can happen.
But paxys is specifically referring to expected (known) side-effects of rollbacks. For example if Foo and Bar are on versions Foo.3 and Bar.3, and we know rolling back to Bar.2 breaks Foo.3, then you broke backwards compatibility. Generally all combinations of possible rollbacks must continue to work, e.g., {Foo.2, Foo.3} x {Bar.2, Bar.3}. The number of rollbacks is limited to 1 or 2 versions, e.g., Foo.1 and Bar.1 cannot be used anymore.
The third one is not too bad to test programmatically, if you write a test harness that spins up a mixed version system.
It's funny that the big tech companies have such high hiring bars and yet when you get inside there are large teams of people actively working to protect the company from the stupidity of their colleagues.
Your mistake is thinking it’s stupidity: what you’re seeing is due to conflicting priorities. Smart, motivated people like to hit their goals and will bend whatever rules they need to so – and if you’re busy it’s easy to think you don’t have time to do it the proper way if that takes a while.
That’s why investing in tools makes sense: when those people think it’s harder to do things the wrong way then your incentives are aligned. SRE culture around error budgets can help, too: having issues reschedule your project team is a much better schedule than chucking it over the wall to be ops’ problem because people learn that they’re not saving time by cutting corners.
That’s why investing in tools makes sense: when those people think it’s harder to do things the wrong way then your incentives are aligned. SRE culture around error budgets can help, too: having issues reschedule your project team is a much better schedule than chucking it over the wall to be ops’ problem because people learn that they’re not saving time by cutting corners.
I’m mostly trying to protect myself from my own stupidity.
That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.
That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.
That's the story of my development life! I know I'm human, and as such, I will err, but I really don't want to be erring as humans do in production.
I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
I had a boss who insisted on calling the root account stupiduser, because when a server gets properly fucked up it's invariably because some stupid user did the wrong thing...
When I say stupidity I don't mean mistakes.
From the article:
> At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board.
Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
From the article:
> At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board.
Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
If you meet an asshole in the morning, you met an asshole. If you meet assholes all day long, you're the asshole. I can think of a few current or former coworkers who talk about how everybody is an idiot and they are the biggest idiots I know.
Insisting people use a tool that is capable of doing dangerous things correctly, every time, is white-knuckling. Yelling at them about screwing it up is running your organization by shame. When you do that, eventually only the people with very high pain tolerance ever get promoted, or even heard. Do you really want to work on a team run by masochists/sadists?
Similar to make the change easy, make the easy change, if you want to have a dialog with people about doing better, the easiest way to do that is to give them a reliable tool (literal or figurative), and then start insisting they use it when it becomes obvious they haven't. The people who insist on doing things the hardest way possible can still do it as long as nobody else feels the consequences. But typically what they're going to learn is that where self-reflection is concerned, they're bad at math and they have way more frequent problems and far more collateral damage than they like to tell themselves.
Insisting people use a tool that is capable of doing dangerous things correctly, every time, is white-knuckling. Yelling at them about screwing it up is running your organization by shame. When you do that, eventually only the people with very high pain tolerance ever get promoted, or even heard. Do you really want to work on a team run by masochists/sadists?
Similar to make the change easy, make the easy change, if you want to have a dialog with people about doing better, the easiest way to do that is to give them a reliable tool (literal or figurative), and then start insisting they use it when it becomes obvious they haven't. The people who insist on doing things the hardest way possible can still do it as long as nobody else feels the consequences. But typically what they're going to learn is that where self-reflection is concerned, they're bad at math and they have way more frequent problems and far more collateral damage than they like to tell themselves.
There are a few strawmen in this post.
1. I never said everybody is stupid. The vast majority of people in Big Tech are great, but there ARE stupid people there too. I resent the implication of your first paragraph.
2. Who said anything about yelling or other toxic behaviors? There are options other than "engineer a foolproof system" and "institute a sado-masochistic management strategy".
3. I never advocated for doing things the hardest way possible, I advocated for doing things correctly (to the best of your ability) even when it is difficult to do so.
The example given in the article is a team choosing to do a global rollout when the option existed to do a staged rollout. There's myriad technical things you can do to make it easier and faster to do a staged rollout, harder to do a global rollout, reduce the risk of a global rollout, etc. etc. ... but what about instilling a healthy respect for production in your colleagues? That's been conspicuously absent from most postmortems I've sat in on.
1. I never said everybody is stupid. The vast majority of people in Big Tech are great, but there ARE stupid people there too. I resent the implication of your first paragraph.
2. Who said anything about yelling or other toxic behaviors? There are options other than "engineer a foolproof system" and "institute a sado-masochistic management strategy".
3. I never advocated for doing things the hardest way possible, I advocated for doing things correctly (to the best of your ability) even when it is difficult to do so.
The example given in the article is a team choosing to do a global rollout when the option existed to do a staged rollout. There's myriad technical things you can do to make it easier and faster to do a staged rollout, harder to do a global rollout, reduce the risk of a global rollout, etc. etc. ... but what about instilling a healthy respect for production in your colleagues? That's been conspicuously absent from most postmortems I've sat in on.
When you serve hundreds or thousands of teams, as this tool did, one of them is bound to be lazy. Defaults should be safe.
> Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
In practice it ends up being more like a request. :)
In practice it ends up being more like a request. :)
I think you've hit the nail on the head. When the company is small (like a traditional startup), the main risk is not delivering features. When the company is large, the main risk is loss of {data, privacy, uptime, reputation, lawsuits, unexpected licensing fees}.
So large companies put in place systems to reduce those risks. Unfortunately the default seems to be to make it hard to deliver features.
So large companies put in place systems to reduce those risks. Unfortunately the default seems to be to make it hard to deliver features.
Smarter people work on harder problems, breaking things in more obscure ways, so you need other smart people to protect the service from breaking for customers...
Deploying to 100% of production when the option exists to do a staged rollout[0] isn't "breaking things in more obscure ways," it's ignorance or laziness.
[0] As mentioned in the article.
[0] As mentioned in the article.
> "it's ignorance or laziness"
Well this kind of bugs me, that explanation can be applied to literally anyone who's ever broken anything by accident. With the benefit of hindsight you can always call them out as ignorant or lazy.
The circumstances in the article were not explicitly lazy or ignorant in my opinion. It was an unusually large update (2 gigabytes), which due to being able to deploy all servers simultaneously, created an outage.
Now, sure they could have done a staged roll-out, but why were they even allowed to deploy to all servers like that? How much due-diligence to they need to do in order to do their job?
Reminds me of [1] that guy who accidentally deleted the entire production database on his first day. Was that "ignorance or laziness"? I would say it's a failure of the company's internal processes and systems.
[1] https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...
Well this kind of bugs me, that explanation can be applied to literally anyone who's ever broken anything by accident. With the benefit of hindsight you can always call them out as ignorant or lazy.
The circumstances in the article were not explicitly lazy or ignorant in my opinion. It was an unusually large update (2 gigabytes), which due to being able to deploy all servers simultaneously, created an outage.
Now, sure they could have done a staged roll-out, but why were they even allowed to deploy to all servers like that? How much due-diligence to they need to do in order to do their job?
Reminds me of [1] that guy who accidentally deleted the entire production database on his first day. Was that "ignorance or laziness"? I would say it's a failure of the company's internal processes and systems.
[1] https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/a...
Wasn't that one of the golang design philosophies? :)
> "This was one of those times when we needed to build a system that made it stupidly easy to do the right thing, so that it would actually be MORE work to do the wrong thing"
This is basically the core of UX. It's kind of amazing to me that UX is so poorly represented in management tools.
This is basically the core of UX. It's kind of amazing to me that UX is so poorly represented in management tools.
The joke about building idiot-proof software comes to mind, but it's surprising to me how many systems really don't do a good job of making the right thing easy and the wrong hard. Dangerous defaults have consequences!
Of course sometimes there's no substitute for learning things the hard way. Where practical, I've found having the responsible team on the hook for cleaning up their mess tends to work wonders in terms of getting people to be appropriately careful. That's not to say mistakes can't still happen, but it cuts down a lot.
Of course sometimes there's no substitute for learning things the hard way. Where practical, I've found having the responsible team on the hook for cleaning up their mess tends to work wonders in terms of getting people to be appropriately careful. That's not to say mistakes can't still happen, but it cuts down a lot.
Really interesting post. Can anyone comment on how serverless deployments help and/or exacerbate the deployment problem?
To give a theoretical starting point - Acme corporation has a serverless API deployed at api.acme.com powering website acme.com. The staging system is deployed at staging.api.acme.com, and the corresponding web app (e.g. staging.acme.com) targets that API. Staging gets deployed to production via a pull request that triggers CI/CD, which has no ability to do an X% rollout (i.e. the underlying serverless function is updated).
Developer John Smith is writing a feature and merging to his personal branch "jsmith" - he can test his branch with jsmith.staging.api.acme.com and a corresponding development web app. When he's finished the feature, he does a PR to the staging branch, which is packaged into a PR to live.
You may be able to see I'm trying to get input on our own release system :) It's worked quite well so far but I'm still paranoid about not having gradual roll-out - in my experience in bigger companies, there was always some A/B testing in place so bad deploys could be reverted quickly. If the sysop gurus of HN gathering on this thread can pass any comments/critique it would be greatly appreciated.
To give a theoretical starting point - Acme corporation has a serverless API deployed at api.acme.com powering website acme.com. The staging system is deployed at staging.api.acme.com, and the corresponding web app (e.g. staging.acme.com) targets that API. Staging gets deployed to production via a pull request that triggers CI/CD, which has no ability to do an X% rollout (i.e. the underlying serverless function is updated).
Developer John Smith is writing a feature and merging to his personal branch "jsmith" - he can test his branch with jsmith.staging.api.acme.com and a corresponding development web app. When he's finished the feature, he does a PR to the staging branch, which is packaged into a PR to live.
You may be able to see I'm trying to get input on our own release system :) It's worked quite well so far but I'm still paranoid about not having gradual roll-out - in my experience in bigger companies, there was always some A/B testing in place so bad deploys could be reverted quickly. If the sysop gurus of HN gathering on this thread can pass any comments/critique it would be greatly appreciated.
The post was talking about fleets of machines, where just restricting the number of machines that do things seems to be useful.
But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines.
What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version.
The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website.
Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name.
The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API.
If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used.
It took a long time, but now it's mission accomplished and problem solved.
But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines.
What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version.
The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website.
Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name.
The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API.
If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used.
It took a long time, but now it's mission accomplished and problem solved.
Thank you for the input! I agree with your strategy for changing parameters, but for something more error-prone like changing internal logic behind the same parameters, I would still feel the same anxiety doing a non-gradual release.
That is a very good point. And it reminds me how you can do gradual releases when you only have one installation: you do it on a per-user basis, using feature flags. If it's only the logic behind an existing interface, you can implement both the old logic and the new logic and choose between them based on a feature flag.
To do a release, create a new serverless function with the new code. Initially, no traffic routes to it, all traffic goes to the old function. Change things so 1% of traffic goes to the new function, and see if it looks okay. If it does, make that 2%, then 3%, or 4%, or 10%, etc. Once it's 100%, delete the old serverless function.
The provider hosting your serverless functions might not give you that much control over routing. In that case, you would need to write another serverless function which sits in front of your serverless functions and does the routing. If you ever need to update that, well, then you have to cross your fingers.
The provider hosting your serverless functions might not give you that much control over routing. In that case, you would need to write another serverless function which sits in front of your serverless functions and does the routing. If you ever need to update that, well, then you have to cross your fingers.
> Once it's 100%, delete the old serverless function.
I suggest keeping it around for a while. A function not being used should have cost equivalent to a database entry, not a running container or VM, so this should be cheap and the benefits are large. It's still possible to discover you need to roll back a change, even after you've been at 100% for a while. (And no, 'roll forward' is not a sufficient option, especially if that requires coding up a fix you may not even understand yet.)
I suggest keeping it around for a while. A function not being used should have cost equivalent to a database entry, not a running container or VM, so this should be cheap and the benefits are large. It's still possible to discover you need to roll back a change, even after you've been at 100% for a while. (And no, 'roll forward' is not a sufficient option, especially if that requires coding up a fix you may not even understand yet.)
This seems like a great strategy, thank you for your input! It can indeed be accomplished by AWS Global Accelerator along with application load balancers targeting Lambda functions to route a percentage of traffic. I am not sure about the overhead of having a Lambda function to route requests, I’ll have to look into the latency that adds (especially for cold starts).
Using a lambda to route requests to other lambdas would result in you effectively doubling your Lambda cost, as your routing lambda would sit idling while waiting for the target lambda to return a result. If you're using API Gateway, it has a canary deployment option which can route percentages of customer traffic to different Stages, which gives you the gradual rollout capability that you wanted.
Why would the ability to roll back deploys be tied to the ability to A/B test?
I'm guessing the A/B functionality was being used as a Blue-Green deployment system.
This lesson, but in a UX context - Apple generally does a good job of making it hard for users to do the wrong thing (for their definition of wrong).
Sounds like a setup for immutable servers, and I think I'm okay with that.
If new copies of your service don't come up cleanly, you do not power down the old ones. If old servers can't talk to the new service, you get alarm bells going off while most of the old copies are still up and running. You've also partially serialized the deployment so you get some peak shaving.
I wonder sometimes if the problem is that despite our protestations (and things like naming servers by unpronounceable names), we still think of servers running, instead of instances running. So the fact that the 20 copies of my service keep bouncing from machine to machine is at the very least a logistics issue, and possibly unsettling.
If new copies of your service don't come up cleanly, you do not power down the old ones. If old servers can't talk to the new service, you get alarm bells going off while most of the old copies are still up and running. You've also partially serialized the deployment so you get some peak shaving.
I wonder sometimes if the problem is that despite our protestations (and things like naming servers by unpronounceable names), we still think of servers running, instead of instances running. So the fact that the 20 copies of my service keep bouncing from machine to machine is at the very least a logistics issue, and possibly unsettling.
Been there, done that.
The next rant will be: Massive outage because of a bug that was fixed last month. But the fix only shipped to 50% of the hosts, because the deployment system doesn't deploy everywhere by default.
The next rant will be: Massive outage because of a bug that was fixed last month. But the fix only shipped to 50% of the hosts, because the deployment system doesn't deploy everywhere by default.
On the one hand, there are decent technical tools to deal with this kind of thing; I'm directly familiar with Ansible's support for controlled rollouts, most container management layers happily support blue/green deploys, etc. On the other hand -
> Some people obviously couldn't be bothered to step their release along over the course of a couple of days, and we had to find some way to protect the production environment from them.
That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
> Some people obviously couldn't be bothered to step their release along over the course of a couple of days, and we had to find some way to protect the production environment from them.
That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
HR is exactly the worst approach to deal with organizational and process problems like this. What do you do? Fire the entire dev team involved with the decision? Fire their managers who failed to set the right expectations? Fire the product team for setting deadlines that were too aggressive? Fire the CEO because hey, ultimately they are the one responsible for everything?
Even if you do all that, you're not going to go back in time and fix things for your customers, and you're definitely not going to prevent the replacement developers from making the exact same mistakes.
It takes really good leadership to build an engineering culture where good practices are valued and people learn from mistakes, which is why filling those positions is so hard and expensive. A company whose solution for every outage is to fire people isn't going to last very long.
Even if you do all that, you're not going to go back in time and fix things for your customers, and you're definitely not going to prevent the replacement developers from making the exact same mistakes.
It takes really good leadership to build an engineering culture where good practices are valued and people learn from mistakes, which is why filling those positions is so hard and expensive. A company whose solution for every outage is to fire people isn't going to last very long.
GP said:
>> That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
If people refuse to be careful. If they make a mistake once, teach them. If they make a mistake again, that happens. But if they make a mistake regularly, warn them, then warn them sternly, then fire them.
>> That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
If people refuse to be careful. If they make a mistake once, teach them. If they make a mistake again, that happens. But if they make a mistake regularly, warn them, then warn them sternly, then fire them.
Maybe so, but then you still have the problem, because in a big organization, some newbie is always present and going to make that first mistake. So you still need to make it hard for people to make mistakes.
To amplify: think about road signs when you are trying to get somewhere new, especially in places with weird parking or entrances. After you've figured it out, it's easy, but everyone is a first timer at some point.
That's simple; you just make sure nobody ever leaves and then you don't have to hire new people...
Five years ago, I never would have believed you. But this is sage advice.
It is unbelievable how important these technical leaders are, and you don't know how important they are until they're gone.
It is unbelievable how important these technical leaders are, and you don't know how important they are until they're gone.
[deleted]
"A person is smart. People are dumb, panicky dangerous animals and you know it."
When I was first hired into a manufacturing plant I wondered why it was so hard to get people to make a quality product. It turns out that people are people, and they will try to do the job you tell them to do, with the tools and instruction you give them.
The better the tools are, the less instruction is needed. Output rate and quality is also improved.
The poorer the tools are, more instruction is needed; output rate and quality suffer, and at some point safety as well.
"A poor workman blames their tools"; A poor manager blames their workman after providing them with poor tools.
The very first thing you need to do is assess if the worker is following process and if not why not. If they are not following process, is it because of a mistake or a slip-up? Did they intentionally decide not to follow a process because it was difficult?
Disaster documentaries are quick to point out where workers failed to follow written procedures, but they VERY RARELY point out how common those contraventions are with no negative impact.
If "it's stupid but it works" it may not be stupid, or you may have gotten lucky.
https://www.youtube.com/watch?v=WPMMNvYTEyI
When I was first hired into a manufacturing plant I wondered why it was so hard to get people to make a quality product. It turns out that people are people, and they will try to do the job you tell them to do, with the tools and instruction you give them.
The better the tools are, the less instruction is needed. Output rate and quality is also improved.
The poorer the tools are, more instruction is needed; output rate and quality suffer, and at some point safety as well.
"A poor workman blames their tools"; A poor manager blames their workman after providing them with poor tools.
The very first thing you need to do is assess if the worker is following process and if not why not. If they are not following process, is it because of a mistake or a slip-up? Did they intentionally decide not to follow a process because it was difficult?
Disaster documentaries are quick to point out where workers failed to follow written procedures, but they VERY RARELY point out how common those contraventions are with no negative impact.
If "it's stupid but it works" it may not be stupid, or you may have gotten lucky.
https://www.youtube.com/watch?v=WPMMNvYTEyI
> That's easy: HR
Unfortunately, that approach doesn't scale well. In a larger company, there are always new users discovering the systems and how they work (to varying degrees). If doing the right thing isn't by-and-far the easiest path, then it's highly likely someone will misuse the tool – no matter how smart they are. It's far easier to make the interface really easy to use than it is to educate users on proper usage.
Unfortunately, that approach doesn't scale well. In a larger company, there are always new users discovering the systems and how they work (to varying degrees). If doing the right thing isn't by-and-far the easiest path, then it's highly likely someone will misuse the tool – no matter how smart they are. It's far easier to make the interface really easy to use than it is to educate users on proper usage.
> there are always new users discovering the systems
Software engineering turnover rates can be high, like 13%/year [1]. Add to that engineers who internally switch teams. New users is a real concern.
In this case, it sounds like a custom deployment tool, so even engineers with good experience at other companies may not find the feature. Good team on-boarding and documentation is important.
I think code review should catch something like this. Even with the high rate of new users, there should be enough experienced code reviewers to catch the issue.
[1] https://www.techrepublic.com/article/software-had-the-highes...
Software engineering turnover rates can be high, like 13%/year [1]. Add to that engineers who internally switch teams. New users is a real concern.
In this case, it sounds like a custom deployment tool, so even engineers with good experience at other companies may not find the feature. Good team on-boarding and documentation is important.
I think code review should catch something like this. Even with the high rate of new users, there should be enough experienced code reviewers to catch the issue.
[1] https://www.techrepublic.com/article/software-had-the-highes...
The problem is when the distance between the team writing the package manager and random team trying to work on some feature is high. That entire team might not be aware of the team that wants them to deploy gradually and in a specific way. To them, maybe even it's a lot of noise and useless complaining and they don't see it as at all connected to getting their feature done.
A common large company problem. People who should be talking are not. New hires are also hired into the "local" subculture of their team and something very, very important elsewhere in the company is not on their radar.
A common large company problem. People who should be talking are not. New hires are also hired into the "local" subculture of their team and something very, very important elsewhere in the company is not on their radar.
HR is not an organization that should be arbitrating technical issues that they don't know anything about.
The solution is for technical managers - at all levels, from team leads up to the CEO - to actually provide leadership to their teams. This can consist of training people about why it's important to do things right and setting up incentives to do things right (e.g., don't promote people who get things done quickly by cutting corners). To me, this story sounds like a classical management failure (been there, done that myself).
The solution is for technical managers - at all levels, from team leads up to the CEO - to actually provide leadership to their teams. This can consist of training people about why it's important to do things right and setting up incentives to do things right (e.g., don't promote people who get things done quickly by cutting corners). To me, this story sounds like a classical management failure (been there, done that myself).
That's dumb. If your system makes it possible for someone to screw up, someone will eventually screw up. And if screwing up (rather than being malicious or stupid) results in people getting fired, then no work will ever get done for fear of getting fired.
You have to fix the system so it's really hard to screw up.
You have to fix the system so it's really hard to screw up.
But I don't want them to burn down someone else's prod, I want them to keep building features for my company.
(Worse, if the other company has a misuse-resistant deployment system like the one the author alludes to, they might not burn down that company's prod and then beat us in the market while we're busy interviewing replacements!)
(Worse, if the other company has a misuse-resistant deployment system like the one the author alludes to, they might not burn down that company's prod and then beat us in the market while we're busy interviewing replacements!)
It seems like the system for sending out packages should've been designed to be unless this is being sent out as a critical security patch (and signed off by a security manager or something in the log), the package is deployed to 5% of the fleet per day (unless someone opts in to get it early for some reason on a package-by-package basis). In a big fleet, anything else would still be chaos if someone clicked the wrong button.
It seems to me there are lots of opinions on how to change an organization or introduce improvements but ultimately if you aren't making people's lives easier, making them want the change you are trying to bring about, you'll find it almost impossible to succeed, human inertia is just too great.