Microservices – Combinatorial Explosion of Versions(worklifenotes.com)
worklifenotes.com
Microservices – Combinatorial Explosion of Versions
https://worklifenotes.com/2020/03/04/microservices-combinatorial-explosion-of-versions/
16 comments
I did a stupid simple thing for a system with a small fanout footprint. Not sure this scales.
The organizing principal was that if my code passed the acceptance tests once, the very same code should still pass with your service/library/data added to the mix. If they don’t, it’s likely your code and not mine.
This might not seem like much but it cuts hours of finger pointing out of a three party system. I expect this effect would be multiplied for a five or ten party scenario.
And in particular, latency has a huge negative impact on behavioral modification. People will keep doing things that they get yelled at over three weeks later. They will stop doing things they get called on hours later.
Which I would hope offsets the additional test matrix complexity.
The organizing principal was that if my code passed the acceptance tests once, the very same code should still pass with your service/library/data added to the mix. If they don’t, it’s likely your code and not mine.
This might not seem like much but it cuts hours of finger pointing out of a three party system. I expect this effect would be multiplied for a five or ten party scenario.
And in particular, latency has a huge negative impact on behavioral modification. People will keep doing things that they get yelled at over three weeks later. They will stop doing things they get called on hours later.
Which I would hope offsets the additional test matrix complexity.
If I'm not mistaken, Pact is using a proprietary JSON format to define the contracts. I'm curious, is it possible to use it with a standard specification format, like OpenAPI/Swagger?
Thank you, great read and Pact looks interesting.
The diagram of the service in the blog post is absurd.
Anyone who designs a microservice environment like this should be fired. The interdependencies between services in the picture make the entire system fragile and everything will fail if a single service goes down.
A real production microservice environment should be designed so that interdependencies are limited and system wide failures won't occur if a service or two go down. Once you limit the interdependencies then the combinatorial explosion doesn't exist anymore. You might have some services that have a wide range of version used by clients, but you don't have the interdependency that complicates things.
Then, versioning isn't such a big deal.
Anyone who designs a microservice environment like this should be fired. The interdependencies between services in the picture make the entire system fragile and everything will fail if a single service goes down.
A real production microservice environment should be designed so that interdependencies are limited and system wide failures won't occur if a service or two go down. Once you limit the interdependencies then the combinatorial explosion doesn't exist anymore. You might have some services that have a wide range of version used by clients, but you don't have the interdependency that complicates things.
Then, versioning isn't such a big deal.
You cannot build software without dependencies.
The dependency pattern around microservices is either a contract on a queue/stream/topic or a version of an API from another depened service.
Don't care how many times you tell me that services should be interdepended and bla bla bla... In the end a service has dependencies and those dependencies must then have a versioning strategy so humans are not in doubt.
Versioning a microservice is no different than versioning a linked library or a Rest API.
Documentation is key. Tell people what versions break.
The dependency pattern around microservices is either a contract on a queue/stream/topic or a version of an API from another depened service.
Don't care how many times you tell me that services should be interdepended and bla bla bla... In the end a service has dependencies and those dependencies must then have a versioning strategy so humans are not in doubt.
Versioning a microservice is no different than versioning a linked library or a Rest API.
Documentation is key. Tell people what versions break.
Thank you, I very much second this. Another way of putting this - if we need to ensure identicity between any 2 environments - we have to care about fine details and exact versions.
Otherwise, we're back at "works on my machine" mentality.
And yes - trying to make microservices independent from each other is also a form of pruning and sometimes works well to a point, but also requires good tooling to be done right.
Otherwise, we're back at "works on my machine" mentality.
And yes - trying to make microservices independent from each other is also a form of pruning and sometimes works well to a point, but also requires good tooling to be done right.
My point is your article is based on a fake premise, or a poorly designed premise. Your architecture that creates this combinatorial explosion is only because your supposed microservice architecture is poorly designed. If you look at a real microservice architecture, it won't be nearly as complicated and it's not as big of a deal to version.
> My point is your article is based on a fake premise, or a poorly designed premise.
That's a strong statement which you should at least try to provide counter-example to support - just saying "you design it wrong" is not enough. I've seen this kind of problem irl left-right-and-center - so I take it there is an issue, and it's not happening because "everybody doing it wrong".
> If one service goes down, they all go down
I believe you don't understand the diagram. Lines are not dependencies, but a way to connect components of different versions into single product. (I.e., you pick either v1 or v2 of each component, and it becomes single product along the lines - it doesn't necessarily mean there is hard dependency).
To my point, I treat whole architecture as a product. I don't necessarily speak about dependencies. Instead, I'm taking general view of this as a math problem - first of all, I establish search space - that is number of available versions to the power of number of microservices.
Then I very much support and want to discuss various ways to reduce this search space via pruning - and what you're talking about is just one of the options how to prune it (by reducing dependencies). But as others pointed (rephrasing in my words), you can apply a greedy algorithm to NP-problem, but first of all understand what the real problem you're dealing with is and second of all realize that your algorithm is greedy - meaning it may have flaws in the edge cases and it's better to be prepared to those.
In your specific case I claim that you can't be completely sure that you actually don't have any interdependency between components - I believe it would be impossible to prove for any system. And again I saw really hard bugs irl coming from those assumptions. Recent example (this is not about microservices - but pls try to solve this case): https://stackoverflow.com/questions/60486853/aws-ecr-uploadi... - Tools in question must be completely independent, but somehow they are not. So assuming something is completely independent is frequently dangerous.
That's a strong statement which you should at least try to provide counter-example to support - just saying "you design it wrong" is not enough. I've seen this kind of problem irl left-right-and-center - so I take it there is an issue, and it's not happening because "everybody doing it wrong".
> If one service goes down, they all go down
I believe you don't understand the diagram. Lines are not dependencies, but a way to connect components of different versions into single product. (I.e., you pick either v1 or v2 of each component, and it becomes single product along the lines - it doesn't necessarily mean there is hard dependency).
To my point, I treat whole architecture as a product. I don't necessarily speak about dependencies. Instead, I'm taking general view of this as a math problem - first of all, I establish search space - that is number of available versions to the power of number of microservices.
Then I very much support and want to discuss various ways to reduce this search space via pruning - and what you're talking about is just one of the options how to prune it (by reducing dependencies). But as others pointed (rephrasing in my words), you can apply a greedy algorithm to NP-problem, but first of all understand what the real problem you're dealing with is and second of all realize that your algorithm is greedy - meaning it may have flaws in the edge cases and it's better to be prepared to those.
In your specific case I claim that you can't be completely sure that you actually don't have any interdependency between components - I believe it would be impossible to prove for any system. And again I saw really hard bugs irl coming from those assumptions. Recent example (this is not about microservices - but pls try to solve this case): https://stackoverflow.com/questions/60486853/aws-ecr-uploadi... - Tools in question must be completely independent, but somehow they are not. So assuming something is completely independent is frequently dangerous.
You didn't understand what I wrote. I said that having interdependent services the way the diagram was drawn leads to an extremely fragile system. If one service goes down, they all go down. It's not a realistic environment or it's a very poorly designed one. If you simplify the diagram to one that is more realistic, with more 1:1 or N:1 relationships, then the combinatorial factor decreases, and then there is no article to write about.
How does versioning differ based on a 1:1 or a 1:N, or a N:N?
It doesn't.
No matter how few or many dependencies you have, when they are updated, the dependent system must either update or not update.
There is nothing new in regards to versioning of dependencies. It has always been hard to do, and with microservices the versioning story has not all of a sudden improved.
What people that are new to versioning, seen from a somewhat autonomous systems perspective, this is something you must have a strategy for.
It doesn't.
No matter how few or many dependencies you have, when they are updated, the dependent system must either update or not update.
There is nothing new in regards to versioning of dependencies. It has always been hard to do, and with microservices the versioning story has not all of a sudden improved.
What people that are new to versioning, seen from a somewhat autonomous systems perspective, this is something you must have a strategy for.
That's ridiculous. Just don't break anything, ever. If you need a service to behave differently, you give it a different name and roll it out and wait for traffic on the original to fall to zero and then you wait a bit longer and them you turn it off and delete it. People who advocate for versioned services just do. not. get it.
Ehh, what you just described is often how service versioning works: launch a v2 service with the v1 still active, keep them running concurrently for a while, then turn off the v1 when it's no longer used.
You can distinguish the versions by using different endpoints, version fields, version headers, etc.. But that's just implementation details.
You can distinguish the versions by using different endpoints, version fields, version headers, etc.. But that's just implementation details.
Unfortunately however most developers (even senior ones) follow fads without proper homework and planning.
A lesson I learned working with microservices that are deployed independently of each other is that you need a policy that ALL changes must be backwards compatible with existing clients.
This sounds crazy, but it's essential if you can't guarantee that you can atomically deploy a new service version AND any other services that call it.
Creating backwards compatible releases means sticking to some rules, things like:
- you can add fields but you can never delete them
- you can add new methods but you can't remove old ones
Having really detailed logging helps a lot. If you want to deprecate a method you can remove it from calling clients first and then use the logs to confirm it isn't being called any more before removing it from the service.
It's a lot if work, and teaching a large engineering team how to be productive in this kind of environment is decidedly non trivial.
This sounds crazy, but it's essential if you can't guarantee that you can atomically deploy a new service version AND any other services that call it.
Creating backwards compatible releases means sticking to some rules, things like:
- you can add fields but you can never delete them
- you can add new methods but you can't remove old ones
Having really detailed logging helps a lot. If you want to deprecate a method you can remove it from calling clients first and then use the logs to confirm it isn't being called any more before removing it from the service.
It's a lot if work, and teaching a large engineering team how to be productive in this kind of environment is decidedly non trivial.
Never say never.
Adding fields or methods is easy. You can do it at any time.
Removing them is difficult. You must first go through a long deprecation cycle and ensure ALL clients are updated.
You need good logging AND good monitoring and analysis of the logs in both client and server to be able to detect access to removed features.
Adding fields or methods is easy. You can do it at any time.
Removing them is difficult. You must first go through a long deprecation cycle and ensure ALL clients are updated.
You need good logging AND good monitoring and analysis of the logs in both client and server to be able to detect access to removed features.
A coworker pointed out that this is a huge point in favour of GraphQL: because it forces clients to explicitly ask for every field you can log those queries and use them to spot when a field is no longer in use.
Actually, if the server knows the client version, he can downgrade returned data to the client in the version he expects. That way, adding, renaming and removing fields should not be a showstopper, assuming methods stay as generic as possible.
Alternatively you could provide different API versions at the same time and keep the old ones running, as long as they are called.
Alternatively you could provide different API versions at the same time and keep the old ones running, as long as they are called.
While you can build machinery to do this. It is far easier to not. If you substantially want to change your api, just make a new one. Much easier to track migration between endpoints than it is versions in an endpoint.
And, in general, don't focus on generic methods.
The more specific a service method, the better. The less it does, the better. Don't have an "order pizza" api. Have a "start order", "add to order", and "complete order" api. If it helps, have multiple add methods. The parameters to add a pizza can and should be different from a salad. Even the pizza may be better served with partial service calls to start and complete.
And, in general, don't focus on generic methods.
The more specific a service method, the better. The less it does, the better. Don't have an "order pizza" api. Have a "start order", "add to order", and "complete order" api. If it helps, have multiple add methods. The parameters to add a pizza can and should be different from a salad. Even the pizza may be better served with partial service calls to start and complete.
The typical approach to catching this that I have seen is to deploy the new version of Service X to only a small percentage of all machines for a certain amount of time. E.g. 1% of machines get v1.1 while all others stay on v1.0. You then monitor error rates in v1.1 (and its dependencies) and stop if there are elevated levels, or crank the release up slowly if the error levels aren't elevated. Obviously some automation in your release management and monitoring tools helps a lot with this.
Of course this does not "solve" the problem of incompatibilities themselves - for that the simple solution that I've seen used is just bog standard versioning of APIs (/v1/... /v2/... /v2019-03-04/... etc) and some procedures for managing the supported versions, e.g. not doing breaking changes in an existing version, only supporting a set number of versions (i.e. previous, current, and next), and having proper sunset-periods for older versions before they are turned off so clients have time to update. Slows velocity to do all this management but that is life anyway in a complex system
Of course this does not "solve" the problem of incompatibilities themselves - for that the simple solution that I've seen used is just bog standard versioning of APIs (/v1/... /v2/... /v2019-03-04/... etc) and some procedures for managing the supported versions, e.g. not doing breaking changes in an existing version, only supporting a set number of versions (i.e. previous, current, and next), and having proper sunset-periods for older versions before they are turned off so clients have time to update. Slows velocity to do all this management but that is life anyway in a complex system
Don't monorepos solve this? Lets presume I'm not a FAANG company (though some of those have monorepos) with a bazillion services, I'm just a startup with say 10 services. At that scale I think its pretty realistic to maintain a monorepo for all 10 services. That way every release only upgrades the version count by one.
A startup has no business running microservices. Microservices are only when the engineering team can't scale properly because they keep stepping on top of each other.
A startup should be using a monolith until it becomes a victim of its own success and then should migrate thoughtfully to microservices only when it has to. It also requires a very heavy investment in devops and tools since microservices will fail in production in every which way possible. It also requires a huge investment in metrics and alerts and logging otherwise you will have no idea what is wrong with your now too-complicated production system.
A startup should be using a monolith until it becomes a victim of its own success and then should migrate thoughtfully to microservices only when it has to. It also requires a very heavy investment in devops and tools since microservices will fail in production in every which way possible. It also requires a huge investment in metrics and alerts and logging otherwise you will have no idea what is wrong with your now too-complicated production system.
> A startup has no business running microservices. Microservices are only when the engineering team can't scale properly because they keep stepping on top of each other.
While I generally agree with you in the vast majority of practical cases, it can make sense in some circumstances, namely, if there is a very clear logical abstraction between one service and another.
A microservice is just another layer of abstraction, similar to functions, classes, files, libraries, and programs, but even higher up the food-chain, and it comes with its own characteristics and peculiarities. If you have high confidence that one component is (1) logically extremely different from all of the other components, (2) has no side effects, (3) is not I/O bound as it's going over a network, and critically, (4) is on a separate development timeline, then you have a solid candidate for a microservice.
While I generally agree with you in the vast majority of practical cases, it can make sense in some circumstances, namely, if there is a very clear logical abstraction between one service and another.
A microservice is just another layer of abstraction, similar to functions, classes, files, libraries, and programs, but even higher up the food-chain, and it comes with its own characteristics and peculiarities. If you have high confidence that one component is (1) logically extremely different from all of the other components, (2) has no side effects, (3) is not I/O bound as it's going over a network, and critically, (4) is on a separate development timeline, then you have a solid candidate for a microservice.
Do you know the domain of the business? Otherwose it sounds like you should reconsider the advice you give out.
My statement applies across the board. It doesn't matter what the domain of the business is. Microservices cost more and have a higher impact on productivity and is only worth it if your engineering team needs the flexibility, at the cost of increase devops investment and more down time.
No.
Ok. Once you have real world experience, then you should continue commenting.
They do in part but developers need to be aware of two major things:
1. Always be backwards compatible in protocols and data formats.
If you store data on disk or in a table you V1 and V2 tables need to be generally compatible.
V2 should always be able to read V1 tables. This helps with rollouts.
In an ideal case V1 should gracefully handle V2 tables. This helps with rollbacks.
2. Service to service communications work between V1 and V2.
If you don't want downtime you have to gracefully handle the situation in which V2 is rolling out. In a zero-downtime deployment config like blue/green V2 instances can get requests from V1 instances and vice versa.
Monorepos don't help with these situations. These are just engineering practices that have to be upheld by the team. Monorepos will make the atomic source code updates applied across the codebase easy.
1. Always be backwards compatible in protocols and data formats.
If you store data on disk or in a table you V1 and V2 tables need to be generally compatible.
V2 should always be able to read V1 tables. This helps with rollouts.
In an ideal case V1 should gracefully handle V2 tables. This helps with rollbacks.
2. Service to service communications work between V1 and V2.
If you don't want downtime you have to gracefully handle the situation in which V2 is rolling out. In a zero-downtime deployment config like blue/green V2 instances can get requests from V1 instances and vice versa.
Monorepos don't help with these situations. These are just engineering practices that have to be upheld by the team. Monorepos will make the atomic source code updates applied across the codebase easy.
Every place you said "should" is actually "must" otherwise how can anything work? Clearly the next version MUST be able to read the stored data of the current version and just as clearly the next version MUST write data that can be read by the current version. If not then releasing the software is totally impossible.
Monorepo can help you in tracking dependencies between services at build time, however services are still executed independently in production, thus existing running services will have a dependency to older versions. To solve that without versioning you then have to shutdown all of them & wait until all off them are online again with the latest commit of your monorepo. Now you have created a network distributed monolith.
An issue you might face is that during rollouts, you will still have some servers running the old binary and some running the new binary. So you must either:
1. Stop all old servers before starting the new ones -> downtime, might be acceptable 2. Have your clients handle the fact that they might hit vN or vN+1 of your services.
1. Stop all old servers before starting the new ones -> downtime, might be acceptable 2. Have your clients handle the fact that they might hit vN or vN+1 of your services.
Thank you for the comment - monorepos indeed make things better to a degree. From one perspective, you now can point your Product (package) to the whole monorepo.
But in reality, you're still facing same issue - let's say your components X and Y are updated in monorepo. You want X in production, but not Y - so now you have to invent something on top of monorepo to capture this. Another words back to square 1.
Regarding startups, in my experience they start feeling this pain when they already have 5-6 microservices. But YMMV - depends on a lot of factors.
But in reality, you're still facing same issue - let's say your components X and Y are updated in monorepo. You want X in production, but not Y - so now you have to invent something on top of monorepo to capture this. Another words back to square 1.
Regarding startups, in my experience they start feeling this pain when they already have 5-6 microservices. But YMMV - depends on a lot of factors.
Yes, perhaps a better way to think about it is to version the set of services as a whole, and deploy them as a holistic, singular product. Then you don't run into the combinatorics problems.... Just challenges with roll backs.
what? i don’t understand how this has anything to do with micro services vs monorepo, if you are using either and you update a particular something what is different between the two
All of the content in a monorepo is versioned together, while microservices have their own independent versioning schemes and for this reason their interfaces can drift.
I guess it would only help if the dependencies can be statically analyzed somehow - e.g. a change in X causes a compiler error in Y etc etc - or you have really really really comprehensive integration tests.
In my experience this is not always the case and the problems creep in in other places that are not easily statically checked or tested, but which come up at runtime, e.g. specific runtime content/data, or serialisation of some message/object that happens at runtime.
In my experience this is not always the case and the problems creep in in other places that are not easily statically checked or tested, but which come up at runtime, e.g. specific runtime content/data, or serialisation of some message/object that happens at runtime.
No, monorepos don't solve this at all. It's not even related. RPC services are described by IDL and you can't ever break the semantics of the service for any client that was ever using any released version of the IDL. Full stop. The is the golden rule of RPC services and it's also the only rule! You can do anything else, you can completely replace the service with a scratch rewrite in Erlang as long as you don't break the one rule.
Don't monorepos have an adjacent problem as well? Feature-flags tend to create a combinatorial explosion of potential code-paths in your system.
Few points based on my experiences with (Micro) services:
Note: None of these are cast in stone. These have worked well for us
1. Decide in your team/org what constitutes a new version and what it means. Does every change means version change? What if an API adds an optional parameter; do I need to update the version?
2. Not everything has to be a service. Think if the same functionality can be consumed in the form of a library/sdk instead of a full service.
3. The dependency amongst services should resemble acyclic graph and not cycles. This limits the version change impact.
4. Think about abstraction. Can we logically group set of services A, B & C and provide unified API via pass-through service D? Only D needs to handle API breakage most of the time.
Note: None of these are cast in stone. These have worked well for us
1. Decide in your team/org what constitutes a new version and what it means. Does every change means version change? What if an API adds an optional parameter; do I need to update the version?
2. Not everything has to be a service. Think if the same functionality can be consumed in the form of a library/sdk instead of a full service.
3. The dependency amongst services should resemble acyclic graph and not cycles. This limits the version change impact.
4. Think about abstraction. Can we logically group set of services A, B & C and provide unified API via pass-through service D? Only D needs to handle API breakage most of the time.
If you have this problem, you don't have microservices -- you have a distributed monolith.
If each service can talk to each other, the number of possible communication paths grows exponentially -- this is manageable up to a point, but even the largest teams will have to start investing into orchestration sooner or later.
There are many orchestration patterns and mechanisms, and this is not the place to list them all -- which one is the right choice will depend on the complexity and needs of the system.
If each service can talk to each other, the number of possible communication paths grows exponentially -- this is manageable up to a point, but even the largest teams will have to start investing into orchestration sooner or later.
There are many orchestration patterns and mechanisms, and this is not the place to list them all -- which one is the right choice will depend on the complexity and needs of the system.
This is not about microservices talking to each other, but rather about seeing customer-facing product as a whole with all its different components and their versions.
And yes we can agree that there are various (usually "greedy") strategies to simplify this - but this problem is a fact of life and it makes thinking process easier to accept it as such. Same as accepting that it is impossible to solve consensus in async system, but there are algorithms that perform fairly well.
+ I summed this up in more details in the comments down below.
And yes we can agree that there are various (usually "greedy") strategies to simplify this - but this problem is a fact of life and it makes thinking process easier to accept it as such. Same as accepting that it is impossible to solve consensus in async system, but there are algorithms that perform fairly well.
+ I summed this up in more details in the comments down below.
Turns out the latest hype is a dud for most companies ^_^
There are good reasons to use microservices, like when you have 100+ engineers working on your planet-scale cloud or social network. At that scale, you need to have internal API documentation to ensure people that never met can still work together productively.
But if you are a normal company, you will probably never have enough engineers to make microservices worthwhile in the first place. So just put everything into one big git repo and call it a day.
Also, the ability to do rolling restarts and no downtime upgrades by running multiple versions in parallel adds a lot of work and complexity. But for most small to medium companies, a planned 5 minute downtime in the middle of the night is completely no problem, so all that zero-downtime-upgrade work is just wasted effort.
I mean, even my bank has a fixed offline maintenance window every night from 3:00 to 3:30 am. As does Amazon RDS.
So don't solve problems that you don't have :) Most companies do fine without microservices.
There are good reasons to use microservices, like when you have 100+ engineers working on your planet-scale cloud or social network. At that scale, you need to have internal API documentation to ensure people that never met can still work together productively.
But if you are a normal company, you will probably never have enough engineers to make microservices worthwhile in the first place. So just put everything into one big git repo and call it a day.
Also, the ability to do rolling restarts and no downtime upgrades by running multiple versions in parallel adds a lot of work and complexity. But for most small to medium companies, a planned 5 minute downtime in the middle of the night is completely no problem, so all that zero-downtime-upgrade work is just wasted effort.
I mean, even my bank has a fixed offline maintenance window every night from 3:00 to 3:30 am. As does Amazon RDS.
So don't solve problems that you don't have :) Most companies do fine without microservices.
> There are good reasons to use microservices, like when you have 100+ engineers
Its enough to have 2 that work in different domains with different constraints and different tools. Company scale is completely irrelevant here.
> But if you are a normal company, you will probably never have enough engineers to make microservices worthwhile in the first place
Many "normal" companies use microservices and common reason is that they simply cannot function otherwise.
> But for most small to medium companies, a planned 5 minute downtime in the middle of the night is completely no problem,
The fact that some site doesn't function for 5 minutes may not be a problem, the fact that any change requires many people agreeing on deployment time often is.
Its enough to have 2 that work in different domains with different constraints and different tools. Company scale is completely irrelevant here.
> But if you are a normal company, you will probably never have enough engineers to make microservices worthwhile in the first place
Many "normal" companies use microservices and common reason is that they simply cannot function otherwise.
> But for most small to medium companies, a planned 5 minute downtime in the middle of the night is completely no problem,
The fact that some site doesn't function for 5 minutes may not be a problem, the fact that any change requires many people agreeing on deployment time often is.
Its enough to have 2 that work in different domains with different constraints and different tools.
Wouldn't that be more or less be just two monoliths?
Wouldn't that be more or less be just two monoliths?
If they both work for the same organization to achieve common goal, communicate through some interface they agree on, having to coordinate occasional changes that affect both codebases, than no.
If they are completely isolated, than yes, but its really rare scenario.
A common example would be a website that gathers/presents some data from users, and a service that provides analytics of said data. Both domains may require completely different technologies.
If they are completely isolated, than yes, but its really rare scenario.
A common example would be a website that gathers/presents some data from users, and a service that provides analytics of said data. Both domains may require completely different technologies.
Agree, it seems silly to bash microservices as if all (new) companies decide to use them. They are a tool for medium and bigger companies to solve the problems those companies have. Fundamentally it's adding some complexity to the overall system by breaking it into components that have smaller scope and are easier for fewer people to wrangle. So one reason services can make sense when the complexity of an initial monolith has grown beyond what the average engineer can be expected to learn in a few months; or if a similar growth trajectory is happening because the engineering team is big enough to write code faster than everyone can keep up with understanding the important changes.
You can have services, just not micro. Splitting into few services that are obviously conceptually separate is enough. If you are a typical web company, you can have a dedicated service for search, maybe a separate service for internal tools and the main app. Granular enough that teams in charge can be focused on on their area and wide enough to avoid complexity.
Imo, that is identical to "microservices" but you're right that there's no reason to go any more than what you need. I always felt the whole microservice hype/movement was simply about defining multiple services as a useful pattern you'll want to use and how to build infrastructure to make it easy.
But I agree wholeheartedly. You don't need 100+ services to get the benefits of converting your tool chain to handle deploying n services even if n=3.
But I agree wholeheartedly. You don't need 100+ services to get the benefits of converting your tool chain to handle deploying n services even if n=3.
In that case I'd say it's unlikely you are truly embracing microservices in the manner that their evangelists intended (and don't misunderstand me, you are probably right not to). One of the characteristics of microservices is that they are small enough that they can be completely written in a couple of weeks.
Like every other fad in technology the nuances and practicalities are lost along the way.
Now that we have all started bashing microservices can we all take a minute to reflect on what a steaming pile of shit monoliths can turn into without a huge amount of respect for the artificial boundaries and interfaces that you create.
Like every other fad in technology the nuances and practicalities are lost along the way.
Now that we have all started bashing microservices can we all take a minute to reflect on what a steaming pile of shit monoliths can turn into without a huge amount of respect for the artificial boundaries and interfaces that you create.
I've always felt the "micro" terminology just meant that a service might be that small and that's ok. I don't think of it as a requirement. I don't really read any comments recommending I break up services more than I need. Usually its a conversation about "hey sometimes its better to have n services and you should spend a little time making that easy."
Where was it stipulated that a characteristic of a microservice is that it can be completely written in a couple of weeks?
If I remember correctly it's mentioned in Building Microservices - Sam Newman. I confess I don't have it on me at the moment to confirm that though.
Thanks for the reference.
I think prefix “micro“ confuses me :) Opens up a debate as to how granular you need to go. Another thing that confuses me is whether microservice architecture is conceptually different to SOA. Some say it is, some view it as another iteration of SOA.
> But for most small to medium companies, a planned 5 minute downtime in the middle of the night is completely no problem, so all that zero-downtime-upgrade work is just wasted effort.
The first half of that is true, but the conclusion overlooks a couple of much bigger issues:
1. There are few indicators of effective software engineering so strong as deployment frequency and lead time for changes. Being able to go from issue to fix in production in minutes rather than days opens up completely new paths for feedback and allows you to operate your service much more efficiently. (See the yearly state of devops reports for further research about this.)
2. Not all downtime is planned downtime. Being able to handle a true crash gracefully is very important to achieve reasonable response times in the high nines.
I'm not a proponent of service-oriented architectures -- my opinion is that enforcing API boundaries is useful, but can be done at the source level rather than application level. I do however think there is great value to being able to deploy/restart your application with virtually no downtime.
The first half of that is true, but the conclusion overlooks a couple of much bigger issues:
1. There are few indicators of effective software engineering so strong as deployment frequency and lead time for changes. Being able to go from issue to fix in production in minutes rather than days opens up completely new paths for feedback and allows you to operate your service much more efficiently. (See the yearly state of devops reports for further research about this.)
2. Not all downtime is planned downtime. Being able to handle a true crash gracefully is very important to achieve reasonable response times in the high nines.
I'm not a proponent of service-oriented architectures -- my opinion is that enforcing API boundaries is useful, but can be done at the source level rather than application level. I do however think there is great value to being able to deploy/restart your application with virtually no downtime.
> 1. There are few indicators of effective software engineering so strong as deployment frequency and lead time for changes. Being able to go from issue to fix in production in minutes rather than days opens up completely new paths for feedback and allows you to operate your service much more efficiently
This is only true under certain assumptions. E.g. if your business is not a public facing website, you often don't have the immediate feedback in case of errors, to make the above observation applicable. I have a customer with an application that gather date for a whole year, and then does alot of stuff with it. Then, there are no short feedback cycles in production, no matter how often a day you deploy. Then, it is important to assure quality before a change reaches production. Otherwise it might replace your data with background noise and you only discover it at the end of the next year.
This is a symptom of the general tendency in developer circles to assume that personal findings apply to everyone at every time in every situation. Which is obviously ridicoulous. But somehow completely acceptable if you generalize from something one Google employer has posted about.
This is only true under certain assumptions. E.g. if your business is not a public facing website, you often don't have the immediate feedback in case of errors, to make the above observation applicable. I have a customer with an application that gather date for a whole year, and then does alot of stuff with it. Then, there are no short feedback cycles in production, no matter how often a day you deploy. Then, it is important to assure quality before a change reaches production. Otherwise it might replace your data with background noise and you only discover it at the end of the next year.
This is a symptom of the general tendency in developer circles to assume that personal findings apply to everyone at every time in every situation. Which is obviously ridicoulous. But somehow completely acceptable if you generalize from something one Google employer has posted about.
There's a point to what you're writing that I hadn't considered, but I'd still hold that building "software used every day, or at least every week" constitutes far more of our collective engineering time than "software used once a year or more rarely."
Even if the latter category is more numerous, I think we spend much less time working on it. And software that's used semi-frequently can benefit from a shorter feedback cycle, even if it doesn't have one right now. Sometimes there's an insurmountable technical barrier to that (e.g. code on a solar orbiter) but it's even in those cases worth investigating.
Even if the latter category is more numerous, I think we spend much less time working on it. And software that's used semi-frequently can benefit from a shorter feedback cycle, even if it doesn't have one right now. Sometimes there's an insurmountable technical barrier to that (e.g. code on a solar orbiter) but it's even in those cases worth investigating.
> Also, the ability to do rolling restarts and no downtime upgrades by running multiple versions in parallel adds a lot of work and complexity.
You don't need (micro)services to do upgrades without downtime. You just need a couple of load-balanced servers instead of one, and some planning to be sure that the two versions can coexists for some time.
This can be a bit tricky until you get used to it, but depending on the company and the customers it can be even easier than organizing a maintenance window.
You don't need (micro)services to do upgrades without downtime. You just need a couple of load-balanced servers instead of one, and some planning to be sure that the two versions can coexists for some time.
This can be a bit tricky until you get used to it, but depending on the company and the customers it can be even easier than organizing a maintenance window.
To be truly useful at smaller scale, most of the services needs to be third party components you just drop in, and we need near zero effort orchestration first so splitting up your services doesn't just shift the burden to devops.
Neither is really there, and so I agree with you today, but it could change with the right tooling.
Neither is really there, and so I agree with you today, but it could change with the right tooling.
Very good point, thank you. My take is we're actually getting there pretty quickly and 3rd party drop-in components where you can mix and match is indeed the future.
or the past ;-)
If you have a mono-repo and a maintenance window you could just make cross service updates in a single commit and do the maintenance deploy of all services too. Microservices alone don't prevent any of that.
Important nuance in your comment why microservices are useful:
3:00-3:30 in your timezone ;)
3:00-3:30 in your timezone ;)
It's still possible to do live upgrades without microservices
Is this a real problem people deal with? People actually build these spaghetti services and deploy several versions of them at once? Each of these versions are somehow routable to and from each other version with no abstraction?
Surely this doesn't actually happen.
Surely this doesn't actually happen.
Consumer driven tests can help with this too. Rather than just service providers writing tests, service consumers write them too, which are then given to the service provider as part of their test suite.
I don’t see how this is a problem. I see that the combinatorial explosion exists, but that isn’t an actual problem. The entire point of microservices is independent deploys, meaning you literally only have two permutations per deploy, new and old. If you think deploying 10 services at once is feasible, you don’t understand the appral of microservices, and should never have opted for them in the first place.
Deploying one at a time may be fine if you always test that each one is working in backward compatible manner. However you still have same search space and same problem as before.
Imagine
a - you have 10 microservices, 1 update per each. Each supposed to be backward-compatible. You start rolling out. One by one. 5 go fine, 6th breaks. You end up in a weird state where 5 out of 10 are updated.
You hope it's fine due to backward compatibility but you never really tested this config exactly. (Which is why I might prefer either converging to deploying all 10 or rolling back fully - if that was a known good state - and having canary cluster rather than canary microservice in many cases).
b - same as above but now you try to catch this behaviour on test / staging. You still have same hard problem at hands.
Key here is you clearly can't try every possible variation of what may break, so need to make conscious decisions about what to do in the case of failure.
Imagine
a - you have 10 microservices, 1 update per each. Each supposed to be backward-compatible. You start rolling out. One by one. 5 go fine, 6th breaks. You end up in a weird state where 5 out of 10 are updated.
You hope it's fine due to backward compatibility but you never really tested this config exactly. (Which is why I might prefer either converging to deploying all 10 or rolling back fully - if that was a known good state - and having canary cluster rather than canary microservice in many cases).
b - same as above but now you try to catch this behaviour on test / staging. You still have same hard problem at hands.
Key here is you clearly can't try every possible variation of what may break, so need to make conscious decisions about what to do in the case of failure.
Although it's an interesting calculation, you rarely have all your microservices communicating with eachother since you tend to group by areas.
And even if that was the case, if good practices are in place (semantic versioning, good test cases, etc) I don't see why that should be a problem.
in my experience, most service API grow by accretion and need to break backward compatibility very very rarely. Naturally, clients should always be Tolerant Readers.
So in many case, you will have only a single version per service.
Moreover, imo, when we are talking about internal apis, there should be nothing preventing the service owner from updating the consumers if he wishes to converge more quickly after breaking compatibility - just like he would when refactoring a monolith. The culture should allow and encourage this kind of collaboration.
So in many case, you will have only a single version per service.
Moreover, imo, when we are talking about internal apis, there should be nothing preventing the service owner from updating the consumers if he wishes to converge more quickly after breaking compatibility - just like he would when refactoring a monolith. The culture should allow and encourage this kind of collaboration.
This is an interesting void for micro services that’s not yet being met.
if you:
Deploy new versions of new services and quickly rollout the number of combinations get manageable.
Making he interface of services backwards compatible, either by testing and/or by using stricter contracts like grpc (which had a way to deal with compatibility)
Then you're mostly covered. However integration testing indeed makes sense
Making he interface of services backwards compatible, either by testing and/or by using stricter contracts like grpc (which had a way to deal with compatibility)
Then you're mostly covered. However integration testing indeed makes sense
Microservices only have two versions you should worry about: the one that's currently live and the next one. Two for each micro service you are responsible for. IMHO roll backs are not a thing. Instead git revert and roll forward when you need to deploy a fix. So once a service has been fully deployed, you permanently stop caring about the previous version.
Microservices should not be mass deployed together but be managed separately. That vastly simplifies the combinatorial explosion of versions you need the worry about: the currently live ones.
The rest is just applying SOLID principles to your microservices and avoid having services with poor cohesiveness or tight coupling (most of the SOLID principles boil down to affecting these two metrics). Bad service design with tight coupling where deploying service A also requires redeploying B,C,and D, is basically a design problem and not a micro services problem. There's a lot of bad design in our industry. Monoliths allow you to get away with that but it's also the reason that breaking them up is a hard problem. Just because you are getting away with it does not mean it is not a problem though.
You can further mitigate integration issues by deploying using modern practices like blue green deployments where you gradually move traffic to a new versions and adapt on things like error rates and other metrics, AB testing, etc. So if it breaks, you don't end up breaking it for everyone and you can fix the problem and try deploying that in a controlled way.
Basically your goal is to keep your customers (aka. dependees) happy and make sure you don't negatively affect them (which you should be actively monitoring). Likewise if you have dependencies (to whom you are a customer) and they stop working when you deploy something new, you probably want to detect and fix that before you break all your customers. If that is a regular thing, consider having integration tests, contract tests, etc.
Staging environments are a controversial topic in this context. My view is that they don't make much sense in a properly run micro service deployment since you are not testing in a real environment with real users, real data, and lots of things happening concurrently and features interacting. And of course your customers doing real things they care about. The only realistic environment that has that in most complex micro service deployments is called production. Once you add serverless and edge computing to the mix, these things become even more true. The bigger the organization, the less feasible it becomes to have a staging environment.
Update. I forgot to add this but doing continuous deployment means small deltas that are low risk. Any bigger change can be behind a feature flag. There are a few more strategies.
Also, I'm not actually a microservices proponent for small teams. It just creates deployment and operational overhead (read go to market bottlenecks).
Microservices should not be mass deployed together but be managed separately. That vastly simplifies the combinatorial explosion of versions you need the worry about: the currently live ones.
The rest is just applying SOLID principles to your microservices and avoid having services with poor cohesiveness or tight coupling (most of the SOLID principles boil down to affecting these two metrics). Bad service design with tight coupling where deploying service A also requires redeploying B,C,and D, is basically a design problem and not a micro services problem. There's a lot of bad design in our industry. Monoliths allow you to get away with that but it's also the reason that breaking them up is a hard problem. Just because you are getting away with it does not mean it is not a problem though.
You can further mitigate integration issues by deploying using modern practices like blue green deployments where you gradually move traffic to a new versions and adapt on things like error rates and other metrics, AB testing, etc. So if it breaks, you don't end up breaking it for everyone and you can fix the problem and try deploying that in a controlled way.
Basically your goal is to keep your customers (aka. dependees) happy and make sure you don't negatively affect them (which you should be actively monitoring). Likewise if you have dependencies (to whom you are a customer) and they stop working when you deploy something new, you probably want to detect and fix that before you break all your customers. If that is a regular thing, consider having integration tests, contract tests, etc.
Staging environments are a controversial topic in this context. My view is that they don't make much sense in a properly run micro service deployment since you are not testing in a real environment with real users, real data, and lots of things happening concurrently and features interacting. And of course your customers doing real things they care about. The only realistic environment that has that in most complex micro service deployments is called production. Once you add serverless and edge computing to the mix, these things become even more true. The bigger the organization, the less feasible it becomes to have a staging environment.
Update. I forgot to add this but doing continuous deployment means small deltas that are low risk. Any bigger change can be behind a feature flag. There are a few more strategies.
Also, I'm not actually a microservices proponent for small teams. It just creates deployment and operational overhead (read go to market bottlenecks).
Pretty much yes on all of this. I don't mind rollbacks though - there are issues with rolling back but if I can, I will. If there is a database migration, I typically will roll forward as you say.
Staging envs are generally a crutch. They can help but they can also hurt.
Staging envs are generally a crutch. They can help but they can also hurt.
From the Pact docs: 'Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a "contract".' By focussing just on the messages we get tests which are fast, give us quick feedback, and scale linearly instead of combinatorially.
Some good resources are:
https://pact.io (for information about contract testing and the Pact tool itself)
https://pactflow.io/how-pact-works/ (explains how Pact works)
https://docs.pact.io/faq/convinceme (answers the question of why you would want to do contract testing)
https://slack.pact.io (a friendly 1000+ member community which is very experienced dealing with these kinds of issues)
https://docs.pact.io/pact_broker/can_i_deploy (addresses how we handle and channel this combinatorial explosion for good instead of evil!)