Inside Azure Search: Chaos Engineering(azure.microsoft.com)
azure.microsoft.com
Inside Azure Search: Chaos Engineering
http://azure.microsoft.com/blog/2015/07/01/inside-azure-search-chaos-engineering
1 comments
Hi, author of the post here. Happy to answer any questions, would love to hear from any of you who have used similar approaches to managing failure in distributed systems.
How do other Azure teams inside Microsoft do their testing (SQL Azure, Azure DocDB etc?) I'd assume there's a common platform that allows you to just throw stuff and inject failures? Further, in production, what percent of bugs if any did you find were not caught by Chaos engineering? Where they of a particular class? Did you still see a necessity for functional tests that test all permutations diligently?
Regarding testing, I think that static functional tests that verify permutations of a specific set of inputs are still essential, in the same way that unit tests are needed even when you have comprehensive integration test coverage.
Like the old axiom, the earlier you find a bug, the cheaper it is to fix. The Search Chaos Monkey tests run against the current state of the repository, are not very precise and are relatively expensive and time-consuming to run. I only know when a test has failed because we start getting notifications that the service is unavailable, or has degraded performance. I still need to go dig up logs to see how the fault actually impacted the system, then go file a bug against code that's already been committed. If I spend 15 minutes digging through logs only to find that an error was caused by a bug in Azure Search code that could have been caught at a lower level of testing before that code was checked in, I won't be very happy.
The monkey is there to test how our code works as the underlying cluster changes states and experiences failures. Before any code gets passed to the monkey, we should be fairly confident through rigorous functional tests that it performs correctly in most circumstances.
Like the old axiom, the earlier you find a bug, the cheaper it is to fix. The Search Chaos Monkey tests run against the current state of the repository, are not very precise and are relatively expensive and time-consuming to run. I only know when a test has failed because we start getting notifications that the service is unavailable, or has degraded performance. I still need to go dig up logs to see how the fault actually impacted the system, then go file a bug against code that's already been committed. If I spend 15 minutes digging through logs only to find that an error was caused by a bug in Azure Search code that could have been caught at a lower level of testing before that code was checked in, I won't be very happy.
The monkey is there to test how our code works as the underlying cluster changes states and experiences failures. Before any code gets passed to the monkey, we should be fairly confident through rigorous functional tests that it performs correctly in most circumstances.
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like restarting VMs and focused mostly on injecting failure into Azure Search components. It's absolutely something that I want to look into for future iterations.
We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.
We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.
Hi thanks for posting.
I was wondering how many engineers are actively working on your search chaos monkey. In my experience it takes a lot of man hours to get something such as this up and running, especially taking into account stabilising the azure search product while bringing the automated functional testing up, stabilising the testing system, and then bringing the chaos monkey monkey up and stabilising all three (product, testing system and chaos system).
Again thanks for the write up.
I was wondering how many engineers are actively working on your search chaos monkey. In my experience it takes a lot of man hours to get something such as this up and running, especially taking into account stabilising the azure search product while bringing the automated functional testing up, stabilising the testing system, and then bringing the chaos monkey monkey up and stabilising all three (product, testing system and chaos system).
Again thanks for the write up.
Hi, kabouseng. Right now, I'm the only engineer working on this.
I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was able to leverage pre-existing Azure Search infrastructure.
We have great tools for monitoring clusters, a proven queue-based job process and a good deployment workflow. Not only does it make it easy to set up systems like this, but it also allows us to smoothly and efficiently manage an ever-increasing number of customer services. It's been a good investment for us.
I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was able to leverage pre-existing Azure Search infrastructure.
We have great tools for monitoring clusters, a proven queue-based job process and a good deployment workflow. Not only does it make it easy to set up systems like this, but it also allows us to smoothly and efficiently manage an ever-increasing number of customer services. It's been a good investment for us.
Thanks for the nice and informative post. Can you add a bit more color to extreme chaos failure with maybe a more concrete action and its outcome? I'm assuming you're pre-classifying availability/sla's for the serving parts of the search system and its sub-systems. Also, a real world example of a silent failure observed in your systems will be nice to learn from.
Thanks for the feedback! I do have an example of a silent failure that would be classified as extreme chaos.
Before we went into production, we were having problems with leader election that could cause multiple masters per cluster. This is an extreme chaos state not only because it could result in corrupted or missing data (as a cloud service, we take data integrity very, very seriously), but also because it didn't raise any immediate notifications, only downstream errors.
This problem has since been fixed, but even before we did that, the highest priority was to ensure our system detected this state as quickly as possible. We added a service that monitored cluster leader status and that quickly alerted us whenever anything was unexpected.
Even after fixing the problem, I've set the Search Chaos Monkey to injecting failure into leader election at various points to ensure that the cluster can always recover in an expected way that won't put customer data at risk.
Before we went into production, we were having problems with leader election that could cause multiple masters per cluster. This is an extreme chaos state not only because it could result in corrupted or missing data (as a cloud service, we take data integrity very, very seriously), but also because it didn't raise any immediate notifications, only downstream errors.
This problem has since been fixed, but even before we did that, the highest priority was to ensure our system detected this state as quickly as possible. We added a service that monitored cluster leader status and that quickly alerted us whenever anything was unexpected.
Even after fixing the problem, I've set the Search Chaos Monkey to injecting failure into leader election at various points to ensure that the cluster can always recover in an expected way that won't put customer data at risk.