Building Microservices: Free Ebook from O’Reilly and NGINX(nginx.com)
nginx.com
Building Microservices: Free Ebook from O’Reilly and NGINX
http://nginx.com/blog/building-microservices-free-ebook-oreilly-nginx/
35 comments
For what it's worth, this is the first three chapters, not the full e-book. It's in the fine print.
It would be good if the mods (or submitter) could change the title to reflect this, I think that a few people might be upvoting before they actually try to download the book and realise that it's not the full version.
Chapters 1, 4 and 11.
I know it's narrow minded, but I was bummed that I had paid full price for the epub. Making a couple of chapters available for free is a great way to promote a good book, let the material talk for itself!
The title is misleading. Super lame.
[deleted]
One of the main problems with breaking up a system into many independent (micro?)systems is that you can't easily combine two or more systems into a logical transaction. Or, in other words, it becomes increasingly more difficult to do things atomically to the complete system.
Is this problem covered?
Is this problem covered?
The problem is covered in Chapter 5. There's isn't really a silver bullet, depends a bit on your use case. He outlines a couple of approaches:
* Try Again Later: queue failed parts of the transaction for retry, moving towards eventual consistency
* Abort the Entire Operation: On failure issue a compensating transaction to roll back the previous successes
* Distributed Transaction: have some overall governing transaction manager process that orchestrates consistency (e.g. duplicate the transaction process of your monlithic data store up in the application layer). You could do a two-phase commit where each participant in the distributed transaction tells the transaction manager whether it thinks its local transaction can go ahead. "A single no vote is enough for the transaction manager to send out a rollback to all parties."
For each of these approaches there are issues, obviously, and to his credit he's pretty even-handed about them. He also says:
"If you do encounter state that really, really wants to be kept consistent, do everything you can to avoid splitting it up in the first place. Try really hard. If you really need to go ahead with the split, think about moving from a purely technical view of the process (e.g., a database transaction) and actually create a concrete concept to represent the transaction itself."
He gives an example of a domain model for a retail company of an order-in-process: instead of attempting to capture a user's order as a single database transaction, model the concept of an order flow of through fulfillment, shipping, marketing, accounting, etc.
* Try Again Later: queue failed parts of the transaction for retry, moving towards eventual consistency
* Abort the Entire Operation: On failure issue a compensating transaction to roll back the previous successes
* Distributed Transaction: have some overall governing transaction manager process that orchestrates consistency (e.g. duplicate the transaction process of your monlithic data store up in the application layer). You could do a two-phase commit where each participant in the distributed transaction tells the transaction manager whether it thinks its local transaction can go ahead. "A single no vote is enough for the transaction manager to send out a rollback to all parties."
For each of these approaches there are issues, obviously, and to his credit he's pretty even-handed about them. He also says:
"If you do encounter state that really, really wants to be kept consistent, do everything you can to avoid splitting it up in the first place. Try really hard. If you really need to go ahead with the split, think about moving from a purely technical view of the process (e.g., a database transaction) and actually create a concrete concept to represent the transaction itself."
He gives an example of a domain model for a retail company of an order-in-process: instead of attempting to capture a user's order as a single database transaction, model the concept of an order flow of through fulfillment, shipping, marketing, accounting, etc.
A concrete example we've faced. A certain operation requires writing data to N flaky services. You successfully write to N-1 of them, but the Nth fails. Now what do you do?
If these N things were just database writes to the same DB, transactions would save you, as you could just rollback. Without that, the answer has to be handled in code -- do you reverse the previous changes (if possible) by sending delete events, or leave the system in some sort of half-baked state and rectify things later via some other process? (I'm interested in hearing of other options...)
If these N things were just database writes to the same DB, transactions would save you, as you could just rollback. Without that, the answer has to be handled in code -- do you reverse the previous changes (if possible) by sending delete events, or leave the system in some sort of half-baked state and rectify things later via some other process? (I'm interested in hearing of other options...)
On my phone so boo cite. Look up Amazon apologetic computing. They also have a presentation describing their general distributed approach. The main thing is that you have a bunch of components that you expect to fail. If they don't get confirmation, they send another message to the next component. Then, there's a stateless, unambiguous way to determine which message "wins" in case of conflict. For instance, an order submits two shipping requests for the same order. They use, for instance, the highest message ID (a GUID) to avoid duplicate processing.
It's all very neat, though I'm doing a bad job explaining. Search for Amazon architectural presentations and hopefully you can find it. It may have been on James Hamilton's blog, maybe.
It's all very neat, though I'm doing a bad job explaining. Search for Amazon architectural presentations and hopefully you can find it. It may have been on James Hamilton's blog, maybe.
You are basically describing the consensus problem [1] and one of the most popular algorithms used to solve is the paxos algorithm [2]
[1] - http://en.wikipedia.org/wiki/Consensus_%28computer_science%2... [2]- http://en.wikipedia.org/wiki/Paxos_(computer_science)
[1] - http://en.wikipedia.org/wiki/Consensus_%28computer_science%2... [2]- http://en.wikipedia.org/wiki/Paxos_(computer_science)
As far as I understand microservices, these kind of issues must be solved by your architecture. Either the service is atomical, or the orchestration of your services must be in "flow", where no race conditions should occur.
It sounds like this is a difficult path to take. Especially considering the fact that, usually, system requirements may change over time.
It absolutely is. It's for exactly these reasons why only highly organised and qualified teams tend to have substantial microservice deployments today.
It's also why I think[1] we need to start building a framework[2] to provide standardised solutions (or at least patterns) to some of these problems.
[1]: https://vimeo.com/118895501 [2]: http://wym.io/
It's also why I think[1] we need to start building a framework[2] to provide standardised solutions (or at least patterns) to some of these problems.
[1]: https://vimeo.com/118895501 [2]: http://wym.io/
Related: I'm recording myself code up a new microservices hobby application. I'm doing the whole thing, from selecting a domain name to services orchestration (much later, of course). The technology stack is F#, Mono, .NET, ubuntu, AWS, BASH, Apache. We're talking about architecture, design, user stories, TDD, philosophical differences between traditional OO coding and FP coding and more.
It's all free.
If you're interested in more microservices stuff, here's the link to the first one. Fair warning, though: this isn't professionally produced or slick. It's just some schmuck (me) programming.
http://vimeo.com/120136738
It's all free.
If you're interested in more microservices stuff, here's the link to the first one. Fair warning, though: this isn't professionally produced or slick. It's just some schmuck (me) programming.
http://vimeo.com/120136738
I'm looking forward to this. We're about to embark on a (probably) .Net-based microservices project, although we'll probably use C# rather than F#.
One thing I've found is that infrastructure stuff abounds for Java-based projects, doing much work for the message bus, metrics, and logging. But I find much less available for .Net, particularly for the latter two. Maybe you'll find some stuff that'll help me.
One thing I've found is that infrastructure stuff abounds for Java-based projects, doing much work for the message bus, metrics, and logging. But I find much less available for .Net, particularly for the latter two. Maybe you'll find some stuff that'll help me.
Being old and cantankerous, my feeling is that we've tooled up about 400 times more than we need for microservices and the cloud -- a good programmer could/should be able to construct and maintain most small-to-medium-sized projects without a lot of third-party tools.
As an Agile coach, what I find interesting is conveying the philosophy of the work. I am nowhere near being a code ninja any more, but the way you think about writing the application is the most important part of doing the work.
But who knows? We'll see.
EDIT: To be a little clearer, message bus? You sure you need a full-bore bus? Make it work with simple files. Take a look at your volume. If you code it correctly from the start, adding on new stuff is easy. If you start with lots of frameworks and such, you can get caught up in complexities that are orthogonal to your mission. One of the beauties of going full functional is that it makes you focus on whether the dang thing is working or not. In an OO environment, many times you get lost in the "wiring" of the construction, making sure you implement the correct interfaces, paying attention to Demeter's Law and the rest. The end of the week comes and all you've got is forty thousand lines of code and 7 new dlls in your build.
As an Agile coach, what I find interesting is conveying the philosophy of the work. I am nowhere near being a code ninja any more, but the way you think about writing the application is the most important part of doing the work.
But who knows? We'll see.
EDIT: To be a little clearer, message bus? You sure you need a full-bore bus? Make it work with simple files. Take a look at your volume. If you code it correctly from the start, adding on new stuff is easy. If you start with lots of frameworks and such, you can get caught up in complexities that are orthogonal to your mission. One of the beauties of going full functional is that it makes you focus on whether the dang thing is working or not. In an OO environment, many times you get lost in the "wiring" of the construction, making sure you implement the correct interfaces, paying attention to Demeter's Law and the rest. The end of the week comes and all you've got is forty thousand lines of code and 7 new dlls in your build.
I suspect that getting our developers to embrace not only this new architecture, but also the new paradigm of functional programming is a bridge too far.
Regarding the "full-bore bus" question, I see it being that way for two reasons:
1. Distributed architecture. I think that the way to foster simple deployment of updates services is making them distinct deployable units, and the most obvious way to do that is to put each on its own cluster of servers - although in most cases this is way overkill in scale. And I think that if the services are on different machines, that pushes away from a really simple approach like your idea of files, toward something more sophisticated. On the other hand, perhaps there's a good way to let the services be co-resident by using docker, which I don't yet know very well.
2. Capacity. Although most of the system isn't too heavy, I've got a couple services that are used ubiquitously, and probably do need serious high-performance. First, the legacy system at peak is handling about 25 product pricing queries per second (our pricing is customer specific and based on near-time availability from suppliers). Second, we have not just localization of text, but customer-specific text, so every bit of text everywhere needs to come from that customization engine, although I suspect that this may wind up being addressed more as caches at the UI layer.
Regarding the "full-bore bus" question, I see it being that way for two reasons:
1. Distributed architecture. I think that the way to foster simple deployment of updates services is making them distinct deployable units, and the most obvious way to do that is to put each on its own cluster of servers - although in most cases this is way overkill in scale. And I think that if the services are on different machines, that pushes away from a really simple approach like your idea of files, toward something more sophisticated. On the other hand, perhaps there's a good way to let the services be co-resident by using docker, which I don't yet know very well.
2. Capacity. Although most of the system isn't too heavy, I've got a couple services that are used ubiquitously, and probably do need serious high-performance. First, the legacy system at peak is handling about 25 product pricing queries per second (our pricing is customer specific and based on near-time availability from suppliers). Second, we have not just localization of text, but customer-specific text, so every bit of text everywhere needs to come from that customization engine, although I suspect that this may wind up being addressed more as caches at the UI layer.
What I should do, once I finish up the basic functionality (which could take a month or two working part-time), is copy this over to an empty box(es) and pound the hell out of it; see how much volume it can handle.
My gut tells me the Ubuntu O/S can take quite a bruising. It'd be interesting to find out. Also there'd be some good lessons in there :)
Remember, the goal of microservices is to completely decouple processing so that you can completely configure the cloud graph any way you'd like. A microservices OO construct by definition would have to do a lot of honking around with the CPU before we got to the "work" part.
Of course what works for one team wouldn't work for another. Fun stuff to play around with, though.
My gut tells me the Ubuntu O/S can take quite a bruising. It'd be interesting to find out. Also there'd be some good lessons in there :)
Remember, the goal of microservices is to completely decouple processing so that you can completely configure the cloud graph any way you'd like. A microservices OO construct by definition would have to do a lot of honking around with the CPU before we got to the "work" part.
Of course what works for one team wouldn't work for another. Fun stuff to play around with, though.
A microservices OO construct by definition would have to do a lot of honking around with the CPU before we got to the "work" part.
No doubt. While I'm enamored with OO myself (having begun my professional life in the late 80s when it was the thing), I think that anything OO would be internal to a given service, and the interfaces between services would be much more conventional, most likely a simply REST approach.
No doubt. While I'm enamored with OO myself (having begun my professional life in the late 80s when it was the thing), I think that anything OO would be internal to a given service, and the interfaces between services would be much more conventional, most likely a simply REST approach.
What are your reasons for using C# over F#?
The team has significant experience with C#, but none with F# (or indeed with functional programming at all). It seems like changing the architecture from a well-partitioned yet still monolithic service to a microservice approach is enough of a paradigm shift. Switching also to a functional approach may lead to some exploding heads.
Note that the full copy is now available to purchase in both ebook and hardcopy: http://shop.oreilly.com/product/0636920033158.do
(No side-interest here, just found it confusing that the page mentioned "hardcopy expected to be published in May")
(No side-interest here, just found it confusing that the page mentioned "hardcopy expected to be published in May")
Indeed, my copy just arrived.
I'm pretty sure the idea behind publishing this free book was gathering leads and potential customer-data for future marketing.
You know, if it's free, you're the product and all that yadda yadda.
By posting the direct DL link here, you are discouraging similar free offers in the future. If I were you and I appreciated free offers, I would consider editing my post.
You know, if it's free, you're the product and all that yadda yadda.
By posting the direct DL link here, you are discouraging similar free offers in the future. If I were you and I appreciated free offers, I would consider editing my post.
> I'm pretty sure the idea behind publishing this free book was gathering leads and potential customer-data for future marketing.
This is exactly why the direct link should be shared. I'd bet that most HN users aren't interested in "future marketing."
This is exactly why the direct link should be shared. I'd bet that most HN users aren't interested in "future marketing."
Then they should sent the pdf not give a link to it, for me it says "share it".
I have given O'Reilly my email before, but the content is absolute crap. They are spamming me even though I have cancel the subscription 3 times, hence direct link may bring them more marketing value than bunch of spam-only emails with fake names and phones numbers.
I have given O'Reilly my email before, but the content is absolute crap. They are spamming me even though I have cancel the subscription 3 times, hence direct link may bring them more marketing value than bunch of spam-only emails with fake names and phones numbers.
Unlikely to garner much favour declaring 'Free Ebook from O’Reilly and NGINX' and giving away only 3 chapters...
I think that's why they put the "NGINX ad page" into that ebook.
Bait and switch. http://en.wikipedia.org/wiki/Bait-and-switch
So inbound.