Building Products at SoundCloud – Part I: Dealing with the Monolith(developers.soundcloud.com)
developers.soundcloud.com
Building Products at SoundCloud – Part I: Dealing with the Monolith
http://developers.soundcloud.com/blog/building-products-at-soundcloud-part-1-dealing-with-the-monolith
19 comments
Loose coupling is awesome and dogfooding your public API is smart as well. However, it seems like there is a lack of strong technical leadership and a lot of big decisions are being made without thinking them through. For instance, "We were constrained by the same rules we enforced on third-party applications... It was not possible, for example, for a microservice to notify users about activity on private tracks". This type of constraint should be immediately obvious to the person/people making large architectural decisions. Throughout the article there are a few examples of "we tried this but shortly found out it wouldn't work". I may be wrong though and this whole post was actually the result of an hour meeting to discuss new architecture but phrased to sound like you actually tried all of those approaches.
> it seems like there is a lack of strong technical leadership and a lot of big decisions are being made without thinking them through
I didn't get that impression at all. The ability to do it right (tm) the first time is quite often hindered by the need to iterate quickly and focus on growth and/or revenue. You'll eventually end up with an increasingly-more-painful monolithic core system, but that's an okay trade off if it got you to a place where your business really takes off.
At that point you have a few options. You can do a major rewrite, re-architecting everything from the ground up. Or, you can take Soundcloud's approach, and slowly migrate the old system to a more service-oriented architecture, reducing the amount of dev resources you need to throw around.
The trick is that the slow migration will be different for every business, and it depends on the needs of the platform and the way things were constructed initially. Just like you can't blame the technical leadership the trade-offs made early on, and you can't blame them for experimenting and iterating before finding solutions that worked for them. (After all, hindsight is 20/20.)
I didn't get that impression at all. The ability to do it right (tm) the first time is quite often hindered by the need to iterate quickly and focus on growth and/or revenue. You'll eventually end up with an increasingly-more-painful monolithic core system, but that's an okay trade off if it got you to a place where your business really takes off.
At that point you have a few options. You can do a major rewrite, re-architecting everything from the ground up. Or, you can take Soundcloud's approach, and slowly migrate the old system to a more service-oriented architecture, reducing the amount of dev resources you need to throw around.
The trick is that the slow migration will be different for every business, and it depends on the needs of the platform and the way things were constructed initially. Just like you can't blame the technical leadership the trade-offs made early on, and you can't blame them for experimenting and iterating before finding solutions that worked for them. (After all, hindsight is 20/20.)
I think iterating quickly is great for product decisions where you don't know how people will react. Technical decisions, however, can be objectively evaluated up front given the right experience. I don't think a migration from a monolithic system is particularly exceptional.
Having been through a similar decoupling, sometimes that few hour meeting would end on a positive note, "this looks good guys, lets get to work." Only for a few hours later, someone gets into a new mindset only to realize a few aspects need reevaluated.
You're reading a lot into a single sentence fragment, and an awful lot of these decisions are not obvious a priori. In this case, the impression I got was "separating a large monolithic app is a risky, expensive move" and that concern overrode other practical concerns.
I doubt these constraints came about as surprises once their hand was forced.
I doubt these constraints came about as surprises once their hand was forced.
I agree with you. Looks like poor engineering leadership.
Guess its one of the disadvantages when you are not based in the Silicon Valley...you simply don't have access to 1st class talent that has done the same task at 2 other high growth startups before.
Guess its one of the disadvantages when you are not based in the Silicon Valley...you simply don't have access to 1st class talent that has done the same task at 2 other high growth startups before.
There are network effects regarding the portion of outliers that SV captures, but if we're talking about experience alone then I'm skeptical of your claims given how much of SV is driven by freshly minted CS grads.
Where else in the world can you hire the guys who solved exactly the same problem (Monolithic Rails Stack) at companies like Twitter, Airbnb, Zendesk, etc.?
You're making a big assumption, which is that all those good engineers love to live in Silicon Valley and want nothing more than to stay in the area their entire lives. For each one of the engineers who feel that way, there's someone who worked here for awhile but now dislikes the area for one reason or another. Enticing good people to spend a few years in Berlin working for a company like Soundcloud might be easier than you think.
I know a lot of top engineers here and non of them would ever consider leaving before retirement or becoming rich.
And in both cases they would not start working for a startup in Berlin or any other place.
If they leave then for southern france, a cow ranch in the mid-west or Bali.
And in both cases they would not start working for a startup in Berlin or any other place.
If they leave then for southern france, a cow ranch in the mid-west or Bali.
I don't see them in Bali.
The disadvantage here is money and VC.
Where else you can burn money and not making profits in the name of engineering efforts?
Where else you can burn money and not making profits in the name of engineering efforts?
I don't get it. Rails monolith is great for product proof-of-concept. Iterative refactoring and incremental change towards an architecture you see working for similar companies is also really sensible. What would you do differently if you had heaps of money and VC?
I'm replying to the parent disagreeing with him.
I definitely agree with iterative refactoring and incremental changes that SoundCloud did.
If SoundCloud has tons of money that can provide a long-term runaway, they can do major refactor or re-write.
i.e.: if competition _AND_ money are not the issue, why not?
I definitely agree with iterative refactoring and incremental changes that SoundCloud did.
If SoundCloud has tons of money that can provide a long-term runaway, they can do major refactor or re-write.
i.e.: if competition _AND_ money are not the issue, why not?
As an outsider to scaling issues, what strikes me about SoundCloud's struggles with shedding their Rails past is how similar their problems sound to Twitter's problems. Granted, a microblogging service has a different set of concerns than an audio delivery service, but the pain is the same.
Going out on a limb, perhaps this similarity contributes to the reasoning for the Twitter/SoundCloud acquisition rumors: more exciting problems for Twitter engineers to solve now that they've tamed their own beast? Or to put the theory in a business light, Twitter engineering leadership may have confidence they can architect SoundCloud better than the SoundCloud team has done.
Going out on a limb, perhaps this similarity contributes to the reasoning for the Twitter/SoundCloud acquisition rumors: more exciting problems for Twitter engineers to solve now that they've tamed their own beast? Or to put the theory in a business light, Twitter engineering leadership may have confidence they can architect SoundCloud better than the SoundCloud team has done.
Great product with some impressive traffic numbers! Looks like a good candidate for flow based programming. side-note: I'd happily pay for streaming that had less compression applied.
My company currently is facing the task of breaking up the monolith. Not going to be fun. Glad that others had this problem as well and solved it in manageable ways. Thanks for sharing.