Using Clojure at MixRadio(dev.mixrad.io)
dev.mixrad.io
Using Clojure at MixRadio
http://dev.mixrad.io/blog/2014/10/19/Clojure-libraries/
5 comments
Yep, we moved a whole team of about 30 people over from Java to Clojure. On the whole it was fairly painless, considering how big a shift that was anyway.
It's hard to know where we started seriously advocating a move to Clojure and where it was just our enthusiasm but we did quite a few things to sell it.
- A Clojure club where we booked a big meeting room and taught some of the basics of Clojure
- Buying a bunch of books and leaving them around the office
- Re-writing an existing service into Clojure as an example of how quick this was to do, and how nice the resulting code was (as a note we haven't gone back to re-write working services)
- High levels of enthusiasm and trying our best to make sure everyone got a chance to be included
At some point we decided that we had enough support to try things more seriously so we got all the devs together in a room and made a proposal to work on a few new services in Clojure. Having a Service Oriented Architecture actually really helps here as we didn't have to make too big of a bet. That was enthusiastically accepted by the team and from there we ended up writing more and more things in Clojure. A few people were more resistant than others and they stuck working on Java projects for longer. After a while though curiosity or necessity led to them working with Clojure and they ended up convinced. I'm not aware of anyone now who isn't a fan.
Training was a combination of self taught, books and, later on, booking a 3 day, onsite training course from the guys at Lambda Next. This is something Cognitect also offer now.
From a dev ops perspective not much had to happen as our existing Java tooling continued to work. That said we have recently rewritten all our deployment tooling (in Clojure) to target AWS (and generally improve it in just about every way). We actually have a blog post due to be published this week talking about our new, Clojure based, tooling.
It's hard to know where we started seriously advocating a move to Clojure and where it was just our enthusiasm but we did quite a few things to sell it.
- A Clojure club where we booked a big meeting room and taught some of the basics of Clojure
- Buying a bunch of books and leaving them around the office
- Re-writing an existing service into Clojure as an example of how quick this was to do, and how nice the resulting code was (as a note we haven't gone back to re-write working services)
- High levels of enthusiasm and trying our best to make sure everyone got a chance to be included
At some point we decided that we had enough support to try things more seriously so we got all the devs together in a room and made a proposal to work on a few new services in Clojure. Having a Service Oriented Architecture actually really helps here as we didn't have to make too big of a bet. That was enthusiastically accepted by the team and from there we ended up writing more and more things in Clojure. A few people were more resistant than others and they stuck working on Java projects for longer. After a while though curiosity or necessity led to them working with Clojure and they ended up convinced. I'm not aware of anyone now who isn't a fan.
Training was a combination of self taught, books and, later on, booking a 3 day, onsite training course from the guys at Lambda Next. This is something Cognitect also offer now.
From a dev ops perspective not much had to happen as our existing Java tooling continued to work. That said we have recently rewritten all our deployment tooling (in Clojure) to target AWS (and generally improve it in just about every way). We actually have a blog post due to be published this week talking about our new, Clojure based, tooling.
How did you convince management to let you use Clojure? Some of the devs where I am are interested, but I suspect management will be quite hesitant to embrace a new language.
That's a good question. The honest answer is that we didn't. By the time we made this decision it was far more about convincing developers. The history of getting to that point is one of trying small experiments with new tech and proving that we make good decisions and can be trusted.
We did have a big meeting with all of R&D where we talked about all the pros and cons. Hiring was a big question early on although it's still my belief that using Clojure rather than Java is a positive in terms of attracting good programmers. I don't think we have enough data to answer that either way though.
I think that most important part was that we weren't betting the whole company on it. We were only betting a few weeks of developer time. If it didn't work out then the worst case was rewriting those services back into Java (in fact until recently we had a single Scala service from a previous experiment). Quite a few companies start off by writing some internal tooling in Clojure as an easy to stomach first step. That may be a good path if management are resistant.
We did have a big meeting with all of R&D where we talked about all the pros and cons. Hiring was a big question early on although it's still my belief that using Clojure rather than Java is a positive in terms of attracting good programmers. I don't think we have enough data to answer that either way though.
I think that most important part was that we weren't betting the whole company on it. We were only betting a few weeks of developer time. If it didn't work out then the worst case was rewriting those services back into Java (in fact until recently we had a single Scala service from a previous experiment). Quite a few companies start off by writing some internal tooling in Clojure as an easy to stomach first step. That may be a good path if management are resistant.
Thanks for the reply. If you don't mind one quick last question - can I ask which books you used? The only half-decent one I've found is Clojure In Action.
We have quite a few floating around but here are the main ones.
- Joy of Clojure. I think a lot of people would say this was their favourite. It's less about learning Clojure and more about loving and writing good Clojure.
- Clojure Programming. There are mixed review on this one but I quite liked it as a reference book early on.
- Programming Clojure - Similar to Clojure Programming. Pretty good.
Thinking about it a lot of people also spent some time working through the 4Clojure exercises.
Personally I found that the only way to really get it was to jump in. We have a few early services that could perhaps do with a re-write but no real harm was done!
- Joy of Clojure. I think a lot of people would say this was their favourite. It's less about learning Clojure and more about loving and writing good Clojure.
- Clojure Programming. There are mixed review on this one but I quite liked it as a reference book early on.
- Programming Clojure - Similar to Clojure Programming. Pretty good.
Thinking about it a lot of people also spent some time working through the 4Clojure exercises.
Personally I found that the only way to really get it was to jump in. We have a few early services that could perhaps do with a re-write but no real harm was done!
Clojure Programming by Chas Emerick, Christophe Grand, and Brian Carper is really good in my opinion. I'm coming to Clojure from Ruby and I've found this book to be most approachable.
Out of curiosity, what sort of protocol are you using for RPC between services?
It's JSON over HTTP for the most part. We have a bit of legacy XML still knocking about and some services use message queues.
Other than JSON over HTTP you can also use something like https://github.com/lynaghk/zmq-async/ .
Will something stop me from using Protocol Buffers if I move to Clojure?
If you're interested in seeing what the source code of a simple Clojure web application would look like, you might take a look at my implementation of Hacker News in Clojure:
https://github.com/taylorlapeyre/hn-clojure
https://github.com/taylorlapeyre/hn-clojure
Completely off topic, and purely out of curiosity, but what's MixRadio, as a business, right now? Are you part of Microsoft? (then wow, Clojure) Or of what's left of Nokia (and licensed to Microsoft somehow)?
Where's the team located? I had expected Finland, but the OP doesn't seem very Finnish.
Btw, great work. I'm a very happy Mixradio user.
Where's the team located? I had expected Finland, but the OP doesn't seem very Finnish.
Btw, great work. I'm a very happy Mixradio user.
For the full version of exactly what we do and where we've come from, we've got a nice write up: http://dev.mixrad.io/blog/2014/08/05/MixRadio-History/
In terms of development, the team is located in sunny Bristol in the UK. Glad you like it!
In terms of development, the team is located in sunny Bristol in the UK. Glad you like it!
Thanks! Nice story.
And why can't I sign up or renew my service?
Extremely off topic, but as an user from India I have been unable to renew my mixradio subscription. Oxicash has stopped selling vouchers since May 2014. Nokia stores don't have any either.
And I can't signup to using the mixradio+ app on Windows 8 which supports 320kbps mp3 downloads.
Extremely off topic, but as an user from India I have been unable to renew my mixradio subscription. Oxicash has stopped selling vouchers since May 2014. Nokia stores don't have any either.
And I can't signup to using the mixradio+ app on Windows 8 which supports 320kbps mp3 downloads.
At the bottom of the page it says:
© Microsoft 2014
© Microsoft 2014
Can you describe your Clojure hardware stack? Everything running on the AWS?
Besides, what libraries do you use for data analysis? Sadly this is an area where Clojure remains a bit stagnant, or at least people don't release what they use.
Besides, what libraries do you use for data analysis? Sadly this is an area where Clojure remains a bit stagnant, or at least people don't release what they use.
Yep, we run everything on AWS as of a few months ago. We've got a post about the nuts and bolts of that here: http://dev.mixrad.io/blog/2014/09/19/Migrating-MixRadio-Infr.... We've got a post about our deployment tools that's going to be going up soon!
In terms of data analysis we use a lot of Cascalog, although a fair amount of ad-hoc work is done in RedShift.
In terms of data analysis we use a lot of Cascalog, although a fair amount of ad-hoc work is done in RedShift.
Interesting!
Just three more little things:
- Do you document using Marginalia or something similar (perhaps lein-midje-doc)?
- Since you do a lot of REST, have you ever considered Liberator?
- Regarding Clojure books, I've found Clojure in Small Pieces quite interesting---but far from complete
Just three more little things:
- Do you document using Marginalia or something similar (perhaps lein-midje-doc)?
- Since you do a lot of REST, have you ever considered Liberator?
- Regarding Clojure books, I've found Clojure in Small Pieces quite interesting---but far from complete
- We still havn't really hit on a solution for internal documentation. I'm something of a heretic on this point to be honest, as i try to actually deliver the 'code as doc' idea, which i think you can realize with a combination of compojure routes and schema definitions.
- We did look at Liberator a while ago, but it didn't seem mature at the time. To be honest we get by perfectly find with compojure, having created our own middleware to deal with any fiddly bits.
- The Joy of Clojure, for me, is the one. It's focus is on why Clojure is such fun to work with and doesn't get bogged down in trying to teach you every little function, it's not a reference book. A good read of that, coupled with the clojure cheatsheet, mix in 'The Little Schema' to get you lisping and then practice (4clojure, exercism.io, euler), which is key.
- We did look at Liberator a while ago, but it didn't seem mature at the time. To be honest we get by perfectly find with compojure, having created our own middleware to deal with any fiddly bits.
- The Joy of Clojure, for me, is the one. It's focus is on why Clojure is such fun to work with and doesn't get bogged down in trying to teach you every little function, it's not a reference book. A good read of that, coupled with the clojure cheatsheet, mix in 'The Little Schema' to get you lisping and then practice (4clojure, exercism.io, euler), which is key.
Running everything in AWS doesn't seem very Microsoftish...
What made you choose clj-http over http-kit?
Nothing more scientific than, at the time we started out with Clojure, that was what we found first and it worked! Recently we've begun using http-kit in some high-load areas with some really good results.
Particularly interested in:
- developer acceptance (unless they were already all clojure guys?)
- training materials
- changes to devOps.