You have to make all code threadsafe that execute from Job or you have to decouple Job code and App code.(which probably be required anyway because I'm not sure sidekiq supports old rubies)
So, in any case you have to rewrite as much as code that I rewrote in Go and decoupled from main App. (its not lot of code, I mentioned in talk)
wkhtmltopdf and phantomjs both worked similarly, currently I'm using phantomjs.
And I'm not splitting pdf but splitting html generation work load, and then create individual pdfs from those html chunks. Then they will be joined together (using pdfunite). I found this much faster then joining html and generating large pdf.
As fast as twilio can accept and process without throttling, beyond that its not much useful.
Unicorn forking benefit is overrated, we used it and we don't see much benefit for long running processes.
Sidekiq is good alternative but that means some rewrite(for our app anyway). Secondly Sidekiq looks mature today, I started working on some of these changes 2 years ago.
Edit: Our requirement was to process this queue as fast as possible and that means more workers. With process based concurrency that is very costly as you have explained.
Our ruby application had memory problems and thats why it needed overhaul, but instead of changing architecture and rewriting same thing in Ruby I choose Go. Sure, rewriting made it better, doing it in Go made it lot better.
One of the thing I said in presentation was "This performance numbers look impressive but ignore them, by rewriting in Ruby could have improve them may be not huge margin but still they would have been better"
I choose Go not because language was better, and not only for performance.
Simple deployment was key point and deployment is not just about deploy and forget, there are entire companies founded around deploying and maintaining Ruby apps for you because it's not simple thing for tiny startup.
Exactly right any language could have worked, and we didn't replaced Ruby, just few part of our big application is rewritten.
I did in Go not because of language is better, ecosystem and culture is better but not language part. I will suggest you to do small real life project in Go.
Problem is I don't know Java. But I know enough Java to know that it will take lot longer for me. And I might not have simple deployment benefits, thats important feature of Go.
You exactly captured my sentiment, when I started doing this I don't realized all benefits. Because of Go's simplicity in deploying and maintaining, many small apps doesn't add much overhead. Now you can scale individual component.
Only risk is, you break things into many component then you should so balance is required.
Exactly, Ruby app was monolith so it was consuming much more then this single feature would require. Now there is double benefit Go already uses lot less memory and its microservice. Plus with Ruby we have to run multiple processes so whatever memory single process consume * # of workers.
I could have rebuilt in Ruby and that was my first thought but deploying and maintaining Ruby apps are lot harder then you can imagine. I didn't like idea of maintaining lots of small ruby apps. Once you deploy single Go application, you might not want to deploy another Ruby app.
I agree that line bit confusing, Ruby app doesn't connect with Go service directly, every requests goes through balancer and then nginx router. Nginx routes then using simple location directive. Authentication is not shared, its either token based or Go service act as proxy and send request to Ruby app and modify response received from Ruby app before sending to client(i.e covering html to pdf)
Initially I used https://revel.github.io because I wanted to use many inbuilt things and hot reload.
Later I started use just standard library with http://www.gorillatoolkit.org and used fswatch for hot reload. Now there are lot of alternative in routers but no clear winner.(and thats good thing) I will suggest go with standard library and simple router but be ready to revert back to framework like revel if that doesn't workout.
After my talk many people voiced same concerned of overhead. But I don't see it that way, overhead is not much over simplicity and familiarity benefits.
JSON has advantage of human readability and for that tooling support is required which probably you will loose with RPC. For example I use many tools to inspect requests that won't support it with RPC
Apple's attitude towards developers complaining about App Store was largely "Deal with it" but seems like they themselves can't deal with it.