Heroku’s log-runtime-metrics Can Crash Your Rails App(siliconisland.ca)
siliconisland.ca
Heroku’s log-runtime-metrics Can Crash Your Rails App
http://siliconisland.ca/2013/04/25/herokus-log-runtime-metrics-can-crash-your-rails-app/
Do not run log-runtime-metrics for extended periods of time, as it can crash your Heroku dynos (and cause them to reboot) multiple times daily.
5 comments
It turns out I was wrong about log-runtime-metrics! Please see the followup post: http://siliconisland.ca/2013/04/26/beware-of-the-zombie-proc...
No. Freaking. Way. I noticed this same exact issue pop up on my New Relic interface. Multiple times a day our app would get H13's, crash, and then restart. Have been trying to get to the bottom of this ever since. Awesome detective work on this. Our app (and tech team) thank you very much!
You're welcome! Our community runs on paying it forward, afterall!
Hey, this is my blog post! Thanks for submitting it to HN. :)
In addition to the actual topic of the post (issues with a Heroku beta feature), I find this post pretty interesting for a number of reasons.
It seems that for blog posts are the preferred method of reporting deeper issues. I guess it's a way of publicizing the work of tracking down the problem and earning developer cred. (Do people have blog posts for pull requests too?)
Also, it's impressive to see how fast Heroku's Ryan Daigle replied. Hopefully this issue will be short-lived as the runtime metrics are pretty handy.
Yeah, I actually sent the post to Heroku and NewRelic so they were the first to see it. I think someone from NewRelic submitted it to HN (just a guess).
Curious: What other method would you suggest for reporting deeper issues? I can't think of anything else that would allow for free-form writing like blogs.
Curious: What other method would you suggest for reporting deeper issues? I can't think of anything else that would allow for free-form writing like blogs.
Thanks for blogging this. Today I had my Account manager and another support person from Heroku pushing me to add that experimental feature, so I could see the memory usage of our dynos. I had reported on twitter and in a support ticket that the memory reported by newRelic didn't match the values that our Account Manager sent me. He sent me this because we were trying to add 4 Unicorn processes to a 1 Gig Dyno, and newRelic was reporting our app never went over 256MB. Our Account manager sent a different trace of memory usage.
I have lost all faith in the values that newRelic reports that come from Heroku.
I have lost all faith in the values that newRelic reports that come from Heroku.
log-runtime-metrics does work -- just enable it for a half hour or so, observe the stats, then turn it off. Just can't keep it running long-term.