This is a great point about the time, thank you for the feedback!
About storing logs in the fs: I was thinking about i.e. the database but then how do I solve cases like "this one server acts up, can I get the verbose logs from there but not from my other servers?"
I can totally understand your point. This is an excessive example with lots of customization. You can get by with a lot less code if you just want to get a sane default-format or a lot more if you want to really fine-tune your logs. The beauty is that you have the option :)
All the mentioned solutions are great!
I personally think it does not matter if we use key-value output or "sentences", they have to be parsed anyway. I prefer writing Function: Foo instead of fn=Foo, just because I think it's easier to read when a human actually has to dig into the raw logs. For machines it's really the same.
I will detail how I parse logs with Logstash in a future post of this series. (And some logs like apache / nginx / mysql cannot be changed to use key-value format easily, so you might not be able to get around string parsing anyway)
Log4r allows you to use a key-value format as well, just modify the PatternFormatter and your output (in the logger.rb in my example).
I would argue that the standard rails logs are perfectly fine for local development and debugging. NewRelic even ships a local version that allows to track down performance issues.
This works great for one or two requests triggered by the developer, but fails when thousands of requests bombard a live application split over a multiple servers with multiple rails instances running on each.
Scout, NewRelic and HoneyBadger are great tools to get started with. Their limitations are that they do what they do, how they think it should be done. If you need different metrics, another way to split up the data or correlate your mongo logs with app requests via NewRelic, you are out of luck.
This is when creating your own logs with exactly the data you needs becomes valuable, when you can measure what you need how you need.
My first post set the foundation for being able to easily get the right metrics and information by creating solid logs. It does not mean that this has to completely replace the aforementioned tools, on the contrary, it can supplement them to give you exactly the data you need when you need it.
I must say that I did not know of the Logging gem. It looks really promising and has some nice features like coloring for i.e. stdout and rspec support.
Sadly the docs are even worse than the ones for Log4R. I will take a look into writing another blogpost as an introduction to Logger as well as the differences and advantages of Logger.
Thanks for making me aware of this gem!