With NSQ there is a built in utility nsq_to_file which just becomes one additional consumer you'd use to archive each message topic to disk. It provides dead simple archiving of messages, but doesn't provide any native replay ability.
thanks! Most comes from credit card, and foursquare but some various other places. strava (bike ride timings) starbucks card (some coffee timings), scripted photo snapshots, scripted computer activity snapshots etc.
I've also posted some of the scripts on github like this one to parse out iMessage/sms logs from an iphone backup. https://gist.github.com/jehiah/4437883
Great writeup of some fun unicode/font details. I've had to use the trick of limiting font files to specific unicode ranges while working on Amharic support for itypeamharic.com (Amharic/Ge'ez glyphs are also typically not present in any fonts on most devices)
My rough approach to post processing this data was to group by 6 minute intervals, and to drop any records where the computer was inactive for more than 120 seconds.
I've not heard of whatpulse, but looks cool. I rolled my own data collection for keyboard stats last year.
I've highlighted some last.fm data in a few of my previous annual reports, but it didn't make it this year (I Wasn't quite into new music enough recently for it to be interesting to me).
I was most surprised by the sheer amount of time I spend on the computer; clearly I need to unplug more often. (Though you can tell I did drop off the grid for a few weeks when my daughter was born, and again for a vacation in August. At least that's a start).
doh. That's probably due to loading a full years worth of images as png's (I've added support for webp images when browsing w/ Chrome but that isn't going to help the iPad). Sorry i didn't get to do more than minimal iOS testing.
lies, damned lies, and statistics. Unfortunately i didn't have time data for my other trips to Starbucks so that dataset is partial (I mostly visit Starbucks on the weekend/holidays).
The transition from Carte Blanche to Burbon Coffee in the middle of the year is when @bitly's office moved, and my coffee habit followed.
coffee data is more driven from foursquare checkins, and only partially credit card transactions.
The motivation for me is a way to learn about my year (not directly to change my actions), and part a fun exercise to get better at data processing, javascript, and UI Interaction/charting. I like the hard deadline that a personal annual report creates naturally (ie: it's no good if i release it in june).
Some of the other scripts just download data from foursquare's API, and post processing transaction download from Mint. Hopefully I'll have time to clean and post those scripts.