Shameless plug here - The author of the article does build time-series databases for a living, and more specifically the Datadog monitoring platform - which will gladly collect your millions of metrics, graph them and alert on them, along with all the events you care to keep :-)
We've been through a number of data stores ourselves, starting with Postgres back in 2010 - then on to Redis + Cassandra before we built our own. But that's a story for another post...
We're hiring engineers that specialize in one of the following:
+ Data visualization in the browser
+ Go and distributed systems
+ System programming and low-level performance optimization
+ Data Engineering on the hadoop/spark/python stack
We're one of the very few NYC startups to be engineering-driven and working on hard, large-scale data analysis and visualization problems. We're also growing very fast :-)
Our product is built by engineers for engineers which means you can and will have a lot of impact on it.
I work at Datadog - we're only using ElasticSearch for full-text structured events, not time-series, which represent 10,000 - 100,000 times more data in volume.
We had to build our own Time-Series streaming / storage / query so we could handle millions of points per second and years of retention.
We don't use the recommendation engine but the underlying platform, which makes it really simple to write and run pig jobs. Though the majority of our business deals with real-time data processing, the ability to crunch numbers in batch without dev or ops overhead is attractive and well worth the price to us.
FWIW we've been using the mortar platform to run large pig jobs without a fuss at http://datadog.com and we've been very happy with it. Glad to see them contribute their recommender code too.
We're looking for engineers passionate about either
* Data visualization using D3
* Large-scale real-time data processing in Go, Python and C
* UX/UI Design and development
* Writing technical pieces you'd want to read
We're a growing, well-funded startup. And we're hiring engineers for all positions from back-end to ops to data visualization.
Think tens of billions of data points a day to stream, store, visualize and analyze. We use Python, Numpy, Cython, D3, JS, Cassandra, Postgres, Redis, Elasticsearch, and more...
Nope - all the data stays there, and can be shown on dashboards. We just don't show the hosts that don't submit data in the host lists, and don't count them towards the total number of hosts charged for. Does that answer your question?
Pup will graph all the metrics Datadog does - it already collects OS metrics, and comes with a set of plug-ins for connecting to common apps, and it wouldn't be very difficult to connect collectd to it.
Pup's feature-set is limited today, though, compared to graphite. You may also check-out the full http://datadoghq.com service for more. It's SaaS, and free up to 5 servers.
They'll disappear automatically when you stop sending data about them (within a few hours) - we've done that for our customers who have elastic / rolling servers on AWS for ex.
If that doesn't cut if for you, we're happy to help - let us know what you're trying to do!
otterley - Don't walk away yet... we do support arbitrary dimensions!
You can attach any arbitrary set of tags to metrics or events - on a per-datapoint basis, and slice / dice / alert based on those tags. Datadog will automatically tag your points by chef role or AWS availability-zone, for example, and you can add any other tag you want. Tags also don't have to be tied to a host and can also relate to a specific volume, mysql index, etc...
(Note to HN, this is a feature of datadoghq - pup will gladly collect and filter on tags, but won't aggregate them, yet)
We designed pup to be first and foremost accessible to developers, but it will work just the same on production systems.
Once you get addicted to metrics and want more aggregation / graphing / alerting / analysis capabilities, there's a number of open-source components you can pipe your statsd data into. Or you can use our own http://datadoghq.com service for that.
We've been through a number of data stores ourselves, starting with Postgres back in 2010 - then on to Redis + Cassandra before we built our own. But that's a story for another post...
http://datadog.com