We're currently using Apache Kafka which is working out great, but does include a broker as a middleman. This is good for us since it's designed to accumulate unconsumed data (or even roll it back) and still provide good performance - rather than having that happen on the producers or consumers.
It would be cool to see some stats on throughput and latency relative to # of producers / consumers and amount of data currently accumulated in the producers (since there is no middleman).
i'm curious because right after i registered, i received an email with the subject "Scouts: Please confirm subscription" which goes to a listserv and has a reply to: will -at- vinova.sg.
We've been using Pig extensively for our MR queries and we're very happy with it. While i haven't used Cascading, i do like the pipelining approach of Pig more than the SQL-influence method of Hive. It lends itself better for meeting our needs.
Testing is a good point. If you're running Pig through their APIs it is definitely easier to test than command-line running scripts. We've written test code that reads and runs pig scripts through the API using fixed sample data (stored in HDFS for easy access), read the results, and compare it to expected results (also stored in HDFS). Honestly, you don't need too much input data to prove the correctness of the query.
Remember that Pig also supports placeholders in your scripts so you that you can set them in run-time to define input/output paths, etc. This makes testing easier.
Dependencies can also be stored in HDFS which makes it simple to run your scripts w/o the need to distribute jars around.
This doesn't actually discuss how Facebook manages and maintains their 100PB of storage, rather just talks a bit about the namenode SPoF issue. Disappointing article for sure.
Just please consider that XPath needs to store the items in a tree structure first (XML needs to be parsed and loaded into memry) before it can be useful. Running that on a large log file would be an interesting performance experiment.
It's nice to see Amazon entering this space as they do have the expertise to keep this up and running as well as provide the scale needed as apps grow.
What i always tho is the ability to run search queries that also involve dynamic grouping (like grouping by random combinations of facets) and providing those aggregated results.
Only thing i've seen that can do this "on the fly" is SenseiDB. CloudSearch/Solr/etc seem to need preprocessing to get this right.
There are quite a few other performance related points to think about for Solr speed for queries and indexing.
Here are some that come to mind right now that are very useful:
- Be smart about your commit strategy if you're indexing a lot of documents (commitWithin is great). Use batches too.
- Many times, i've seen Solr index documents faster than the database could create them (considering joins, denormalizing, etc). Cache these somewhere so you don't have to recreate the ones that haven't changed.
- Set up and use the Solr caches properly. Think about what you want to warm and when. Take advantage of the Filter Queries and their cache! It will improve performance quite a bit.
- Don't store what you don't need for search. I personally only use Solr to return IDs of the data. I can usually pull that up easily in batch from the DB / KV store. Beats having to reindex data that was just for show anyway...
- Solr (Lucene really) is memory greedy and picky about the GC type. Make sure that you're sorted out in that respect and you'll enjoy good stability and consistent speed.
- Shards are useful for large datasets, but test first. Some query features aren't available in a sharded environment (YMMV).
- Solr is improving quickly and v4 should include some nice cloud functionality (zookeeper ftw).
Curious...what throughput do I get when I use your service? Twilio limits you to 1 SMS / second unless you get a short code (30 SMSs / second) or otherwise use multiple numbers.
Do I get any benefit from you guys regarding this?