Hedge funds charge high fees, but a subset of them provide a product you can't get anywhere else: returns orthogonal to other investments. Sophisticated investors pay happily for it. What's being stolen?
At first I tried indexing all links (not just bitly ones) but the volume of results was too much to keep up with. I'm just using the plain old twitter search API.
And the project was for the Bitly API Contest, so I figured I'd just look at bitly links and leave it at that.
One thing I'd like to add to the site is automatic categorization of links, like on Google News. If anyone has suggestions on how to accomplish that, please comment.
I'm also struggling with intermittent 502 proxy errors on the server where Apache has problems proxying to news.arc on localhost.
The filtering approach (taken by Proxlet, for example) is pretty much the only scalable way to raise the signal-to-noise ratio of a Twitter stream. At first it seems hack-y and inelegant, but by using it in the field I've found that it is super effective.
I use a custom filtering script and a long list of regexes I've accumulated over time, expressions like this:
The core problem with having humans identify the best sites is that it doesn't scale. It's probably ok for big topics like travel or healthcare, but it shafts those users who are searching for long tail topics.