Yes, the database is all on the machine's local 512 GB hard drive. I did store the downloaded flat text files to an external drive and loaded them into the db from there.
Sure, there are undoubtedly lots of examples of businesses that opened in desolate areas and created new taxi activity where there had been none previously, I just happened to focus on Williamsburg for the post.
Another idea that I didn't get around to doing was to look at concert venues and measure taxi traffic around particular concerts to see if it would correlate to bands' overall popularity
Yep, all on a 2012 MacBook Air. Data size was over 400GB with indexes
Simple queries on indexed columns of the trips table take a minute or two, more complicated queries that require a full sequence scan can take up to a few hours
Mostly to keep the app simple and still deployable on the free Heroku plan. Sure, it could handle multiple locations, but then it would quickly outgrow the free tier
I think the projection becomes relevant when you're looking at "straight" borders like Colorado. A great circle will bend slightly north of the northern border, so you can actually just barely touch Wyoming and Nebraska, but I don't think the projection is significant when looking at some of the more irregularly shaped states
Simulation author here with some additional analysis using a faster distribution of request times. If you use a distribution with median 50 ms, 90th percentile 225 ms, and 99.9th percentile 898 ms, then you need 30 intelligent dynos to handle 9000 requests/minute without queueing. In the same scenario with 30 naive dynos, 44% of requests get queued.
I'm not familiar with minimum-of-two-random routing, but it does seem like assigning request to dynos in sequence would perform much better than assigning randomly (ie in a scenario with n dyno capacity, request 1 => dyno 1, request 2 => dyno 2, ... request n => dyno n, request n+1 => dyno 1, ..., repeat)
That'd be probably significantly better than the case of (request i => dyno picked out of hat) for all i
But you never want to have a queue on any of your dynos! A queued request means that a user is waiting with no response. If your goal is to have 0 (or less than epsilon) requests queued, it takes far fewer dynos if the requests are routed intelligently
If you have 10 dynos and 1000 simultaneous requests, the difference between naive and intelligent might well be reduced, but that's also a scenario in which your end user response times would be horrendously slow and so you'd need more dynos either way
I haven't worked with XBRL though, so am not sure if it's more involved