What are you talking about? Did you even look at their section on job search? It is most definitely a macro application.
Disregarding the fact that job search is a subtopic of unemployment, one of the main concepts in macro, you'll notice their model parameters are human capital, investment, and wage. The only quasi micro flavor of that model is search effort as these variables will almost always be in equilibrium based on some kind of an optimal stopping rule. Whether that rule has to do with finding a new job or discovering information about, say, the prices in the market dictates whether it would be a micro or macro use case. Again, if you actually looked at the model, their optimal stopping rule is when the person finds another job. And once again, the idea of jobs are the main ingredient in the concept of unemployment.... a macro topic. Not to mention, whether or not I'm searching for a job dictates whether I'm factored into the unemployment rate or not.
I thinks it's great that 2 well known economists with household names in the academy have taken to marketing programming to economists. I just wish they would have called this venture something other than Quantitative Economics when they launched it last year. It's not that it's the wrong name, it is just misleading.
A better name would have been Quantitative Macroeconomics, because that is clearly the target market. Yes, I know there are some generic topics in there like Linear Algebra that both micro and macro economists could use, but it's not like there is a lack of numpy and scipy examples on the Internet.
You bring up a good point in that the use cases for each language varies. R is a good example because one of the main reasons it is chosen is due to the fact that it already has the largest set of drop in econometrics operations for tons of extremely specific situations. In any other language, you'd often have to code a bunch of complex linear algebra by hand, which would totally negate any marginal benefits due to computation time.
In terms of econometrics, Python is going to surpass closed source software packages like SPSS, SAS, eViews, etc soon simply because the rate at which econometric procedures are being implemented in Python is growing steadily (e.g. via a statsmodels/pandas/numpy/scipy based stack). I don't think Python will ever pass R in this respect though as R has a large share of the statisticians helping add the implementations relative to Python.
I used to be gung-ho about convincing economists to use Python (started econpy.org in the 1st year of my PhD in economics, stopped updating it after 2 years). But now I just don't care. The vast majority of economists are horrible programmers, however most know how to script in at least 1 language, and a small number of them are actually really good at scripting. An extremely small number of economists know they're way around at least 1 entire general purpose programming language.
It seems that every time this gets brought up online a sea of infamous Fortran-programming-economists always tell you how fast Fortran is compared to everything else, signing it with "Fortran or GTFO". I've met hundreds/thousands of economists and econ grad students at schools ranked from 1 to 200, and I don't recall a single one that actually used Fortran (I'm sure there is a non-zero quantity -- I probably just haven't cared to talk with them b/c they are most likely macro theorists).
The fact of the matter is that in economics graduate school, your professors could care less what language you use. It isn't an algorithm competition, it's a story telling competition. But the type of story telling economists do is no less noble than writing elegant algos. Writing a story based on economic mechanisms and behaviors and supporting it with quality data, sound statistics, and logical/exact theories is no simple task.
I have been researching the used car market for the last year in an academic lab. I would love to share stories about data collection and management if Carlypso folks are interested. Not looking to be a consultant or being consulted or any kind of a "gig". As I'm sure they know, used car data is a crazy world of imperfect data points, especially as the car becomes older (>6 years old I'd say data quality begins to break down fast). What proportions of your data are scraped, purchased, or maybe even obtained from a free API like Edmunds.com? I doubt you're using Edmunds as I didn't catch any affiliation info on the site.
I laughed at the "oh yeah that makes sense" story of the big truck being priced higher in TX. My goto story is always the price of convertibles in the winter in MN vs. TX, or the price of an AWD sedan in the winter in MN vs. TX.
Speaking of AWD sedans, I know it's a rare car so this is nitpicky but your 2008 Saab 9-3 data classifies a Turbo X as an Aero. Also the Turbo X trim isn't listed for the wagon. At any rate, your estimate for the Sedan Turbo X is about 20% too low. Granted they only made 600 for the US market, I was still bummed that a car which is so dear to me had this issue. Haha, I'm not trying to be that guy whatsoever! In fact I was happy when I saw Saab was even listed as a Make as many don't even acknowledge them anymore! :(
Anyways let me know if you guys are up for a friendly talk.
Thanks for the info and I do look forward to the new API!
However my question still remains with regards to historic posts/comments. The historic aspect is really the import element here. Generally speaking, building an ngram viewer requires a collection of texts over time, with each text having some kind of metadata that is categorical, boolean, datetime, or numeric. Categorical data can always be made of numeric data by creating bins -- i.e. posts by people with karma or a ranking of 1-50, 51-150, 151-300, etc at the time the comment/post was created. Datetimes can also be made into useful categorical variables for an ngram viewer such as day of the week (to spot weekly seasonality trends) or day of the year (annual seasonality trends).
If I was allowed, I would be willing to write a scraper/crawler to discover as many historic threads (since: threads -> comments) as possible using HNSearch, but this could take a long time depending on rate limits and/or be subject to unknown biases within my discovery method. I'm sure you can understand why a "top-down" approach like a database dump would make for a much higher quality corpus than attempting the "bottom-up" approach of a crawler. I have no idea if a "database dump of everything" is even feasible as I don't know anything about the HN's backend infrastructure. However, if it is feasible, then I'm certain that I can work with whatever would be available. Adding structure to unstructured data is my bread and butter.
I really think this would be a very cool tool that a lot of people would enjoy, so I'm willing to do what is needed on my end to help make it work. After all, I'd be on the clock while working on this rather than just a hobby project, so the incentives are definitely aligned on my end.
If you want to discuss anything in private, I can be reached at the following reversed address: moc{dot}liamg{at}yalkcin{dot}wehttam
eBay's completed listings is definitely one of the best applications of obtaining sales data on the Internet that I'm aware of. Besides that, in some cases there are ways to imperfectly estimate quantities when best seller rankings are available (e.g. at Amazon) -- Chevalier and Goolsbee where the first to suggest this approach back in 2003.[1]
As you mentioned, monitoring half-life is another imperfect approach, but it is of course plagued by false positives (a listing goes away but no sale was made). There was a Google Tech Talk many years ago where some economists took this approach[2], except they were looking at pricing power instead of measuring quantity sold.
In your original business model you wanted to understand the price of everything. In what ways did the problem of a lack of information on the demand side come up? That is, it is easy to scrape the price in many markets (supply side), but what kinds of conversations came up within your team about the lack of information on how many units were actually sold at a posted price?
By the way, glad to see you guys were able to make a business out of crawling. I've landed a handful of freelance gigs since leaving grad school based on scraping data for clients, but never tried to expand it to anything beyond consulting projects.
As a long time pandas user, I'd say this is one of the better write-ups I've seen that illustrates the versatility and functions of the Series and DataFrame objects without being too long winded.
Just one thing to point out regarding the final example: read_csv will actually fetch a URL if it's given as input, so there is no need to use urllib2 and StringIO. Instead, you can just do:
Minor in economics. Skip on taking extra macro classes and instead take as many econometrics and micro that you can. Economics (or another quantitative social science) is often the missing ingredient for a solid data science background. Sure physicists have good quantitative skills, but you also need to understand what the important/relevant [business] questions are to ask.
Although not primarily in the discovery space, if you're a user of arXiv, you may be interested in the n-gram viewer our lab made: http://arxiv.culturomics.org
Despite this clip being a classic classroom example in a game theory course, it's technically not a prisoner's dilemma, for a couple reasons. First, in the PD the players cannot communicate. The ability to communicate makes it more of a signalling game since a player can say/do something to alter the other players' beliefs about their type.
Secondly, in the PD both players have an incentive to defect from their cooperative strategy (stay silent), which results in a Nash equilibrium where both players are worse off. This is, after all, why the prisoner's dilemma is so interesting -- by playing their dominant strategies, both players end up in a worse situation.
In this clip, stealing is a weakly dominant strategy. That is, if I know the other guy is going to steal, I can't be made any better or worse by changing my choice to split or steal. If I know the other guy is splitting, then I of course would want to steal (which is why splitting is an unstable strategy).
Porsche is owned by the Volkswagen Group who has a completely different strategy than Tesla. Volkswagen already makes money off makes and models marketed across the spectrum of consumers (from Skoda to SEAT, VW, Audi, Porsche, Bentley, Lamborghini, Buggati) -- all of which are owned by VW.
The Volkswagen Group doesn't get nearly as much press as it should, but I suppose the incumbent doesn't need it -- they already sell hundreds of thousands of vehicles world wide every year.
That being said, Tesla certainly has a long ways to go, but I think they have a good strategy going forward. Not to mention, I think the Model S is a beautiful car.
I'd encourage any economist that has a knack for programming to really put in some time towards the area. I've owned econpy.org for about 3 years now (although I haven't updated it in a long time). I also own economics.io and run econpy.blogspot.com.
It's much easier (relatively speaking) for an economist to pick up some programming than it is for a programmer to pick up some economics. Economists are already familiar with the types of questions that are important to economists, and more importantly, how to frame them. The trouble with economics is that you can't just pick it up overnight as it is a way of thinking more than it is a tool set. Programming on the other hand is something that you can "get working" overnight (economic programmers don't need to be algorithmic theorists -- they just need to be really good at getting/scraping data and organizing it so they can run analyses on it).
Over a year ago, I dropped out of my PhD program in economics because I was not at a school that was going to allow me to do the econ/cs type work I was working on. Leaving my PhD program was one the best things I ever did because it has allowed me to pursue whatever I want to do with the skills I've acquired.
The problem with academic economics is that the data most economists use is so bad and outdated -- such as data from FRED, BLS, and other publicly available sources where everyone and their uncle can download the same CSV dataset that was aggregated by some government employee. The race then is to see who can put together the most elegant econometric model to handle all the issues with the data. The rules of the game change when you create your own dataset and thus have control over while variables to include, the aggregation, the frequency, etc.
Long story short, if you are an economist wanting to do programming, learn to adapt those skills in academia (best way is to find a great advisor -- if there isn't one in your economics department, check the business school as bschool professors are often much more open to highly empirical analyses and care [marginally] less about getting the theory perfect). Or, if you want an easier lifestyle that is much more rewarding, ditch academia for the private sector. You'll find the economists in the private sector to be much more knowledgeable about cutting edge technologies and willing to listen and learn from what you have to say.
Fun fact about Aeroflot: The slogan for Aeroflot back in the days of the USSR was "You Have Made The Right Choice", which is ironic since Aeroflot was the only airline in the Soviet Union.
It's my creation. :)