Thanks for the feedback, folks. I'm actually a bit surprised this hit hacker news without any of us authors posting it, but heck, I'll jump in.
Think of this as an experiment in exploring a document collection at a higher level than search. Specifically, what you're seeing is Stanford's dissertations through the lens of a text model that tries to distill high-level patterns in the data. It doesn't always succeed, but it often hits the mark. There are plenty of ways that the visualization and the underlying text model could be improved.
For the curious, I'll tell you a bit more on how the numbers are computed: we build a unigram language model of the contents of every Stanford department based on their dissertations. Then, we posit that every dissertation comes from a mixture of those department models (using a supervised topic model, Labeled LDA). This lets us infer, for every dissertation, a weighted mixture of departments that best characterizes that abstract. So, say, dissertation X is 60% computer science, 20% physics, and so on. These scores are aggregated to compute the average similarities between departments, and are sliced to give the view over time.
So what you're looking at is, essentially, a visualization of word overlap between departments measured by letting the dissertations in one department borrow from words from another department. Which departments borrow the most words from which others?
When you zoom in two-levels (click on a department twice), individual dissertations are plotted on a line between each dissertation's home department and it's next highest scoring department. So the relative position of two dissertations near each other is not meaningful unless they are on the same radial line. Dissertations from other departments that have a high score for the central, focused department, are also shown.
For instance, take a look at Computer Science in 2005. You'll see three dissertations along the radial line to Linguistics - those are the three students that graduated from the Stanford NLP group that year. There are plenty of other places you find similar things that work, and also places where things don't work as nicely as you'd expect.
The visualization Jason built was really interesting from the text modeling perspective, because it let us experiment with many model variations (lda, tf-idf, etc etc) to see how well each matched our intuitions. This model, though still wanting, was by far the best. Good enough, even, for us to put online for the world to play with, and for hacker news to pick apart ;)
Think of this as an experiment in exploring a document collection at a higher level than search. Specifically, what you're seeing is Stanford's dissertations through the lens of a text model that tries to distill high-level patterns in the data. It doesn't always succeed, but it often hits the mark. There are plenty of ways that the visualization and the underlying text model could be improved.
For the curious, I'll tell you a bit more on how the numbers are computed: we build a unigram language model of the contents of every Stanford department based on their dissertations. Then, we posit that every dissertation comes from a mixture of those department models (using a supervised topic model, Labeled LDA). This lets us infer, for every dissertation, a weighted mixture of departments that best characterizes that abstract. So, say, dissertation X is 60% computer science, 20% physics, and so on. These scores are aggregated to compute the average similarities between departments, and are sliced to give the view over time.
So what you're looking at is, essentially, a visualization of word overlap between departments measured by letting the dissertations in one department borrow from words from another department. Which departments borrow the most words from which others?
When you zoom in two-levels (click on a department twice), individual dissertations are plotted on a line between each dissertation's home department and it's next highest scoring department. So the relative position of two dissertations near each other is not meaningful unless they are on the same radial line. Dissertations from other departments that have a high score for the central, focused department, are also shown.
For instance, take a look at Computer Science in 2005. You'll see three dissertations along the radial line to Linguistics - those are the three students that graduated from the Stanford NLP group that year. There are plenty of other places you find similar things that work, and also places where things don't work as nicely as you'd expect.
The visualization Jason built was really interesting from the text modeling perspective, because it let us experiment with many model variations (lda, tf-idf, etc etc) to see how well each matched our intuitions. This model, though still wanting, was by far the best. Good enough, even, for us to put online for the world to play with, and for hacker news to pick apart ;)