- decompress (users|posts)
- split into batches of 10,000
- xsltproc the batch into sql statements
- pipe the batches of statements into sqlite in parallel using flocks for coordination
On my M1 Max it takes about 40 minutes for the whole network. Then I compress each database with brotli which takes about 5 hours.
This site is on a Cloudflare R2 bucket because (and only because) they have free egress. While not datacenter sized some of these files are large. Just opening up your credit card to 10 cents a gigabyte will be a bad time anywhere else.
I mean fine, write an article that overweights recent events to weave a tale of the decline of Stack Exchange...but don't leave out the moderator strike, the data dump fiasco, or the CEO that simply doesn't get it. Jon Ericson's writing on the issue is much more informative. https://jlericson.com/
PCA/SVD aim for maximizing explained variance, not preserving distance. They tend to "preserve" large distances at the expense of smaller ones, but that's not an explicit goal, nor can you bound the distortion. The [answer here](https://stats.stackexchange.com/a/176801/60) gives a pretty good intuition about why.
You can also compare them on computational complexity, where random projection (O(numPoints * numOriginalDimensions * numProjectedDimensions) smokes PCA or SVD which are cubic in the number of original dimensions.
And then there's simplicity. The random projection method turns on sampling from a normal distribution and then doing a matrix multiplication. There's a whole lot more about PCA to understand (standardizing your data, calculating the covariance matrix, eigenvector decomposition). I doubt I could implement it correctly myself, and I surely couldn't do it in high dimension.
We have offices in NYC, London, and Denver. If I'm remembering correctly, I can count the number of developers who've left the company on one hand. We currently have ~20 engineering staff in the NYC office. Retention doesn't seem to be a huge problem.
I'd also like to point out that...we're a largely remote company. We don't index our salaries to the most expensive time and place to hire developers ever. We don't have to support Bay Area rent, or duel it out with AppAmaMicroGoog to keep our people from getting poached. We hire people everywhere in the world and through that lens, we're competitive. I live like a king in Buffalo, NY.
Hey, I'm the Jason Punyon drob's gushing about in the post. I wanted to tell you that this is 100% totally realistic, everyone on the Data team at Stack Overflow besides Dave Robinson is living proof.
In the beginning the Data Science Team at Stack Overflow was just me and Kevin Montrose. We're both ~30, we had some skills that were data-ish, but they were general critical thinking and math skills. By no means were we statisticians or data scientists by trade like Dave Robinson. However, the data at Stack Overflow was very amenable to analysis, and we were able to ship a personalized machine learning product after a year of work.
That year was plagued by (my, mostly) engineering failures (http://jasonpunyon.com/blog/2015/02/12/providence-failure-is...) and a bit of unfortunate lack-of-empiricism, not "data science" problems. And post launch of Providence, the engineering remains the bigger issue. It's just hard to get this shit right. Our record on getting experiments right the first time is spotty, and at Stack Overflow levels of traffic you might be dumping a week or more of time on a single screwup. We've had to rerun many experiments because of various things going wrong with the code (screwed up the A/B testing code, not counting the right things, visual issues with the ads we didn't catch during design, etc etc etc). There's a litany of things you've gotta get right engineering wise before data even comes into the picture.
So if you're gonna take the plunge, know that it's 100% possible but 100% difficult, and maybe not for the reasons you think.
#1 kind of happens naturally as we're all working on different things. There are "a lot" (< 100) of people slinging code/design/SRE/IT, but only a few work in the same areas at the same time and rapid good communication generally happens among those subsets of the overall team. We also seem to have perpetuated a tendency toward good citizenship, so we generally talk to people before traipsing through areas of the code we aren't familiar with.
#2 Absolutely. This is some basic principle stuff. We don't hire people to not trust them.
#3 Related to #2: Automated acceptance testing is done as is deemed appropriate by the person developing the system. I've been on teams that valued and developed automated testing more, and less. My personal experience at the company has been it is neither necessary nor sufficient for success.
Much more important than any pre-deploy automated testing is our effort to monitor what's deployed (both in terms of software/hardware metrics and business goals). Bosun (http://bosun.org), developed by our SRE team, gives us some pretty great introspection/alerting abilities. I'd be incredibly sad to not have it. Bosun monitoring combined with the ability to have a build out in <5 minutes keeps me pretty happy.
A new site may come to replace it, but what of the 10MM+ questions and scads more answers created by the Stack Overflow community over the last 7 years? Without the CC license all that hard work gets lost in the case SO goes rogue.