Perfect interview questions to get to the juice details haha!
The problem is that choice of compression is very much dependent on the sample size, so this is why just choosing the algorithm based on running benchmarks on the sample will be off.
However, computing certain statistics on the sample and then doing machine learning makes a lot of sense!
Obviously not for simple/cheap to compute features such as number of rows/columns which actually do not take longer to compute as the data gets larger.
But I actually do this for predicting the uniqueness of values! You basically want to get an idea of how many unique values you have per column. But.. if you take a sample this will give you completely wrong numbers.
You can see my approach here: https://github.com/kootenpv/shrynk/blob/6a8675061d82aa65fc3b...
Basically the formula calculates the uniqueness on a sample (e.g. 10000 rows), but then extrapolates the result to your actual data size. E.g. it finds 100 unique values in a sample of 10000, but in reality that means you maybe have 500 unique values.
Yea, I thought parameters per compression algorithm should indeed be added in a next version :) more compute but definitely an improvement.
I think pandas doesn't offer zstd as option with csv, but I'll check once more.
EDIT: indeed, it's missing in `to_csv` - seems like an oversight.
Yea I am aware of meta hyperparameter approach for ML, except they only focus on accuracy instead of also including train/prediction times in to the equation :) That's what I was referring to! (you can save A LOT of compute and zoom in on things that work if you can weed out slow / badly performing algorithms as part of meta learning hyperparameters).
To make it extra clear: by doing a lot of compute on different datasets and not only recording the accuracy but also time it took, and then by including that as dimension it will even give better results.
Spot on (I briefly touch on this in the article)! This is why I try to work with cheap-to-compute features. I used to calculate how unique all values were, but ended up taking a sample instead to speed that part up for large data!
Yea - that is related to genetic programming. That, and using auto-encoders for e.g. image compression are known approaches in "AI".
I'm particularly proud of this meta approach and I am actually thinking this could become huge: the same thing can be done for hyperparameter optimization in machine learning tasks.
Hyperparamter optimization is currently focused on minimizing cross-validation error, but using this concept you could have weights on accuracy, training time and prediction time (very similar to compression where the 3 dimensions are size, write time and read time), and then given a new unknown dataset you could predict what model/hyperparameters to use.
shrynk is not doing any compression itself, it's using ML to do a "meta" approach: it computes features on your data like how many rows, how many columns, how much duplication etc, and then predicts which existing compression algorithm (out of the available compressions) will be best given your requirements/preferences in terms of size, write time and read time.
Definitely! I really optimized for "no development time spent" and was just using pandas to extract html tables into csv and just store them :-). 2 lines of code really. I had no idea I would have it running for so long.
It was really just the example that made me wonder why I have to consider which compression would be best for files with my characteristics - but not saying it was best practice to begin with haha!
I am working on an open-source implementation that is based on the fundamental idea of privacy. The consumers should have full control eventually.
For now only Google etc have all the information on us, but the great thing is that because of GDPR we see a move towards us being able to download it.
The system I am building allows us to combine data from across companies. This is huge. We are talking so much opportunity. The real challenge is making it easy for people to onboard while still providing security.
Feel free to contact me if you're more interested.
I'm happy to share with you the result of this project, it certainly has been a lot of fun!
First of all it was shocking how many messages we had over such a long time (more than 300k!)... and it was very interesting to see the patterns occurring.
Next, I trained a bot for several nights (+ experimentation) that in the end does pretty well.
From the video, you can see that it doesn't take the current time into account, which actually turned out to be its biggest flaw, lol (no spoilers)!
Anyway, writing the "facebook driver" with selenium was fun too.
What turned out to be an interesting part of the model is that it had to split up a message into more messages (there's a difference between a newline and a new bubble), which I represented as a special character in my data.
The other way too - I had to make sure it did not reply to every message individually, but instead, I had it wait some time. It was considering messages starting with a capital letter as "mobile" (and allow a longer wait than non-mobile).