1. Only for the first version, not for this version. I am sorry!
2. Yeah ours is guaranteed ok, as we wrote code to generate it basically just from plain torch ops. The code to run inference is available, just not the training code and data generation.
3. We have put it to work on time series data, which is very business relevant for example https://github.com/liam-sbhoo/tabpfn-time-series, and we have a table in the Appendix with all datasets we evaluate on in our main analysis to give you some ideas for possible datasets.
Yes, there are normalizations applied before the features are fed to the neural network. Additionally, the neural network is trained on a very diverse set of artificial datasets.
Author here: The new introduction of attention between features did make a big impact compared to the first variant of TabPFN. The old model handled every feature like it was completely different to be feature 5 vs 15, but actually features are typically more-or-less permutation invariant. So the logic is similar to why a CNN is better for images than an MLP.
To put it very simply, the trick is that while the others train a new model for each problem, TabPFN is pre-trained to handle any kind of problem on the fly.
To draw a parallel to NLP: previously people trained a neural network for each kind of text classification they wanted to do, but then LLMs came around that pre-trained to learn to perform new tasks on the fly.
Similarly, TabPFN learns to do new tasks on the fly just from the context (dataset) given.
Training and prediction in these models is by default one and the same, similar to how the prediction of the next token in an LLM is not split into learning from context and then doing the actual prediction.
There is a way to split this even up, though, then the predictions, I believe, take something like 1/10s for medium-sized datasets.
No, it is *much* stronger, a different architecture and scales to 10x the number of examples. It can also do regression now, and handle categorical features. Please, have a quick look at the abstract before making such claims.
I think I was not very specific, but I think there is a lot of video on YouTube that does not make any money for the producers and in the past YouTube also did not show ads for these videos, but now they show Ads even if the producers of the content don't receive any money. I watch mostly lectures and niche videos on YouTube, I am pretty sure for most of these videos, the only entity getting Money out of it is YouTube.
This trick “they found” is part of the standard torch implementation of multi head attention, namely it is called, add_zero_attention. They add a zero to the logits, resulting in a one in the denominator as e^0=1 https://pytorch.org/docs/stable/generated/torch.nn.Multihead...