While I can see this being a very useful skill for many, we must also remember sometimes reading slow is desired, especially when a deep understanding and analysis of a text is required (e.g reading and analysing literature)
not too great, i submitted a photo taken of me at graduation. it got the location totally wrong and was around 50%-70% accurate on my hobbies and interests. it was able to correctly guess my sexuality and ethnicity, which is rather unsurprising.
> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state
This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a lot.
I have a lot more CPU RAM in my PC, and this would likely increase the size of models I can train locally.
This should honestly have been implemented a long time ago. Much of academia is pressured to churn out papers month after month as academia is prioritizing volume over quality or impact.
wow - this is really well made! i've been doing research w/ Transformer-based audio/speech models and this is made with incredible detail. Attention as a concept itself is already quite unintuitive for beginners due to is non-linearity, so this also explains it very well
Wow. Arco was my first introduction into Arch-based systems - it taught me invaluable amounts of information. I run Arch Linux now, but Arco always has had a special place in my heart. So sad to see it go.