| „[…] was a friend telling me his LaTeX thesis took 90 seconds to compile towards the end“
Sure, but in order to iterate you won’t have to compile the whole document but can just keep the chapter you are working on by structuring it with \includes
This was a great read, thanks a lot!
One a side note, any one has a good guess what tool/software they used to create the visualisations for matrix multiplications or memory outline?
Okay, I see that for inference. But for training it shouldn't matter because I need to hold on to all my activations for my backwards pass anyways?
But yeah, fair point!
"[...] modern neural network (NN) architectures have complex designs with many components [...]"
I find the Transformer architecture actually very simple compared to previous models like LSTMs or other recurrent models. You could argue that their vision counterparts like ViT are conceptually maybe even simpler than ConvNets?
Also, can someone explain why they are so keen to remove the skip connections? At least when it comes to coding, nothing is simpler than adding a skip connection and computationally the effect should be marginal?
hey, off topic but can you explain or link a post which explains what the benefits of the alias -> function definition are over just defining the function directly?
Thanks!
Very cool!
However, I often feel like the process of generating the question/answer necessary for the Anki card is an important part of the learning process because it forces you do deeply think about the material and reflect on it. So I think there is a trade-off involved
Sure, but in order to iterate you won’t have to compile the whole document but can just keep the chapter you are working on by structuring it with \includes