I think some of his statements are a little off-base, for instance, regarding the choice of tanh() for the activation function he says: "But mostly it’s 'because it works like this and it doesn’t work when we change it.'"
People have spent a fair amount of time investigating the properties of different activation functions, and my understanding is that tanh() is generally not in favor, and ReLU (rectified linear) is probably a better choice for many applications, for reasons that are well understood. Maybe the author isn't all that familiar with the field?