Cool, glad to have helped. It seems I have caused a further minor point of confusion though, so a correction to the correction:
The original Nesterov Accelerated Gradient paper is about gradient descent, it's just not about stochastic gradient descent. It's useful to make the distinction between "traditional" optimization methods like Newton's method, Conjugate Gradient, BFGS and so on, which are all gradient descent methods in the sense they require at least a full gradient calculation per iteration, and a lot of the algorithms mentioned in the article, which are suitable for stochastic gradient descent and originate with the deep learning community (there's nothing to stop them being used elsewhere, it just doesn't seem that common).
Some extra (unnecessary) detail on NAG to put things into a bit more context, if you are so inclined:
Although NAG has received a fair amount of theoretical attention, as far as I know it isn't widely used practically because its convergence properties rely on an exact line search and a rather specific schedule for its momentum-like term.
The Sutskever contribution is interesting because first, it expressed the NAG formula in a way that could be easily understood by machine learning practitioners. Then, by moving the procedure a half step, they showed you could think of it as momentum followed by gradient descent.
It's always useful to see different SGD methods written with a consistent nomenclature. A few thoughts:
1. Is the 1999 Qian paper on momentum really the most appropriate one, given the comparison of the publication date to NAG? As a cursory examination of the paper reveals, momentum has been used for a long time before 1999!
2. Similarly, the original NAG paper isn't about stochastic gradient descent and doesn't really use the equation as written. A more appropriate reference is to the Sutskever, Martens, Dahl and Hinton paper of 2013 http://proceedings.mlr.press/v28/sutskever13.html which is the publication that described/reworked NAG in this way.
"There may not be one perplexity value that will capture distances across all clusters—and sadly perplexity is a global parameter. Fixing this problem might be an interesting area for future research."
There are some suggestions in the literature for fixing this. Michel Verleysen's group suggested a "multi-scale" approach:
Their approach is to calculate the input probabilities using multiple perplexities and use the average. They also suggest tweaking the output probabilities, but it uses a free parameter that isn't present in the standard formulation of t-SNE (their suggested algorithm takes the same approach as t-SNE, but uses a different cost function and output weighting function).
The original Nesterov Accelerated Gradient paper is about gradient descent, it's just not about stochastic gradient descent. It's useful to make the distinction between "traditional" optimization methods like Newton's method, Conjugate Gradient, BFGS and so on, which are all gradient descent methods in the sense they require at least a full gradient calculation per iteration, and a lot of the algorithms mentioned in the article, which are suitable for stochastic gradient descent and originate with the deep learning community (there's nothing to stop them being used elsewhere, it just doesn't seem that common).
Some extra (unnecessary) detail on NAG to put things into a bit more context, if you are so inclined:
Although NAG has received a fair amount of theoretical attention, as far as I know it isn't widely used practically because its convergence properties rely on an exact line search and a rather specific schedule for its momentum-like term.
The Sutskever contribution is interesting because first, it expressed the NAG formula in a way that could be easily understood by machine learning practitioners. Then, by moving the procedure a half step, they showed you could think of it as momentum followed by gradient descent.