Are you sure about the padding? On page 1746, bottom right it says "padded as necessary". And intuitively it makes sense that all your inputs need to be the same size for a CNN.
This is a great point, would be worth further investigation. And I agree with your general interpretation. It would be interesting to look further at where CNN is failing to detect bad ones and where the feature engineered one picks them up.
I think deep learning can be seen as a class of machine learning techniques with more flexibility and which uses neural networks (usually with quite a few layers).
50K is not that tiny, but I'd say in the hundreds is pretty small. I'd say hundreds of training examples is pretty normal in academia, but typically quite small in industry, particularly for problems that are actually quite abstract, such as this (sentiment is another similar problem).
This is really great idea. Actually if there is something you can share along these lines, that would be amazing. I know Crowd Flower has a great "internal only" tool, which is kind of similar to what you are designing, but you have to pay for it. Actually I think there is a huge need for a generic tool along the lines of what you have started to build.
Great question and I share your intuition but I think its all properly regularizing your model. I guess for neural networks, Dropout works really darn well as a regularization strategy. I could have tried to see whether performance dropped significantly without dropout.
Great point! I considering using fasttext as a baseline, however in practice fasttext really didn't work well at all with the small data set, much worse than the tfidf baseline. I think Fasttext's classification approach might not work well with such a small dataset. I'm not sure but I suspect its because it tries to learn embeddings - but there just isn't anywhere near enough data for that. I'd love an outside perspective on this.