Current Time Series Anomaly Detection Benchmarks Are Flawed(arxiv.org)
arxiv.org
Current Time Series Anomaly Detection Benchmarks Are Flawed
https://arxiv.org/abs/2009.13807
2 comments
> This article is consistent with my recent experience with academics being a frenzy to produce papers riding the AI/ML hype, yet producing bunch of practically useless papers.
Much of CS is like this; useless papers with incremental improvements at best. Without having read the paper, I'd guess plain ARIMA etc. do well enough in univariate forecasts.
Much of CS is like this; useless papers with incremental improvements at best. Without having read the paper, I'd guess plain ARIMA etc. do well enough in univariate forecasts.
Thanks for your kind words (I am one of the authors)
For those interested, there is an expanded review/critique of datasets here [a]
There is also a video here [b]
[a] https://www.dropbox.com/scl/fi/cwduv5idkwx9ci328nfpy/Problem...
[b] https://www.youtube.com/watch?v=Vg1p3DouX8w&
There is also a video here [b]
[a] https://www.dropbox.com/scl/fi/cwduv5idkwx9ci328nfpy/Problem...
[b] https://www.youtube.com/watch?v=Vg1p3DouX8w&
I am currently reading your article with great interest as I am about to embark on implementing time series anomaly detection algorithm(s) for predictive maintenance applications. I would like to avoid unnecessarily complex algorithms that are unlikely provide real, practical benefits.
Please let me know if I can help.
A paper published today finds an anomaly in an "InternalBleeding" dataset, after setting eighteen parameters [a].
Could we find the anomaly with a completely parameter-free algorithm? As the figure below shows, the answer is YES, if you use MADRID [b].
One line of code >>MADRID(UCRAnomalyInternalBleeding)
So Yes, simple is better.
[a] Learning Rate, Dropout Rate, Dim Feedforward, Batch Size, Encoder Layers, Decoder Layers, Activation Func, Time Warping, Time Masking, Gaussian Noise, Linear Embedding, Phase Type, Self Conditioning, Layer Norm, Pos. Enc. Type, FFN Layers, Window Size.
[b] https://www.dropbox.com/scl/fi/hd9gt0xs8v8mrsx3upwd3/ICDM23_...
A paper published today finds an anomaly in an "InternalBleeding" dataset, after setting eighteen parameters [a].
Could we find the anomaly with a completely parameter-free algorithm? As the figure below shows, the answer is YES, if you use MADRID [b].
One line of code >>MADRID(UCRAnomalyInternalBleeding)
So Yes, simple is better.
[a] Learning Rate, Dropout Rate, Dim Feedforward, Batch Size, Encoder Layers, Decoder Layers, Activation Func, Time Warping, Time Masking, Gaussian Noise, Linear Embedding, Phase Type, Self Conditioning, Layer Norm, Pos. Enc. Type, FFN Layers, Window Size.
[b] https://www.dropbox.com/scl/fi/hd9gt0xs8v8mrsx3upwd3/ICDM23_...
The following quote from the article is striking: "Qiu et al. introduce a “novel anomaly detector for time-series KPIs based on supervised deep-learning models with convolution and long short-term memory (LSTM) neural networks, and a variational auto-encoder (VAE) oversampling model.” This description sounds like it has many “moving parts”, and indeed, the dozen or so explicitly listed parameters include: convolution filter, activation, kernel size, strides, padding, LSTM input size, dense input size, softmax loss function, window size, learning rate and batch size. All of this is to demonstrate “accuracy exceeding 0.90.” However, as we will show, much of the results of this complex approach can be duplicated with a single line of code and a few minutes of effort."