Add anomaly.ReconstructionDetection

[sebiwtt]

Hey everyone,

I've been working on this idea for my bachelor thesis for a while now and I figured that it might be interesting to add this kind of anomaly detection algorithm to river.

The idea is to use some sort of online estimator to learn the "normal behavior" of data (in the use case I am researching, this is always time series data). This "reconstruction model" of the data is then used to forecast/predict a target based on some input features, while the actual ground truth is known.

It then calculates the error of the prediction and the ground truth (while keeping track of the mean and standard deviation of all the errors it has made). When the estimator is good at "reconstructing" the data that error should always have a relatively low mean and a small standard deviation. But when there is an anomaly in the data the prediction will be further away from the ground truth resulting in a larger error. When using the squared error this difference between the mean error of the estimator and such a large error (in case of an anomaly) will be fairly easy to detect. My first approach here was to build some sort of threshold: when an error is larger than the mean + 3 times the STD the example will be classified as anomalous. I also have some other ideas but this is the basis.

I would highly appreciate any input on this topic since this will be the basis of a large part of my bachelor's thesis. I hope that you guys find that this is a nice idea to add to the library.

[MaxHalford]

Hey there Sebastian.

I think that's a really good topic. It's a solid model: if you have a good model, then comparing its prediction to the ground truth is a good way to detect an anomaly. Then again, a ground truth isn't always given. But you could pick any feature and use it as a label. So yes it's a solid approach to explore, especially online.

I would differentiate the scoring from the thresholding: a good anomaly detector is not necessarily one that outputs 0/1 correctly. It outputs a score, and then it's a separate concern to threshold that score into a 0/1. That's how all River's anomaly detectors operate. See what I mean?

[sebiwtt]

Thanks for the feedback 😀 I see where you're getting at with the scoring. Now I'm wondering how to go about it. Would it be feasible to compute a score ranging from 0 - 1 based on how far the error of single example is away from the mean (and STD)? At least thats the first thing that comes to my mind.

[sebiwtt]

And regarding the estimator that does the forecast: I though about making this "plug and play", so that maybe there is a default, but when instantiating an object of this new class the user can pass a model that is to be used (based on the data at hand)

[sebiwtt]

@MaxHalford, I fooled around with the scoring. What worked quite well is to have a fixed value be mapped to a score of 1, for example if the error >= (mean + x * STD). The user could test what x works for them. All the values in between the mean and this cap will be distributed linearly so that we have a scoring ranging from 0 to 1.