kNN works differently in scikit-learn 1.0

[markotoplak]

scikit-learn introduces some changes. @janezd fixed the tests to pass in #5608.

The failed ROC test points out some changes in the underlying learners. Here I tried replicating the old version of the test in workflow with old scikit. This one seems to correspond to what the tests were expecting.

And here is how it looks with new scikit-learn.

I find differences in kNN disturbing. It would be prudent to also check other learners.

[markotoplak]

Hm, I saw the following warning in the scikit docs:

Regarding the Nearest Neighbors algorithms, if two neighbors and have identical distances but different labels, the result will depend on the ordering of the training data.

I tried playing with shuffling the data in the following workflow (the Data Sampler is set to sample 100% of the data) and saw that both the old and new versions are sensitive to the order of the data. The degenerate case we can observe on 1.0 on titanic disappears when the data is shuffled. Makes sense: titanic contains many repeated rows and is nicely sorted.

So perhaps we don't need to worry about it.