Handle different number of classes between y_true and y_pred

[valosekj]

Handle ValueError: y_true and y_pred contain different number of classes

[SomeoneInParticular]

Is this finalized? Or are you still testing it in-draft?

[valosekj]

I believe it can be tested by you and merged

[SomeoneInParticular]

Apologies for the delay, wanted to really dig through everything before accepting this PR.

I don't think this is going to work long-term, as there is an edge case this does not account for currently. Namely, where their are unique classes in both the training and testing data. For example, if training has classes [A, B, C], and the testing has [A, B, D], right now the result for predict_proba for class C will be used to 'represent' the probability for class D in the testing dataset, which is inherently incorrect.

This is actually an issue with most of the supervised categorical metrics at the moment, and as such we will need to look into a more universal solution here... I'll dig into it later this week, but for now I don't think its smart to push this PR.

[valosekj]

For example, if training has classes [A, B, C], and the testing has [A, B, D], right now the result for predict_proba for class C will be used to 'represent' the probability for class D in the testing dataset, which is inherently incorrect.

Very good point!

Okay, let's not processed with merging.