tinybert task pred_distil dev data loader is overwritten by mnli-mm

[TobiasLee]

According to your code in task_distill.py, the evaluation on MNLI-mm dataset will overwrite the original dev loader of MNLI-m, thus the best checkpoints are selected based on the MNLI-mm instead of the MNLI-m.

[itsucks]

Yeah, this is a bug

[iMountTai]

@TobiasLee Have you ever encountered a situation where mnli task acc remains unchanged？

[programmingLearner]

@TobiasLee Have you ever encountered a situation where mnli task acc remains unchanged？

They may not encounter this issue if they use older versions of transformers instead of dataset lib to fine tune the Teacher. I’ve solved this here: #229