Performance on MMLU

[allanj]

Have you evaluated the performance on MMLU compared to original?

[chiayewken]

Hi, we have noticed slightly lower MMLU scores for declare-lab/flan-alpaca-xl compared to google/flan-t5-xl. This may be due to the zero-shot format of the Alpaca data compared to few-shot for MMLU, and we are benchmarking multiple models here:

https://github.com/declare-lab/flan-eval

[allanj]

Thanks a lot for the efforts

[allanj]

Are they evaluated using CoT prompting?

[chiayewken]

Hi, the evaluation is using direct prompting for MMLU

[allanj]

Btw, It seems you are doing few-shot prompting, am I right?

https://github.com/declare-lab/flan-eval/blob/47910e01894420971665eade0c92e3046b8f1f3d/mmlu.py#L142-L151

[chiayewken]

Yes, we used 5-shot prompting for MMLU based on the Flan-T5 paper