Template in Pre-Training vs. SFT

[serwansj]

Hi, thank you for your cool work!

After looking at your code, I wanted to know if I understood your use of the conversation template correctly. The templates found in https://github.com/NVlabs/VILA/blob/main/llava/conversation.py are only used during SFT, right? They are not used during the pre-training stage(s)?

[Lyken17]

We apply the template in both pretraining and SFT stage. Only in the alignment (stage1), we use the raw text image pairs.

[serwansj]

@Lyken17 Thank you for your reply! How are you applying the template in the case of the MMC4 dataset, i.e. what instruction prompt are you using in this case?