How to use models with timm's arch #685
-
Hi, I want to use the whole clip model to get a zero-shot head, and then convert the vision part to a timm model, how can I build the model and are there any pretrained weights I can use? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
@ChChCh8 this is doable already. look at any of the open_clip/src/open_clip/model_configs/convnext_base_w.json Lines 1 to 19 in e7b39e4 |
Beta Was this translation helpful? Give feedback.
-
further @rsomani95 ans, all of these require a bit of massaging, but it's not much code
|
Beta Was this translation helpful? Give feedback.
-
I will point out, I found it was better to fine-tune the headless model (at least with imagenet as the target) than to use a zero-shot head. Also, starting the fine-tune from the zero-shot head did not seem to help much, it was shorter but I found it difficult to get a better end result. However, that could change with smaller target datasets... |
Beta Was this translation helpful? Give feedback.
@ChChCh8 this is doable already. look at any of the
convnext_*
models, they're all coming fromtimm
. Here's an example cfg:open_clip/src/open_clip/model_configs/convnext_base_w.json
Lines 1 to 19 in e7b39e4