Add results of voyage-3 and voyage-3-lite #35
Conversation
There was a problem hiding this comment.
Looks great! Seems like these models are not in the mteb models folder. We would love to have them added to ensure reproducibility
|
@Liuhong99 seems like tests fail due to missing model_meta.json file |
|
Thanks for the reminder! @KennethEnevoldsen Where shall we put into the |
|
I think this is blocking the leaderboard GitHub now. We should add a test to block the leaderboard from allowing it to pass if the results aren’t present. @Liuhong99 it should be able to go in the no_revision_available folder, I think. Let me know if you’re able to do this, otherwise I will revert the leaderboard commit until we’re ready. |
Head branch was pushed to by a user without write access
|
@orionw Added |
|
Thanks! It's close, the names have to match in the folder name the model_meta.json -- see the test error:
e.g. the |
|
Got it; thanks! Fixed the names. |
|
Thanks @Liuhong99! I don't think the remaining errors are yours, pinging @KennethEnevoldsen if he knows. Seems like some issue with the test runner tbh. |
|
@KennethEnevoldsen @orionw After fixing the results format, all checks have passed! Thanks guys! |
| @@ -0,0 +1 @@ | |||
| {"name": "voyageai__voyage-3-lite", "revision": "no_revision_available", "release_date": "2024-09-22", "languages": null, "n_parameters": null, "memory_usage": null, "max_tokens": 32000, "embed_dim": 512, "license": null, "open_source": false, "similarity_fn_name": null, "framework": [], "loader": "VoyageWrapper"} | |||
There was a problem hiding this comment.
Does it use the MTEB voyage wrapper?
There was a problem hiding this comment.
(if so I believe we can merge this in - otherwise we can simply leave it empty)
There was a problem hiding this comment.
Yes confirmed all results come from mteb/models/voyage_models.py.
They were removed until the results were added, [now done](embeddings-benchmark/results#35 (comment))
* Re-Add Voyage 3 Models They were removed until the results were added, [now done](embeddings-benchmark/results#35 (comment)) * Spacing fix
resultsresults.pypaths.json