Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug?] uploaded monot5-large-msmarco-10k and monot5-large-msmarco model is exactly the same #330

Open
soyoung97 opened this issue Oct 17, 2023 · 2 comments

Comments

@soyoung97
Copy link

soyoung97 commented Oct 17, 2023

Hello, I was trying to evaluate the monot5-large model on the beir benchmark, but I noticed that the uploaded models - 10k and 100k models - has exactly the same model. (when I do diff on terminal, it shows no difference)
image

Also, the performance of large models is a lot lower than the performance of base models - Could you check if the two uploaded large models are accurate? (maybe relevant to the previous issue)

Thank you!

@soyoung97
Copy link
Author

Another example showing the weight of two models is exactly the same: (code from here)

image

@KID-22
Copy link

KID-22 commented Jan 3, 2024

I ran into the same problem, I tested the 10K and 100K monoT5 on BEIR and found the results were the same, and both were very low, close to zero

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants