Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

198 Open 179 Closed
198 Open 179 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

GOAL: 1.0 French Benchmarks milestone A goal that other cards add up to
#820 opened Jan 27, 2025 by wpietri
Goal: French 1.0 benchmarks milestone A goal that other cards add up to
#819 opened Jan 27, 2025 by wpietri
Add evaluator type to hazard UIDs
#816 opened Jan 24, 2025 by bkorycki
modelbench should not accept degenerate or surprising option combinations
#814 opened Jan 23, 2025 by rogthefrog
@rogthefrog
1
Try to replicate production failures using Docker shenanigans
#811 opened Jan 22, 2025 by wpietri
French calibration - full prompt set
#810 opened Jan 22, 2025 by rogthefrog
@rogthefrog
2
Preserve existing run_data in standards file when regenerating the file
#805 opened Jan 18, 2025 by rogthefrog
@rogthefrog
1
update or kill PyPi releases
#804 opened Jan 17, 2025 by wpietri
Version number is off
#803 opened Jan 17, 2025 by wpietri
Start measuring how vendor-hosted model scores change over time
#802 opened Jan 16, 2025 by wpietri
Update supported Python versions
#800 opened Jan 15, 2025 by rogthefrog
@rogthefrog
1
Make modelbench anonymous (offline) practice runs use the 10% extract instead of the full practice set.
#798 opened Jan 15, 2025 by bollacker
1
UI to access benchmarks
#796 opened Jan 14, 2025 by wpietri
run the french models (first pass)
#795 opened Jan 14, 2025 by wpietri
1
For common models, document their frenchness
#794 opened Jan 14, 2025 by wpietri
French calibration - first pass
#793 opened Jan 14, 2025 by wpietri
@rogthefrog
2
Test switching from threads to processes
#792 opened Jan 14, 2025 by wpietri
Add pypi build test back to modelbench
#790 opened Jan 13, 2025 by rogthefrog
@rogthefrog
Improve modelbench CLI input validation + secret checks
#789 opened Jan 13, 2025 by bkorycki
Fix cache corruption bug
#785 opened Jan 9, 2025 by wpietri
Ensemble tests are registered in modelgauge
#782 opened Jan 8, 2025 by bkorycki
Improve how locales and prompt file name interact with each other
#781 opened Jan 7, 2025 by rogthefrog
@rogthefrog
1
Fix modelbench cache bug bug Something isn't working
#778 opened Jan 6, 2025 by bkorycki
fix the smoke tests
#777 opened Jan 2, 2025 by wpietri
Replace Locale object with just strings
#764 opened Dec 18, 2024 by rogthefrog
@rogthefrog
2
ProTip! Find all open issues with in progress development work with linked:pr.