Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

p-values for bootstrapped performance comparison #376

Open
ndiamant opened this issue Jul 29, 2020 · 1 comment
Open

p-values for bootstrapped performance comparison #376

ndiamant opened this issue Jul 29, 2020 · 1 comment
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@ndiamant
Copy link
Contributor

What
plots._protected_subplots makes box plots for model performance per protected class. It should also give a p-value for whether the performance across classes is the same.

Why
What we ultimately want to know is whether the performance is different across classes. You can get an idea of that from the box plots, but it's unclear what conclusion to draw from them without a p-value.

How
Figure out what p-value to calculate then make a helper function to calculate it in plots.py. Currently performance is evaluated in plots._bootstrap_performance and plots._performance_by_index.

Acceptance Criteria
plots._protected_subplots calculates and displays p-values for performance across classes being the same.

@ndiamant ndiamant added enhancement New feature or request good first issue Good for newcomers labels Jul 29, 2020
@lucidtronix
Copy link
Collaborator

Should we do Mann Whitney or Chi squared test here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants