Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scoring on (epi)weekly scale #110

Closed
5 tasks done
SamuelBrand1 opened this issue Nov 8, 2024 · 4 comments · Fixed by #159
Closed
5 tasks done

Scoring on (epi)weekly scale #110

SamuelBrand1 opened this issue Nov 8, 2024 · 4 comments · Fixed by #159

Comments

@SamuelBrand1
Copy link
Collaborator

SamuelBrand1 commented Nov 8, 2024

At the moment scoring (on the scoring branch) is done daily, and averaged over a week.

This could be useful to keep this, but noting that this includes scoring within-week effects, but we should also have weekly aggregated then scored utility.

Sub-task planning

  • Land Issue 105: Generate test forecasts and scoring output #148
  • Add generate epiweekly truth data from daily truth data step
  • Add run baseline models on epiweekly
  • Add extra pyrenew-hew postprocess to chunk the daily posterior preds into epiweekly
  • check how much revision the scoring function needs (might be zero)
@SamuelBrand1
Copy link
Collaborator Author

Connected to #79

@SamuelBrand1
Copy link
Collaborator Author

A wrinkle here is that there are more than one ways to add to the code base in order to do this:

  1. Add functionality so that all forecast models that currently forecast daily scale have an optional step to collate into weekly scale. The downside here is that the cdc baseline forecaster needs to be averaged (only the % forecasts are available which can't be used to calculate the actual weekly % due to missing actual denominator).
  2. Add functionality so that the pyrenew forecaster has weekly binning, and that the forecasts of the other models are based on ingesting only weekly data.

The main difference between 1/2 is that the baseline models are then aimed at being optimised for scoring on weekly scale. The belief is that the semi-mechanistic pyrenew model will forecast best based on daily inference, even if the target is weekly. This seems reasonable, and the daily scale of dynamics is important for the modelling, but is currently untested.

@SamuelBrand1
Copy link
Collaborator Author

Blocked by #148 , but when resolved this will be in a good place to resolve.

@SamuelBrand1
Copy link
Collaborator Author

#148 got done so this is ready for closing by #159 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant