Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 110: Add epiweekly fits and scoring #159

Merged
merged 38 commits into from
Dec 2, 2024
Merged

Conversation

SamuelBrand1
Copy link
Collaborator

@SamuelBrand1 SamuelBrand1 commented Nov 22, 2024

This PR closes #110 .

NB this PR uses forecasttools functions, in particular daily_to_epiweekly. Very handy.

The goal is to compare the epiweekly forecasts of baseline models (which are fitted to epiweekly data) to the epiweekly grouped forecasts from pyrenew_hew.

The daily analysis is unaffected; that is baseline models are still run for daily reporting as well.

Tasks to contribute

  • Add generate epiweekly truth data from daily truth data step
  • Add run baseline models on epiweekly
  • Add extra pyrenew-hew postprocess to chunk the daily posterior preds into epiweekly
  • check how much revision the scoring function needs (might be zero)

Testing

At each step the test mode pipeline is run to check for failure or breaking changes.

@SamuelBrand1
Copy link
Collaborator Author

Added a step to convert daily resolved data to epiweekly cf generate_epiweekly.R.

@SamuelBrand1
Copy link
Collaborator Author

Modified timeseries_forecasts.R to run baseline forecasters for both for daily and epiweekly models.

@SamuelBrand1
Copy link
Collaborator Author

Added scoring on epiweekly scale for pyrenew. The algorithm is that if epiweekly = TRUE then the pyrenew forecasts for counts and other counts are group summed by epiweek (which auto does by draw), pivoted out to create the correct weekly percentage (which is not the mean of the daily percentages), and then pivoted longer.

Dates for the week are supplied by more forecasttools helpers.

No other changes were need because the scoring is flexible enough to ingest this.

@SamuelBrand1 SamuelBrand1 marked this pull request as ready for review November 22, 2024 13:15
@SamuelBrand1
Copy link
Collaborator Author

This is ready for review now.

Note that the plotting is unchanged and the ultimate scoring end product is both:

  • score_table.rds
  • epiweekly_score_table.rds

The collate functions seem to be currently broken, but I'm interpreting that as a new issue.

@SamuelBrand1
Copy link
Collaborator Author

Also note that this PR has inherited the failing CI from #160

@SamuelBrand1 SamuelBrand1 mentioned this pull request Nov 22, 2024
5 tasks
@damonbayer
Copy link
Collaborator

damonbayer commented Nov 22, 2024

@SamuelBrand1 Since we are making even more model results and anticipate adding PyRenew-EW to the workflow soon, I would really like to push toward #101. Could the fable-related parts of that issue be accomplished in this PR (i.e. let's store all the results from these time series models in a fable directory)?

Copy link
Collaborator

@damonbayer damonbayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing that would necessarily prevent a merge, but I had lots of comments and questions.

@SamuelBrand1
Copy link
Collaborator Author

I've closed the comments marked Outdated and we are onto our last few.

I think @damonbayer remains the reviewer here because this PR is a mix of me and @dylanhmorris

Copy link
Collaborator

@damonbayer damonbayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dylanhmorris This PR has my tentative approval. Please respond to (or make changes based on) the remaining comments before merging.

@dylanhmorris dylanhmorris merged commit c4ce53f into main Dec 2, 2024
7 of 8 checks passed
@dylanhmorris dylanhmorris deleted the epiweekly-fits branch December 2, 2024 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Scoring on (epi)weekly scale
3 participants