Ewm3143 apply diffcal during norm workflow #469

walshmm · 2024-10-09T20:11:40Z

Description of work

This supplies the features requested in ewm3143:

The application of a diffraction calibration to the normaliztion data in the Normalization Workflow for user assessment
The logging of metadata on the workspace if it uses the default calibration in the Normalization Workflow

This also supplies features in the neighborhood of what it asked for:

The logging of metadata on the workspaces if diffraction calibration/normalization is missing in the Reduction Workflow

I have also refactored:

StrEnum to live in a seperate Enum module.
WorkspaceMetadata now uses StrEnum instead of literals.
Recipes now can supply an init method for their given Ingredients.

To test

NOTE: Use the View Sample Logs option when right clicking on a workspace to confirm it has the snapred metadata.

For reduction
Runnumber: 59039 

for calib:
sample: LA11b6
Grouping: Column
Peak Intensity Thresh: 0.01
Chi: 200
dmin: 1

for norm:
rnunumber: 58810
bgrunnumber: 58813

Attempt to run either [Normalization Worflow, Reduction Workflow] -> [without a state, with a default state, with a valid state]

Normalization

without a state

should return a runtime error if you lack permission to init state
should ask you to init state if you have write permission

with a default state

Should prompt with a continue anyway informing the user that the output will be marked as potentially inaccurate by means of it using a default state.

with a valid state

Continue as normal

Reduction

As it is in next except:

it actually loads the diffcal mask
the outputs should have appropriately marked metadata

Link to EWM item

EWM#3143
EWM#7758

Verification

the author has read the EWM story and acceptance critera (Malcolm has additionally review and asked me to add AC)
the reviewer has read the EWM story and acceptance criteria
the reviewer certifies the acceptance criteria below reflect the criteria in EWM

Acceptance Criteria

This list is for ease of reference, and does not replace reading the EWM story as part of the review. Verify this list matches the EWM story before reviewing.

~~Unfortunately there is not an explicity Acceptance Criteria outlined in the story, what I can infer from it are as follows~~
Malcolm has since reviewed the AC and made clarifications.

in the normalisation workflow, the raw vanadium data shall be treated exactly the same as the sample data being Reduced for examination.
in the normalisation workflow, If there is only a default diffcal (This workflow requires at least an initialized state), there should be a warning that the output will be labeled as using such default diffcal and proceed anyway prompt.
The outputs should have a log on them denoting the existence/status of diffcal/normalization.
in the normalisation workflow, the raw vanadium data shall be treated exactly the same as the sample data being Reduced for examination.
the difcal index is queried to find available difcal for the normalization run
if found the diffcal is applied to the raw
Any corresponding diffcal mask is also applied to the raw vanadium

7758

The correct behaviour would be to warn the user that the state does not exist, prompt user to create the state (probably with a notice that only an instrument scientist can actually do this)
if state is initialized as above and the user elects to proceed with default calibration, it should not fail trying to load such calibration

codecov · 2024-10-10T18:34:30Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.48%. Comparing base (09342de) to head (9a466fd).
Report is 2 commits behind head on next.

Additional details and impacted files

@@             Coverage Diff             @@
##             next     #469       +/-   ##
===========================================
+ Coverage   55.65%   96.48%   +40.82%     
===========================================
  Files          64       65        +1     
  Lines        4713     4832      +119     
===========================================
+ Hits         2623     4662     +2039     
+ Misses       2090      170     -1920

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rboston628

I notice a lot of places flagged by codecov needing test coverage

src/snapred/backend/dao/WorkspaceMetadata.py

src/snapred/backend/data/LocalDataService.py

rboston628 · 2024-10-28T20:51:19Z

src/snapred/backend/data/LocalDataService.py

+                raise RecoverableException.stateUninitialized(runId, useLiteMode)
+            else:
+                raise RuntimeError(
+                    "No calibration exists, and you lack permissions to create one, please contact your CIS."


Comma splice

src/snapred/backend/recipe/PreprocessReductionRecipe.py

walshmm · 2024-10-29T18:19:36Z

src/snapred/backend/data/LocalDataService.py

+                raise RecoverableException.stateUninitialized(runId, useLiteMode)
+            else:
+                raise RuntimeError(
+                    "No calibration exists, and you lack permissions to create one." " Please contact your CIS."


ruff seems to think this should be one line so it shall be

if you put a # it could silence it

walshmm · 2024-10-29T18:23:15Z

I notice a lot of places flagged by codecov needing test coverage

not seeing those now so I'm not sure what that was about

rboston628 · 2024-10-30T20:28:48Z

Start running normalization with no states. I am prompted to init

It inits successfully, meaning there is a default.

Start running normalization with default state. I am prompted as such:

I create a valid calibration file then run normalization. There is no prompt. It continues normally.

I run reduction. I can see the SNAPRed logs.

I'm not currently seeing a way to verify the pixel masks.

…as intended just a couple line breaks fix tests up coverage fix broken test mark workspaces with metadata if no calibration/normalization is present up test coverage up timeout so integration test doesnt fail? actually account for closing the new QMessageBox.warning from new alt flow normalization workflow should initialize state and then warn the user its using default diffcal revert integration_test.yml fix integration tests, update state init and perms check for reduction up test coverage remove completed TODO fix incorrectly scoped patch merge and fix integration tests the right wayTM fix broken unit test update according to pr comments tell ruff to stop formmating that line

rboston628

👍

walshmm force-pushed the ewm3143_apply_diffcal_during_norm_workflow branch from 7683229 to 90a64d5 Compare October 10, 2024 18:31

walshmm marked this pull request as ready for review October 22, 2024 19:58

walshmm force-pushed the ewm3143_apply_diffcal_during_norm_workflow branch from 0a0eccb to 4e2852e Compare October 25, 2024 20:15

rboston628 reviewed Oct 29, 2024

View reviewed changes

walshmm commented Oct 29, 2024

View reviewed changes

rboston628 previously approved these changes Oct 30, 2024

View reviewed changes

walshmm force-pushed the ewm3143_apply_diffcal_during_norm_workflow branch from 72879d3 to 19d62bb Compare October 31, 2024 13:17

force it to call the validate method

9a466fd

walshmm dismissed rboston628’s stale review via 9a466fd October 31, 2024 13:52

rboston628 approved these changes Oct 31, 2024

View reviewed changes

walshmm merged commit 8398c8e into next Oct 31, 2024
8 checks passed

walshmm deleted the ewm3143_apply_diffcal_during_norm_workflow branch October 31, 2024 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ewm3143 apply diffcal during norm workflow #469

Ewm3143 apply diffcal during norm workflow #469

walshmm commented Oct 9, 2024 •

edited by rboston628

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading

rboston628 left a comment

rboston628 Oct 28, 2024

walshmm Oct 29, 2024

rboston628 Oct 30, 2024

walshmm commented Oct 29, 2024

rboston628 commented Oct 30, 2024

rboston628 left a comment

Ewm3143 apply diffcal during norm workflow #469

Ewm3143 apply diffcal during norm workflow #469

Conversation

walshmm commented Oct 9, 2024 • edited by rboston628 Loading

Description of work

To test

Normalization

without a state

with a default state

with a valid state

Reduction

Link to EWM item

Verification

Acceptance Criteria

codecov bot commented Oct 10, 2024 • edited Loading

Codecov Report

rboston628 left a comment

Choose a reason for hiding this comment

rboston628 Oct 28, 2024

Choose a reason for hiding this comment

walshmm Oct 29, 2024

Choose a reason for hiding this comment

rboston628 Oct 30, 2024

Choose a reason for hiding this comment

walshmm commented Oct 29, 2024

rboston628 commented Oct 30, 2024

rboston628 left a comment

Choose a reason for hiding this comment

walshmm commented Oct 9, 2024 •

edited by rboston628

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading