Ewm7121 refactor art norm #487

darshdinger · 2024-10-28T20:50:04Z

Description of work

The purpose for this PR is the implement the previously created artificial normalization within the Reduction workflow. The changes ensure the workflow can handle missing normalization data by applying artificial normalization and introduce enhancements to the user interface, logging, and data handling processes.

Explanation of work

Key updates include:

Artificial Normalization:

Added the ArtificialNormalizationRecipe to GenericRecipe that uses the CreateArtificialNormalizationAlgo to handle cases when normalization data is missing.
Automatically extracts parameters and queues the algorithm to create an artificial normalization workspace when needed.

Workflow Updates:

The Reduction workflow can now dynamically adjust for missing calibration or normalization data, applying artificial normalization when necessary.
Updated the UI to prompt users for artificial normalization inputs and seamlessly continue the workflow.

Error Handling and Logging:

Improved logging and error messages to guide users through situations where data is missing.
Enhanced exception handling to gracefully manage missing states or permissions.

Backend Improvements:

Refactored the ReductionService to handle normalization and calibration more flexibly.
Expanded unit tests to cover new cases with artificial normalization.

To test

Insure all pytests pass. Please launch SNAPRed UI with a fresh (no states initialized) local copy of Calibration_next specified within application.yml. First run the Diffraction Calibration with the following inputs:

Run Number: 58882
Convergence Threshold: 0.1
Bins Across Peak Width: 10
Sample: Silicon_NIST_640D_001.json
Grouping File: Column

Create a new state when prompted. Then continue on through workflow through successful completion.
Now moving onto the Reduction tab, notice there is a new tab called "Artificial Normalization". Press the Continue button and notice the UI handling for no inputs:

Finally, without running a normalization, run the Reduction workflow with the following inputs:

Run Number: 58882

Continue the workflow to notice the following message pop-up:

Continue Past the possible warning for permissions check but clicking Yes. Find that the workflow completes successfully.

Finally, re-run this but with a normalization this time and see that the Artificial Normalization flow is not used in this case.

Dev testing

Insure all pytests pass

CIS testing

See above.

Link to EWM item

EWM # 7123

&

EWM # 7121

Verification

the author has read the EWM story and acceptance critera
the reviewer has read the EWM story and acceptance criteria
the reviewer certifies the acceptance criteria below reflect the criteria in EWM

Acceptance Criteria

This list is for ease of reference, and does not replace reading the EWM story as part of the review. Verify this list matches the EWM story before reviewing.

1) The user can input variables within an "artificial normalisation" step following a UI prompt informing the user that an artificial normalization is needed.
2) The backend can calculate the arficifial normalisation function
3) The backend applied the artificial normalisation function
4) The output reduced data are available for all pixel groups (as per normal)
5) The output data are labelled as having had artificial normalisation applied.

codecov · 2024-10-28T20:52:19Z

Codecov Report

Attention: Patch coverage is 95.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 96.35%. Comparing base (09342de) to head (53d2f01).
Report is 1 commits behind head on next.

Files with missing lines	Patch %	Lines
src/snapred/backend/service/ReductionService.py	94.59%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             next     #487       +/-   ##
===========================================
+ Coverage   55.65%   96.35%   +40.69%     
===========================================
  Files          64       64               
  Lines        4713     4744       +31     
===========================================
+ Hits         2623     4571     +1948     
+ Misses       2090      173     -1917

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/snapred/ui/workflow/ReductionWorkflow.py

walshmm · 2024-10-30T14:27:45Z

When attempting to proceed with neither normalization nor calibration and an uninitialized state I get an error

though this error I think gets fixed in my other pr https://github.com/neutrons/SNAPRed/pull/469/files#diff-8f698031b2852fad83d059775e7f13b373b64302000cb3f5dc92d1556c0f6ed0R90-R106
so I will init state elsewhere and proceed with further testing

walshmm · 2024-10-30T14:34:20Z

Attempting to proceed with an a freshly initialized state nets the following error

this is the current state of my data folder

I am able to load default calibration with my pr here https://github.com/neutrons/SNAPRed/pull/486/files#diff-494c3d2dbf59e6e214cb384bd29120e6bc67085459feb33eff42d2e32caeb3dcR169-R175
so I am unsure why this fails.

I will perform a calibration and proceed again, but the above needs fixing

walshmm · 2024-10-30T14:37:29Z

I forgot to note in my last comment that I did get an error message specifying that diffcal and norm was missing but it said the data would not be normalized, is that correct? I though we would apply an artificial norm.

walshmm · 2024-10-30T14:41:48Z

Even with diffcal created it seems to be erroring out?

Why are you specifying column here? https://github.com/neutrons/SNAPRed/pull/487/files#diff-8f698031b2852fad83d059775e7f13b373b64302000cb3f5dc92d1556c0f6ed0R484-R485
I tried a grouping of All and its failing

walshmm · 2024-10-30T14:46:12Z

recalibrating specifically with a column grouped calibration allows me to proceed

However these true false fields are unlabeled, please give them some.

Reduction then proceeds and completes successfully

walshmm · 2024-10-30T14:55:47Z

Reduction with a proper calibration and normalization proceeds and completes

Successfully skipping artificial norm and informing the user that it is not necessary.

walshmm · 2024-10-30T18:07:30Z

src/snapred/backend/service/ReductionService.py

+        calVersion = None
+        calVersion = self.dataFactoryService.getThisOrLatestCalibrationVersion(request.runNumber, request.useLiteMode)
+        calRecord = self.dataFactoryService.getCalibrationRecord(request.runNumber, request.useLiteMode, calVersion)
+        filePath = self.dataFactoryService.getCalibrationDataPath(request.runNumber, request.useLiteMode, calVersion)
+        diffCalOutput = calRecord.workspaces[wngt.DIFFCAL_OUTPUT][0]
+        diffcalOutputFilePath = str(filePath) + "/" + str(diffCalOutput) + ".nxs.h5"
+
+        groceries = self.groceryService.fetchWorkspace(diffcalOutputFilePath, "diffractionWorkspace")


Could you move this logic to a method on groceryService instead? I can imagine it would be very useful elsewhere to be able to read a calibration's run data with a simple (runnumber, litemode, calversion).

It would also be more in line with the rest of the architecture.

walshmm · 2024-10-30T18:17:33Z

fresh state displays this:

default state lacks data:

walshmm · 2024-10-30T18:24:49Z

All Grouping Calibration works now

true false fields are also labeled now

walshmm · 2024-10-30T19:34:19Z

proceeding anyway despite there being no calibration now nets an error explaining that this feature has yet to be implemented yet due to lack peaks available from the default calibration.

walshmm · 2024-10-30T19:47:50Z

1) The user can input variables within an "artificial normalisation" step following a UI prompt informing the user that an artificial normalization is needed.
2) The backend can calculate the arficifial normalisation function
3) The backend applied the artificial normalisation function
I think that looks right.
4) The output reduced data are available for all pixel groups (as per normal)
5) The output data are labelled as having had artificial normalisation applied.
this is actually taken care of in a different pr

darshdinger added 7 commits October 25, 2024 11:20

Refactor of previous implementation of ArtificialNormalization

c5f8986

Some more updates

7e2fa88

Whoops

fec2e3c

Fixes within workflow view

e5f7fae

Final fixes to the workflow.

597ebb4

Update tests

240e00d

Merge branch 'next' into ewm7121-refactor-art-norm

057fdce

Code cov

dc643ad

darshdinger marked this pull request as ready for review October 28, 2024 21:24

Updates to warning messages in ReductionService.py for Defect 7756

96ffb6a

walshmm reviewed Oct 29, 2024

View reviewed changes

src/snapred/ui/workflow/ReductionWorkflow.py Outdated Show resolved Hide resolved

walshmm reviewed Oct 29, 2024

View reviewed changes

src/snapred/ui/workflow/ReductionWorkflow.py Outdated Show resolved Hide resolved

darshdinger added 2 commits October 29, 2024 14:22

Merge branch 'next' into ewm7121-refactor-art-norm

3b2714c

Updates based on Michael's comments

19b1437

More updates based on comments

302de0b

walshmm reviewed Oct 30, 2024

View reviewed changes

Last updates.

53d2f01

darshdinger enabled auto-merge (squash) October 30, 2024 19:38

walshmm approved these changes Oct 30, 2024

View reviewed changes

darshdinger merged commit 66af7ef into next Oct 30, 2024
8 checks passed

darshdinger deleted the ewm7121-refactor-art-norm branch October 30, 2024 19:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ewm7121 refactor art norm #487

Ewm7121 refactor art norm #487

darshdinger commented Oct 28, 2024 •

edited by walshmm

Loading

codecov bot commented Oct 28, 2024 •

edited

Loading

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024 •

edited

Loading

Ewm7121 refactor art norm #487

Ewm7121 refactor art norm #487

Conversation

darshdinger commented Oct 28, 2024 • edited by walshmm Loading

Description of work

Explanation of work

To test

Dev testing

CIS testing

Link to EWM item

Verification

Acceptance Criteria

codecov bot commented Oct 28, 2024 • edited Loading

Codecov Report

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm Oct 30, 2024

Choose a reason for hiding this comment

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024

walshmm commented Oct 30, 2024 • edited Loading

darshdinger commented Oct 28, 2024 •

edited by walshmm

Loading

codecov bot commented Oct 28, 2024 •

edited

Loading

walshmm commented Oct 30, 2024 •

edited

Loading