-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Echelle improvements #54
Conversation
maven_iuvs/fits_processing.py
Outdated
|
||
Parameters | ||
---------- | ||
file_index : index file (.npy) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the loaded index file is an array, not an .npy filename, so this might be confusing... I guess we could define a new datatype, but let's not if we can get away with it.
Thoughts about keeping the index generation routines in the echelle.py module for now, since we only plan to use this for echelle data? |
We could, I just figured it made sense to agnosticize it now. Would you prefer it that way? |
Right now I think that the index generation routines are agnostic to the contents of the directories, but the echelle routines that look at the index to generate the reports and file pairing don't have any protections to make sure that the files in the index are only echelle files. Until we add protections in one or both places I think the safer option is to keep all of the index generation stuff in the echelle submodule. |
For now I just moved those functions back to the echelle file. What do you think? |
I think to be consistent all of the functions that are specific to echelle and the file index need to be in echelle.py . Same for functions like get_countrate_diagnostics, which is really only useful for the echelle with the pixel ranges specified. I think the geometry functions and the average pixel countrate functions might belong in geometry.py and statistics.py / integration.py ? Some of the binning stuff could go into a new binning.py file? Or does another organization scheme make more sense to you? |
Yeah, I've been having difficulty deciding how to organize since a lot of the other functions are older, the module names very general ("instrument.py"), or the functions are specific to certain data levels (e.g. l1b). As for the index, at this point the update_index function doesn't care what's in the folder, which doesn't seem to me to be directly tied to echelle, it's more of a folder/fits parsing function. If you insist it be part of echelle, I will put it there though. get_countrate_diagnostics (and related functions) should be in there though, those I forgot about. I'm also missing some docstring info actually, so I will add that. Overall I thought it would be reasonable to upload these new functions now and then make a more concerted effort to organize next. My instinct would be to have l1a functions and l1b functions separate, but that will probably lead to duplicated code, so the right answer is probably to sort by topics ("binning", "line fitting", "geometry" etc) and then divide those files into sections if there need to be separate functions for l1b/l1a. But I can also organize them first instead. |
Yeah, probably I am too in the weeds re:organization. There's a bunch of
stuff that can be moved / deleted in other files, we should try to find an
hour to go through and sort them all out.
I agree that the indexing routines can stay in the generic file, but since
it theoretically calls get_countrate_diagnostics(), which is specific to
echelle, there should at least be some error trapping added before merging.
Adding a simple is_echelle() test and raising an error if countrates is
called on a non-echelle file is probably all that's needed at this point.
…On Fri, Jun 9, 2023, 10:12 AM Eryn Cangi ***@***.***> wrote:
Yeah, I've been having difficulty deciding how to organize since a lot of
the other functions are older, the module names very general
("instrument.py"), or the functions are specific to certain data levels
(e.g. l1b).
As for the index, at this point the update_index function doesn't care
what's in the folder, which doesn't seem to me to be directly tied to
echelle, it's more of a folder/fits parsing function. If you insist it be
part of echelle, I will put it there though. get_countrate_diagnostics
should be in there though, that one I forgot about.
Overall I thought it would be reasonable to upload these new functions now
and then make a more concerted effort to organize next. My instinct would
be to have l1a functions and l1b functions separate, but that will probably
lead to duplicated code, so the right answer is probably to sort by topics
("binning", "line fitting", "geometry" etc) and then divide those files
into sections if there need to be separate functions for l1b/l1a. But I can
also organize them first instead.
—
Reply to this email directly, view it on GitHub
<#54 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADIZCKRRH3FA4VKMYHWTRY3XKNDQNANCNFSM6AAAAAAY7PWZSI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hmmm, yeah, let's look at it next week. |
… local destination for syncing files will cause rsync to silently fail, as well as related troubleshooting print statements (may be removed if desired).
…e improvements to classification of files as darks.
…ble names. Also created get_pix_range as its own function, as its functionality was needed in more than one place while developing echelle quicklooks.
…eate geometry-related plots.
…w also has a variety of extra arguments to allow greater control of plot attributes.
Any reason you prefer alphabetical order? A different order would probably be more readable, but alphabetical is probably better than what we have now. https://stackoverflow.com/questions/10289461/what-is-a-good-way-to-order-methods-in-a-python-class |
Mostly because I don't know what other order would be logical, and through use of the routines, we become familiar with them and remember names of ones we use, so we can find them quickly. Open to other ordering ideas though! Could also maybe include an index comment string at the top, but I don't know offhand if that's discourged in PEP8 or 257 or whatever they call the style now. |
The link I shared has some decent suggestions; put the most important stuff first, followed by helper methods; group related routines by subject matter. We don’t need to be strict about this but alphabetical seems cumbersome to maintain… |
Oh thanks, sorry I missed it due to distractions. will have a look later.
Sent from Proton Mail mobile
…-------- Original Message --------
On Aug 8, 2023, 6:38 PM, Mike Chaffin wrote:
The link I shared has some decent suggestions; put the most important stuff first, followed by helper methods; group related routines by subject matter. We don’t need to be strict about this but alphabetical seems cumbersome to maintain…
—
Reply to this email directly, [view it on GitHub](#54 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABB36LVTXGJB23DGV3QALDLXULLZJANCNFSM6AAAAAAY7PWZSI).
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
…he slit on the detector image at Majd's request.
…lle plots to fail to draw.
…s former was throwing errors.
…it or date; encapsulated several functions relating to selecting files for which to run quicklooks (downselect_data), dark frames (get_dark_frames, subtract_darks), and coadding light frames (coadd_lights). Corrected an error in get_file_metadata's datetime field. Added some information back to the quicklook file per John Clarke's recommendations.
…top), added a bit of logic to allow for use with l1a or l1b files.
…utine - Move get_grad_colors to graphics.py - Move update_index routine from fits_processing.py to echelle.py - Move binning-related functions from fits_processing.py to new binning.py - Create a routine in binning.py to get the number of pixels per bin for given fits file - Delete fits_processing.py; move contents to binning.py (new) or miscellaneous.py - Add a routine to obtain data product level from filename - Update imported modules in miscellaneous.py and __init__.py according to changes.
…eometry routines to echelle.py. Remove unused imports in geometry.py and add imported modules to support change to graphics.py. Echelle module changes handled in later commit due to extensive changes to that module. Add idl_colorbars import to support geometry plotting routines.
…own hosts; delete commented-out code previously used for troubleshooting.
…gly, rather than using try/except block.
…r l1b data and l1a echelle data. - Restore original functionality of detector_image - Create second detector_image_echelle to support echelle quicklook creation - Move plot code previously in detector_image to a lower-level routine, plot_detector, called by detector_image and detector_image_echelle - Implement **kwargs for detector_image, detector_image_echelle - Replace nan_color variable (previously deleted) with existing gray color - Remove echelle-specific plotting choices, to be re-implemented in higher level echelle routines - Delete imports no longer needed here as a result of these changes
…ion of separate echelle_graphics.py module. - Move quicklook code to new module, echelle_graphics.py - Modify coadd_lights to include necessary routines to perform dark subtraction and divide by total frames here rather than in the plot making code - Update subtract_darks to maintain the same number of frames as light observation, but set nans if frames are bad - Remove code handling orbit 9070 that was ported directly from IDL but not actually incorporated - Improve get_dark_frames to handle case where both darks failed, and where an average is requested - Move echelle slit start/end pixels to instrument.py - Update module imports in echelle.py to account for above changes
…fleshing out docstrings for functions previously missing docstrings/accurate docstrings, removing old code that was commented out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great--- I added some small comments, let me know what you think. Hoping code changes will be significantly less onerous in the future
maven_iuvs/echelle.py
Outdated
""" | ||
|
||
# Retrieve dark frames for subtraction | ||
first_dark, second_dark = get_dark_frames(dark_fits) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think assuming that there is always a first and second dark will cause problems when processing the very earliest mission data, but we can deal with that later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now it throws an error if there's only 1, returns the 2 if there are 2, and returns 2 with the second being an average of 2:end if there's > 2. I can change the unpacking to a tuple at a later point though
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Started on this with bcddfdf
@@ -29,7 +29,7 @@ def __init__(self, filename): | |||
self.__basename = _os.path.basename(filename) | |||
self.__check_input_is_iuvs_data_filename() | |||
hdulist = HDUList.fromfile(filename, mode='readonly', | |||
memmap=True, save_backup=False, | |||
memmap=False, save_backup=False, # Note: Changed memmap to False because having it True precluded making quicklooks. --EMC 8/10/23 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy to change all the loading routines to use fits.open() instead; IUVSFITS adds a lot of overhead processing time.
… dark_subtracted instead of coadd_lights.
…ha to account for echelle resolution.
…int lat/lon, spacecraft altitude)
…er adaptation to earlier in the mission when only 1 dark may have been present. Adjustments for the case of only one dark will be added later.
Move some non-top-level imports of astropy.io.fits to the top in integration.py; Replace code in echelle.py and echelle_graphics.py that was using IUVSFITS class
. Add import to echelle.py to support b72c88c
…with new function that uses regular expressions.
still working on this? happy to review now or later |
I think I've addressed every comment, but it's hard to tell because of the way it displays. Take another final look? |
LGTM, thanks for persevering through these changes! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Pull Request Checklist
Code Follows PEP8 guidelines (explain any exceptions)
Some lines are over length to avoid breaking in awkward positions
New routines don't duplicate existing functionality
Original authors of modified routines have been added as reviewers