Virtual Observatory plugin for Jdaviz

[duytnguyendtn]

Description

NOTE: This PR depends on https://github.com/astropy/pyvo/tree/main until it is released with the changes introduced in astropy/pyvo#594 PyVO 1.5.3 has been released and the pin updated accordingly

This PR introduces a new plugin to query resources and load data products published by collaborators of the International Virtual Observatory Alliance. This Plugin currently queries resources implementing the original 1.0 version of the Simple Image Access protocol by specifying a source and a waveband and loads the selected products into Imviz. Though currently only implemented into Imviz, I placed this plugin into the default config in hopes this plugin can be expanded to also work for Specviz via the Simple Spectrum Access Protocol, and Cubeviz as well. Documentation and Unit Tests have been implemented, and this is formally ready to review!

Additionally, I placed the VO plugin closer to the top mainly for my own testing, but also because loading data generally occurs before data analysis. Feel free to adjust the location of the plugin within the plugin bar to your liking.

Thanks for your inputs/help getting us this far! We at the HEASARC hope this is a useful contribution to the Jdaviz tool and community :)

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)? 🐱

[codecov]

Codecov Report

Attention: Patch coverage is 95.36424% with 7 lines in your changes missing coverage. Please review.

Project coverage is 88.94%. Comparing base (6fb47ab) to head (c250c1f).
Report is 9 commits behind head on main.

Files	Patch %	Lines
...s/default/plugins/virtual_observatory/vo_plugin.py	95.30%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2872      +/-   ##
==========================================
+ Coverage   88.91%   88.94%   +0.02%     
==========================================
  Files         111      114       +3     
  Lines       17372    17542     +170     
==========================================
+ Hits        15446    15602     +156     
- Misses       1926     1940      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[duytnguyendtn]

Hi @camipacifici! (Starting to move the conversation to GitHub)

I'm starting to address your review comments, and specifically regarding your request to block data loading unless WCS linking is enabled, I think we're stuck in a catch-22: If there is no data loaded into Imviz, the orientation plugin actually locks the link type to Pixels only:

This requirement currently effectively prevents users from using the VO plugin to load in their first data product (which I think is a usecase we have discussed wanting to support). With this consideration, the only solution I have is to load the data with Pixel linking, then immediately switch it to WCS linking. This does go against your request to block the user with a warning rather than doing it automatically. And if we add this one exception to when there's no data (immediately link WCS if there's no data, but block if there's preexisting data) then that creates inconsistent behavior.

What do you think?

[pllim]

request to block data loading unless WCS linking is enabled

Not quite sure if I follow here. If there is no data or only one data, there is no point in WCS linking (as there is nothing to link).

If there is one or more prior data, then I think it is reasonable for the plugin to automatically toggle WCS linking as long as such a behavior is clearly documented.

But that does not stop user form manually switching back to pixel linking afterwards.

What is the objection of pixel linking anyway?

[duytnguyendtn]

What is the objection of pixel linking anyway?

I don't personally have one, but I'm following through with a request from @camipacifici to block users from loading VO data products unless WCS linking is enabled. Maybe Cami can provide insight to the requirement here?

[camipacifici]

If there is no data at all, I agree that it does not matter. I am thinking more the case where the user has already loaded an image then looks in the VO plugin to find the same field from another telescope. If the image is loaded when linking is by pixel, the user could be confused because the images will not match. Personally, I would not automatically toggle WCS linking, but I would add an orange box that prompts the user to go toggle the linking to WCS.

[camipacifici]

Sorry, I just reread your previous message. I do not want to block the user from loading data. I just want to warn them that they will not see the images aligned if they do not link by WCS.

[duytnguyendtn]

@camipacifici I just pushed up 10d712d which provides a warning when:

1. WCS linking is not enabled and
2. There is pre-existing data in the data collection

On point 2, I figured a warning about misaligned data would be nonsensical if there's only one dataset total (the one the user is loading). Give it a try and see if it's the behavior you're expecting. Thanks!

[camipacifici]

I pulled your latest version here but I do not see the warning. Maybe because I had the plugin already open when I loaded data?

Here is a case that returns a problem:


Any clues? It is also still thinking for some reason, but the rest of the tool is responsive.

[duytnguyendtn]

Hi Cami,

On the smaller points, I pushed up a change that should fix the "hanging" issue. I forcibly stopped the plugin whenever a load data error occurred, but forgot to properly clear the loading flag. Instead, now a load_data error should not stop the plugin; it will notify via snackbar message and then move on to other selected files.

On the lack of linking error message, that one concerns me; let me explicitly spell out what conditions it should hit and you let me know which path you were in, or if you hit a case I'm not covering:

1. If there is NO data loaded, then the VO plugin loads the first data entry into the data collection, no error message should appear, regardless of link type
2. If there is already data in the data collection then the VO plugin loads another data, and pixel linking is on, the warning snackbar should appear
3. If there is already data in the data collection then the VO plugin loads another data, and pixel linking is off, no warning should appear.

I have a test that checks for these conditions in CI, which is passing, so I'm concerned you may have found a scenario I'm not covering for. If you are in one of the above, could you detail what was in your data_collection when you tried to download from the VO, and were expecting the warning?

Lastly, SkyView is down at the moment due to a storage system error so I'll test your other error message once it comes back online. Thanks!

[duytnguyendtn]

Hi Cami!

Now that SkyView is back up I was able to start debugging your aforementioned issue. In short, I did get an error, but it was a download timeout error, not the join error your screenshot has. After I pushed 5a61cc9 which effectively disables the timeout, I was able to successfully download the file you specified in the screenshot. Could you try again? I'm wondering if either you ran into an issue with SkyView when we were figuring out our storage failure, or if you found a platform-specific error. I included a code snippet for reproducibility. Thanks!

vo_plugin = imviz.plugins['Virtual Observatory']._obj
vo_plugin.viewer_selected = "Manual"
vo_plugin.source = "186.70493, 21.83403"
vo_plugin.waveband_selected = "optical"
vo_plugin.vue_query_registry_resources()
vo_plugin.resource_selected = "SDSS"
vo_plugin.vue_query_resource()
vo_plugin.table.selected_rows = [vo_plugin.table.items[0]]
vo_plugin.vue_load_selected_data()

[duytnguyendtn]

Hi @camipacifici! I've implemented basic versions of the two requests you had:

1. 03a00e3 implements the "recalculate center" button for the viewer coordinate select (which only shows when the user is NOT on manual mode)
2. 783262a adds a dropdown to change the units of the search radius (Currently have degrees, radians, arcmins, and arcsecs).

NOTE: I have a pyvo PR open merged but not released to simplify the code needed in Jdaviz to enable (2), so this currently depends on the main branch. Please install main when you try it out.

There are still nice-to-haves I'd like to add when I have a chance, such as detecting when the coordinates pulled are out of date (probably by changing the color of the centering icon), but hopefully this increment gives you enough to try out the behavior. Let me know your thoughts!

[duytnguyendtn]

Hi @camipacifici!

Another quick update. So... I know we were concerned about the performance of auto-updating coordinates during pan-zoom, but I was curious and went ahead and implemented it anyways to see how it would perform. Surprisingly, it does quite well! Any lag I feel doesn't seem to get worse if I enable the "following" switch (alternatively, I don't see a lag improvement when I disable the toggle):

Please try it out and see how it performs on your side! I feel like it massively improves the user experience in searching for coordinates, but if it doesn't work for you, I can always remove it.

Thanks!
Duy

[camipacifici]

Thank you for all the updates! Apparently, my installation was not updating and that's why I could not see the latest improvements. I now see the warning properly, the new choice for the units, and the recentering (which is surprisingly efficient!).

I went one step further and tried to load two images (using the same search of optical-CANDELS). It loaded them fine and I could link them by WCS, but I saw some weirdness when I tried to change the plot options. I now have 3 layers (A, B, C). Layer B is all black and I have not checked if the image is missing or all black in the database or something else is wrong. When I try to edit layer C, the tool keeps trying to edit layer B. I have not checked yet if this problem is also in main. Here are three screenshots:
The name of the image in layer B.

The name of the image in layer C.

The fact that C is not responding to plot options (eye says it is showing, tab says it is not).

[duytnguyendtn]

Hi @rosteen!

Thanks for your patience! I've rebased this PR to get it up to date, and it appears a few things have changed in the meantime:

1. There seems to be a registry querying bug that is present in pyvo release, but fixed in main. When testing, please install dev pyvo for the time being
2. A few tracebacks popped up regarding the default value of some list-based traitlets. I pushed a commit to fix those.

Ok with that out of the way, I did take a look at your review comments!

1. As discussed in your last comment, I added detection for the forced FORMAT violation and fix for it here: efa0aaf. I was able to successfully load data products from SDSSDR9
2. Unfortunately I'm unable to reproduce your "max recursion" bug in your screenshot. I was able to previously, but for some reason after rebasing am no longer able to. My theory is it had to do something with the way messages were being passed around, so I wonder if message handling has changed under the hood somewhere. If you could retry and see if you could reproduce it, and provide more context to the state of the app when you hit it, that would greatly help! The video is starting from a blank session of the ImvizExample notebook. Starting the first x-ray registry query with coverage filtering on or off doesn't seem to change the result for me.

ImvizExample.auto-Y._.2.-.JupyterLab.-.Brave.2025-02-03.15-22-58.mp4

Thanks!