Make updater smarter about remote and local list of update sites

[carandraug]

As mentioned on the discussion to issue #65 the updater does not update the sites URLs when they change on the wiki list.

[frauzufall]

I want to fix this since we need it for the HTTPS support. Here's how it could look like:

1. load local list of update sites
2. load remote list of update sites
3. pair up by name
4. if URLs not equal, use remote URL

However this would mean that if someone modifies the URLs manually the updater would always reset them again. I would therefore add an attribute to an update site storing the manual choice and not update the URL to the remote URL in this case.

This would only work for future manual changes, anyone who edited these URLs in the past would get the default URLs from the list of update sites back.

[ctrueden]

I think a scheme like this makes sense. There are some other related edge cases to consider:

• Someone adds an update site "Foo" to the list manually, because it is not included on the central list of update sites. Later, the Foo site is added to the list of update sites. In this situation, we want Foo from the remote to take over for all users who had added Foo manually before.
• Someone has "Baz" enabled from the list of update sites. Later, Baz becomes obsolete, merging with another site, so Baz is removed from the list of update sites. In this situation, we may want to either: A) remove Baz from the list for them; or B) warn the user that Baz is no longer listed, and they may want to consider removing it from their list manually.

[carandraug]

• Someone adds an update site "Foo" to the list manually, because it is not included on the central list of update sites. Later, the Foo site is added to the list of update sites. In this situation, we want Foo from the remote to take over for all users who had added Foo manually before.

I'm not sure this is good. The manually added "Foo" site may not be the "Foo" site that is later added to the central/remote list. Say I have a "micron" update site for internal use which our users and facility systems use. Someone else sets a "micron" site on the central/remote list. This buggers up all our systems and our users on the next upgrade.

The name of the site should be for convenience and readability, but what really matters is the actual URL. That this changes automatically under my feet sounds worrying. It's ok for the server in the URL to then redirect because by selecting that URL, the user claims to trust the server. This is different from having something else (the ImageJ list of update sites), changing the URL itself.

So I don't think this is a requirement to support https. I also don't think that updating the list of sites in a central list of sites is the solution when one update sites moves location. The responsibility of this falls on the update site maintainer which means setting up a redirect. This redirect may even be Moved Permanently, in which case I guess it may make sense to update the URL.

[ctrueden]

The name of the site should be for convenience and readability, but what really matters is the actual URL.

I agree. To clarify my point, here are two scenarios and how I would propose handling them:

Scenario 1

• You create a Micron update site at https://myfacility.edu/micron, instructing your users to add this update site manually to their list of update sites locally.
• Some other organization adds a Micron update site at https://theirinstitute.org/micron to the list of update sites.
• The Updater is smart enough to realize that these two Micron sites are not the same thing, because the URLs differ. The manually added Micron update sites are treated as "local only", with the other Micron update site also being listed independently.

Scenario 2

• Some organization creates a Micron update site at https://theinstitute.org/micron, at first for internal use only, and instructs their users to enable it manually locally for now.
• Once bugs are ironed out, the organization adds this same Micron update site to the list of update sites.
• The Updater is smart enough to realize that these two Micron sites are the same thing, because the URLs match, so all the manually added Micron entries are not treated as "locally modified"/"local only" anymore.

How to achieve it

When a site is added locally (as opposed to being edited from an entry originally added via the list of update sites), it gets a special status "local only". Whether it retains that status after a new list of update sites is retrieved and compared will depend on the situation, as described above.

I do not think it is trivial to consider and code all these various different edge cases, but I think it is helpful if we can make things work as seamlessly as possible over time as the list evolves.

@frauzufall Perhaps it makes sense to assign each site on the list a unique ID, and store those locally, so that it becomes unambiguous whether a particular local site is really the same site linked to the list of sites.

[frauzufall]

@carandraug I understand your concerns. I was not happy about matching the update sites by name, I just wanted to make sure any installation supporting HTTPS will also use HTTPS without users having to manually update their URLs. I will look into your PR #67 next.
@ctrueden having an ID sounds very reasonable.

[frauzufall]

As I commented here, we do need to update these URLs in order to serve HTTPS updates to existing installations because the ImageJ server has no redirect from HTTP to HTTPS for requests from the updater (@ctrueden correct my if I'm wrong). I would love to get this going, any suggestions welcome. We could also whitelist links from @carandraug's server in the code so that they will not be affected.

[frauzufall]

We whitelisted https://downloads.micron.ox.ac.uk here and all other URLs should now be updated according to the list of available update sites. There initial issue is therefore fixed. Since there were scenarios pointed out in this issue by @ctrueden where the updater should be smarter about how to handle official and personal update sites, I'll just rename this issue.

[ctrueden]

the ImageJ server has no redirect from HTTP to HTTPS for requests from the updater

Yes, intentionally so. For most user agents, we do now blanket redirect from HTTP to HTTPS. But if the user agent is the ImageJ Updater, we do not. This is so that old Fijis requiring HTTP will still function, rather than spewing SSLHandshakeException.

One thing we could do is change the user agent string of the new Updater to be distinct from that of the old Updater—either all the time, or maybe even only when the Java version is new enough. Then we could have Apache behave differently for the two, doing an HTTPS redirect only if it would work. But I don't know if this would be helpful, or just overly complex.