OAK-11232 - indexing-job - Simplify download from Mongo logic by traversing only by _modified instead of (_modified, _id)

[nfsantos]

The Mongo downloader traverses the repository by order of the fields (_modified, _id). In case of disconnection from Mongo, this allows resuming the download from where it was interrupted without redownloading any document.

However, the downloader does not need to ensure that no duplicate documents are downloaded, because the merge-sort stage of the Pipelined strategy discards duplicates. Avoiding duplicates in case of reconnections is only a performance optimization for a relatively rare occurrence.

This PR simplifies the downloader by traversing only by _modified. In case of failure, the download resumes from the last value of _modified seen. So if the last _modified value seen was 1000, the downloader will again download the documents with _modified=1000 that had previously downloaded. But this would likely take just a few seconds even in the worst case scenario. _modified has a resolution of 5 seconds, so the number of documents with the same value is limited by how much Oak can write to Mongo in a 5 seconds window. The downloader is streaming the results directly using a Mongo query, which is much faster than what Oak can write. So likely, the downloader will download all values with the same _modified value in a fraction of the time it took Oak to write them, which is an acceptable overhead in the rare case of disconnection from Mongo.

This change greatly simplifies the logic of the downloader:

• No need to track both the _modified and _id fields. The reconnection logic becomes simpler. Before, on reconnection the downloader had to first do a query to finish downloading documents with the previously seen _modified (_modified=modified_last_seen, _id>id_last_seen) and then another query with _modified>modified_last_seen. Now it is enough to query for _modified>=modified_last_seen.
• Slight speed gain from not having to deserialize the _id field in the downloader threads.
• Potentially faster query execution on Mongo because the traverse and sort conditions are simpler: instead of traversing and ordering by (_modified, _id), it now only does it by _modified. I have not observed a significant speed-up on Mongo traversal speed, but in theory the Mongo optimizer has more freedom to optimize the new simpler query.

Other changes in the PR:

• Do not use Thread interrupt to cancel download threads, instead close only the Mongo connection. Thread interrupts are very problematic
• Reduce the frequency of logging of progress on the download threads.