REF: Change how costs are reduced

[carterbox]

Purpose

Try to avoid host synchronization during cost function computation.

Approach

Store objective function values in an GPU array until the end of the epoch.

Pre-Merge Checklists

Submitter

• Write a helpfully descriptive pull request title.
• Organize changes into logically grouped commits with descriptive commit messages.
• Document all new functions.
• Click 'details' on the readthedocs check to view the updated docs.
• Write tests for new functions or explain why they are not needed.
• Address any complaints from pep8speaks.

Reviewer

• Actually read all of the code.
• Run the new code yourself; the included tests should make this easy.
• Write a summary of the changes as you understand them.
• Thank the submitter.

[pep8speaks]

Hello @carterbox! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/tike/ptycho/solvers/lstsq.py:

Line 291:81: E501 line too long (90 > 80 characters)
Line 321:81: E501 line too long (90 > 80 characters)

• In the file src/tike/ptycho/solvers/rpie.py:

Line 105:81: E501 line too long (85 > 80 characters)
Line 185:81: E501 line too long (82 > 80 characters)

Comment last updated at 2024-07-29 15:19:00 UTC

[a4894z]

I had to look up what itertools.chain() does; it looks like the point of this change is to not reduce (i.e. compute the mean) of the costs here in ptycho.py because we need these for momentum acceleration in rPIE or LSTSQ?

And by using itertools.chain() here, we keep track of the costs vs scan position until any momentum acceleration computation is completed and then we can take the mean in LSTSQ or rPIE?

[carterbox]

We want to track a unique objective value for each GPU because of momentum acceleration. e.g. If you do a reconstruction with 4 GPUs for 100 epochs, the resulting costs matrix should be shaped (100, 4).

Locally (to each GPU), the convergence behavior can be different than the other GPUs, so we don't want to use the global mean cost to make decisions about momentum acceleration.

The FIXME exists because I think the momentum acceleration method is still using the global mean to make decisions about momentum acceleration. I should just fix that before merging this PR.