Implement Decipher model in external

[justjhong]

CC @ANazaret

Implements Decipher model (https://github.com/azizilab/decipher, https://www.biorxiv.org/content/10.1101/2023.11.11.566719v1) into external/

For now, it only includes base implementation without many of the downstream workflows from the original implementation.
Includes minor non-breaking changes to the LowLevelPyroTrainingPlan.

Test: was able to approximately reproduce figures from the tutorial (https://github.com/azizilab/decipher/blob/main/examples/1-tutorial.ipynb), some of the v plots for several random seeds below:

Original implementation:

New implementation:

[codecov]

Codecov Report

Attention: Patch coverage is 96.67774% with 10 lines in your changes missing coverage. Please review.

Project coverage is 84.61%. Comparing base (38e1d65) to head (e25173b).

Files with missing lines	Patch %	Lines
src/scvi/external/decipher/_components.py	89.36%	5 Missing ⚠️
src/scvi/external/decipher/_module.py	97.34%	3 Missing ⚠️
src/scvi/external/decipher/_trainingplan.py	97.10%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3015      +/-   ##
==========================================
- Coverage   84.80%   84.61%   -0.20%     
==========================================
  Files         173      178       +5     
  Lines       14797    15107     +310     
==========================================
+ Hits        12549    12783     +234     
- Misses       2248     2324      +76     

Files with missing lines	Coverage Δ
src/scvi/external/__init__.py	100.00% <100.00%> (ø)
src/scvi/external/decipher/__init__.py	100.00% <100.00%> (ø)
src/scvi/external/decipher/_model.py	100.00% <100.00%> (ø)
src/scvi/external/decipher/_trainingplan.py	97.10% <97.10%> (ø)
src/scvi/external/decipher/_module.py	97.54% <97.34%> (ø)
src/scvi/external/decipher/_components.py	89.36% <89.36%> (ø)

... and 1 file with indirect coverage changes

[justjhong]

Not sure why the cuda tests are failing on import of the Decipher class. Any ideas? @canergen @ori-kron-wis

[ori-kron-wis]

@justjhong I updated the main branch with the updated cuda test file we need now to run on WIS servers and merged this PR with it. Now the tests work.
However, test_decipher is failing. Not sure why, I can only say it is passing in a non-CUDA environment, and failing in both single and multi GPU environments. Can it be related to the fact that it is Pyro based?

[justjhong]

@ori-kron-wis thanks for your help confirming its a real bug. I will debug this on my local CUDA machine and update the PR accordingly

[justjhong]

@ori-kron-wis I implemented the fix which passes local CUDA tests, but receiving this import error once again. How were you able to bypass this error the previous time? Otherwise, I've prepped the PR with the corresponding release note and docs for the next release. Thank you!

[ori-kron-wis]

you are right, it wasn't solved.
I cleared the runner cache locally on the server + restarted it and now it looks ok again. one of those voodoo things ;-)

[justjhong]

@ori-kron-wis Haha these flaky things are never satisfying but thank you for finding the fix! I will add decipher to the docs first before asking you for the final review.

Aside: do you think there should be a simple PR checklist for the standard things for typical contributions? I keep forgetting things (e.g., CHANGELOG, function level documentation, add classes to docs if necessary, add backport label if needed before merge)

[justjhong]

@ori-kron-wis once checks pass, this PR should be ready for review and merge 👍

[justjhong]

@ori-kron-wis cuda tests failing after merging with main. do you mind taking a look before the pr gets out of date again?

[ori-kron-wis]

@justjhong yeah , had to do major updates on the servers. hope that it will be fine now.

[justjhong]

Thanks @ori-kron-wis ! if everything looks can you approve the PR? @canergen