Add module to support clustering analysis

[sjspielman]

Proposed analysis

I am proposing an analysis module that can support clustering analysis. I am proposing this because, I expect, it will be common for contributors to want to cluster their data and/or assess clustering robustness. A module along those lines will provide a uniform way throughout OpenScPCA to perform this common analysis.

To be frank, my main motivation for this module is that I am interested in exploring the sc-SHC method. This method has two main functionalities, and for now I am primarily interested in the first here (I added bold for emphasis; copied from their README):

This approach is implemented in two ways: (1) a stand-alone clustering pipeline with built-in hypothesis testing to produce clusters corresponding to distinct cell populations and (2) a post-hoc method that can evaluate the statistical significance of any provided set of clusters.

So, to begin, I'd like to build this module around some sc-SHC exploration explicitly, and as we learn more about the method, the module may become a more general clustering module.

Step 1: Is there a there there?

To begin, I envision a benchmark sort of analysis to answer, "is there a there there?" for using sc-SHC:

• Calculate clusters with sc-SHC
• Shuffle expression values to create randomized versions of those clusters
• Calculate some metrics for all, e.g. using metrics we have previously considered as part of scpca-downstream-analyses (cluster purity, silhouette width, cluster stability)
• I suppose we could formally calculate P-values for the sc-SHC clustering values compared to the null distributions (e.g. using medians for each distribution?), but to start off, it's worth doing a simple visual assessment of those distributions before getting into the weeds too much.
• If the sc-SHC clusters are indeed "significantly" better than random, we can proceed to compare those clusters to other clustering approaches, e.g. via varying parameters for walktrap or louvain leiden (again, similar to scpca-downstream-analysis).
  • Here, we'd like to see that sc-SHC is either comparable or more robust than other algorithms.

Step 2: Write helper functions for clustering

There are broadly two sets of findings step 1 could yield and how we would proceed accordingly:

1. If sc-SHC seems like a useful framework, we could proceed to write a helper script and/or function to support others to use it.
   • Or, if it really seems useful, we could actually run ScPCA datasets through the algorithm to generate usable clustering results that complement the "quick 'n dirty" ones we have in ScPCA objects.
2. If sc-SHC does not seem like a useful framework, we might consider adapting, in some way, the scpca-downstream-analyses clustering code for convenience for OpenScPCA contributors. I'm not entirely sure yet how we might provide results - would we pick a "best" clustering result to return, or provide a report for contributors to pick their clustering parameters?
   • This is not terribly fleshed out yet, so to be discussed later further if this ends up being the route!

Scientific goals

The primary goals of this analysis are two-fold:

• Assess if sc-SHC is a valuable method for performing clustering
• Provide a module that can be flexibly used by OpenScPCA contributors to perform and/or evaluate clusters, which will overall improve object processing consistency across modules

There are also some bonus goals!

• If sc-SHC turns out to perform quite well as a clustering method in the first place, we could consider running it across ScPCA data to generate clusters that contributors to use and would likely be more reliable than the quick graph-based results we have in the SCEs.
• This module might also be informative internally in the Data Lab since we're often asked about assessing clustering. We will learn more about sc-SHC method so we can find out if it's something we want to recommend, e.g. to workshop participants.

Methods or approach

To start, mostly sc-SHC which runs fine on a laptop.

Existing modules

N/A

Input data

ScPCA data!

Scientific literature

Here is the sc-SHC paper: https://www.nature.com/articles/s41592-023-01933-9

Other details

No response