[Suggestion] Implement GaussianMixturePosterior via torch.distributions.MixtureSameFamily

[chrisyeh96]

Currently, GaussianMixturePosterior is implemented as a MultivariateNormal distributions with a batch_shape equal to the number of components in the Gaussian mixture. I wonder if it would make more sense (and be more semantically accurate) to use a MixtureSameFamily distribution with MultivariateNormal components without a batch_shape. An additional benefit is that MixtureSameFamily already implements the mean and variance functions. (The quantile function would still need to be implemented within BoTorch.)

[Balandat]

Thanks for the suggestion - overall this makes sense to me conceptually, but we should look a bit deeper into how this would interact with some of the specifics in gpytorch's MultivariateNormal distribution (e.g. handing the _is_mt property). At the end of the day, accounting for this might end up complicating the implementation rather than simplifying it. That said, if we wanted to expand the functionality towards more generally having non-uniform component weights and/or more general posterior distributions, this would be the way to go.

cc @dme65, @sdaulton

[saitcakmak]

Also worth noting that the current design goes hand in hand with the implementation of SaasFullyBayesianSingleTaskGP (and variants) that are batched models that produce batched MVNs. Changing the posterior implementation would require either breaking down the batched MVN once it is computed, or changing the model internals from batched computations to a list of computations, which will come with performance penalties.