Sampling conditioned on hyperparameters?

[marius311]

I'm developing an inference algorithm for hierarchical problems which I'd like to hook into Turing, and one of its requirements is sampling observed quantities conditioned on a set of values for the hyperparameters (ie, the parameters on the first "rung" in the hierarchy; fwiw, a similar requirement as for ABC). Statistically this is straightforward by just doing a forward simulation, but I'm not sure to what extent the Turing interface allows it generally.

As a toy example, consider the hierrchical model:

@model function hierarchical(x)
    θ ~ Uniform(0, 1) # hyperparameter
    z ~ Normal(0, θ)  # latent parameter
    x ~ Normal(z, 1)  # observed quantity 
end

The user would generally call model = hierarchical(x) for some observed value x, then pass model along to my algorithm (or to Turing.sample, Turing.optimize, etc...). The requirement of my algorithm is that as part of it, I need to draw samples from P(x|θ) for different values of θ which the algorithm chooses. Is this possible from only the model object?

One option (which I'd like to avoid) is I could force the user to include the hyperparameters as an explicit argument:

@model function hierarchical(x, θ=missing)
    θ ~ Uniform(0, 1) # hyperparameter
    z ~ Normal(0, θ)  # latent parameter
    x ~ Normal(z, 1)  # observed parameter 
end

in which case evaluating something like hierarchical(missing, 0.3)() gives me exactly the quantity I'm looking for, and I can just force the user to also pass along the hierarchy function itself, not just model. But is there any way to do this without making the user do, so that it works like more standard Turing? Thanks.

[torfjelde]

So in the particular case of Uniform, you could just make it an argument without any distribution as it only contributes a constant factor to the resulting log-density computation:)

But in general, we probably need something like a condition as hypothesized in TuringLang/AbstractPPL.jl#10 (comment). It's a "trivial" step after TuringLang/DynamicPPL.jl#249 has been concluded.

Can I ask exactly what you need from the model? E.g. is it just the joint probability you need? Do you need to sample from the prior? Depending on what you need it, it might be possible without waiting for the above work:)

[marius311]

So in the particular case of Uniform, you could just make it an argument without any distribution as it only contributes a constant factor to the resulting log-density computation:)

But the posterior which I want the model to represent is P(θ,z|x) so I think I still need to make θ a random variable so that the sampling algorithms will sample it? (I could be missing some Turing thing, I'm still somewhat new to it)

Can I ask exactly what you need from the model? E.g. is it just the joint probability you need? Do you need to sample from the prior? Depending on what you need it, it might be possible without waiting for the above work:)

The algorithm needs two things.

• It needs to sample x from P(x|θ)
• It needs to compute d/dz log P(θ,z|x)

The latter I've already figured how to get via Turing.gradient_logp and I think is pretty standard. This question is about the former.

But in general, we probably need something like a condition as hypothesized in TuringLang/AbstractPPL.jl#10 (comment). It's a "trivial" step after TuringLang/DynamicPPL.jl#249 has been concluded

Yea, reading over the first issue sounds like condition(model, (θ=θ, x=missing)) is what I'd need. Basically take the model, make the observed value missing so that its sampled, then fix θ at some value of my choosing.