The MCMC Procedure

Conjugate Sampling

Conjugate prior is a family of prior distributions in which the prior and the posterior distributions are of the same family of distributions. For example, if you model an independently and identically distributed random variable $y_ i$ using a normal likelihood with known variance $\sigma ^2$,

\[ y_ i \sim \mbox{normal}(\mu , \sigma ^2) \]

a normal prior on $\mu $

\[ \mu \sim \mbox{normal}(\mu _0, \sigma _0^2) \]

is a conjugate prior because the posterior distribution of $\mu $ is also a normal distribution given $\mbox{y} = \{ y_ i\} $, $\sigma ^2$, $\mu _0$, and $\sigma _0^2$ :

\[ \mu | \mbox{y} \sim \mbox{normal} \left( \left(\frac{1}{\sigma _0^2} + \frac{n}{\sigma ^2} \right)^{-1} \cdot \left(\frac{\mu _0}{\sigma _0^2} + \frac{n\cdot \bar{\mb{y}}}{\sigma ^2} \right), \left(\frac{1}{\sigma _0^2} + \frac{n}{\sigma ^2} \right)^{-1} \right) \]

Conjugate sampling is efficient because it enables the Markov chain to obtain samples from the target distribution directly. When appropriate, PROC MCMC uses conjugate sampling methods to draw conditional posterior samples. Table 73.6 lists scenarios that lead to conjugate sampling in PROC MCMC.

Table 73.6: Conjugate Sampling in PROC MCMC

Family

Parameter

Prior

Normal with known $\mu $

Variance $\sigma ^2$

Inverse gamma family

Normal with known $\mu $

Precision $\tau $

Gamma family

Normal with known scale parameter ($\sigma ^2$, $\sigma $, or $\tau $)

Mean $\mu $

Normal

Multivariate normal with known $\Sigma $

Mean $\bm {\mu }$

Multivariate normal

Multivariate normal with known $\bm {\mu }$

Covariance $\Sigma $

Inverse Wishart

Multinomial

$\mb{p}$

Dirichlet

Binomial/binary

p

Beta

Poisson

$\lambda $

Gamma family


In most cases, Family in Table 73.6 refers to the likelihood function. However, it does not necessarily have to be the case. The Family is a distribution that is conditional on the parameter of interest, and it can appear in any level of the hierarchical model, including on the random-effects level.

PROC MCMC can detect conjugacy only if the model parameter (not a function or a transformation of the model parameter) is used in the prior and Family distributions. For example, the following statements lead to a conjugate sampler being used on the parameter mu: 

parm mu;
prior mu ~ n(0, sd=1000);
model y ~ n(mu, var=s2);

However, if you modify the program slightly in the following way, although the conjugacy still holds in theory, PROC MCMC cannot detect conjugacy on mu because the parameter enters the normal likelihood function through the symbol w: 

parm mu;
prior mu ~ n(0, sd=1000);
w = mu;
model y ~ n(w, var=s2);

In this case, PROC MCMC resorts to the default sampling algorithm, which is a random walk Metropolis based on a normal kernel.

Similarly, the following statements also prevent PROC MCMC from detecting conjugacy on the parameter mu: 

parm mu;
prior mu ~ n(0, sd=1000);
model y ~ n(mu + 2, var=s2);

In a normal family, an often-used and often-confused conjugate prior on the variance is the inverse gamma distribution, and a conjugate prior on the precision is the gamma distribution. See Gamma and Inverse-Gamma Distributions for typical usages of these prior distributions.

When conjugacy is detected in a model, PROC MCMC performs a numerical optimization on the joint posterior distribution at the start of the MCMC simulation. If the only sampling methods required in the program are conjugate samplers or direct samplers, PROC MCMC omits this optimization step. To turn off this optimization routine, use the PROPCOV=IND option in the PROC MCMC statement.