The DISCRIM Procedure

Posterior Probability Error-Rate Estimates

The posterior probability error-rate estimates (Fukunaga and Kessel, 1973; Glick, 1978; Hora and Wilcox, 1982) for each group are based on the posterior probabilities of the observations classified into that same group.

A sample of observations with classification results can be used to estimate the posterior error rates. The following notation is used to describe the sample:

${\mathcal S}$: the set of observations in the (training) sample
n: the number of observations in ${\mathcal S}$
$n_ t$: the number of observations in ${\mathcal S}$ in group t
${\mathcal R}_ t$: the set of observations such that the posterior probability belonging to group t is the largest
${\mathcal R}_{ut}$: the set of observations from group u such that the posterior probability belonging to group t is the largest

The classification error rate for group t is defined as

$e_ t = 1- \int _{{\mathcal R}_ t} f_ t(\mb{x}) d\mb{x}$

The posterior probability of $\mb{x}$ for group t can be written as

$p(t|\mb{x}) = \frac{q_ t f_ t(\mb{x})}{f(\mb{x})}$

where $f(\mb{x}) = \sum _ u q_ u f_ u(\mb{x})$ is the unconditional density of $\mb{x}$ .

Thus, if you replace $f_ t(\mb{x})$ with $p(t|\mb{x}) f(\mb{x})/q_ t$ , the error rate is

$e_ t = 1 - \frac{1}{q_ t} \int _{{\mathcal R}_ t} p(t|\mb{x}) f(\mb{x}) d\mb{x}$

An estimator of $e_ t$ , unstratified over the groups from which the observations come, is then given by

$\hat{e}_ t \mbox{ (unstratified)} = 1 - \frac{1}{nq_ t} \sum _{{\mathcal R}_ t} p(t|\mb{x})$

where $p(t|\mb{x})$ is estimated from the classification criterion, and the summation is over all sample observations of ${\mathcal S}$ classified into group t. The true group membership of each observation is not required in the estimation. The term $nq_ t$ is the number of observations that are expected to be classified into group t, given the priors. If more observations than expected are classified into group t, then $\hat{e}_ t$ can be negative.

Further, if you replace $f(\mb{x})$ with $\sum _ u q_ u f_ u(\mb{x})$ , the error rate can be written as

$e_ t = 1 - \frac{1}{q_ t} \sum _ u q_ u \int _{{\mathcal R}_{ut}} p(t|\mb{x}) f_ u(\mb{x}) d\mb{x}$

and an estimator stratified over the group from which the observations come is given by

$\hat{e}_ t \mbox{ (stratified)} = 1 - \frac{1}{q_ t} \sum _ u q_ u \frac{1}{n_ u} \left( \sum _{{\mathcal R}_{ut}} p(t|\mb{x}) \right)$

The inner summation is over all sample observations of ${\mathcal S}$ coming from group u and classified into group t, and $n_ u$ is the number of observations originally from group u. The stratified estimate uses only the observations with known group membership. When the prior probabilities of the group membership are proportional to the group sizes, the stratified estimate is the same as the unstratified estimator.

The estimated group-specific error rates can be less than zero, usually due to a large discrepancy between prior probabilities of group membership and group sizes. To have a reliable estimate for group-specific error rate estimates, you should use group sizes that are at least approximately proportional to the prior probabilities of group membership.

A total error rate is defined as a weighted average of the individual group error rates

$e = \sum _ t q_ t e_ t$

and can be estimated from

$\hat{e} \mbox{ (unstratified)} = \sum _ t q_ t \hat{e}_ t \mbox{ (unstratified)}$

$\hat{e} \mbox{ (stratified)} = \sum _ t q_ t \hat{e}_ t \mbox{ (stratified)}$

The total unstratified error-rate estimate can also be written as

$\hat{e} \mbox{ (unstratified)} = 1 - \frac{1}{n} \sum _ t \sum _{{\mathcal R}_ t} p(t|\mb{x})$

which is one minus the average value of the maximum posterior probabilities for each observation in the sample. The prior probabilities of group membership do not appear explicitly in this overall estimate.