The empirical distribution function (EDF) is a nonparametric estimate of the cumulative distribution function (CDF) of the distribution. PROC HPSEVERITY computes EDF estimates for two purposes: to send the estimates to a distribution’s PARMINIT subroutine in order to initialize the distribution parameters, and to compute the EDF-based statistics of fit.
To reduce the time that it takes to compute the EDF estimates, you can use the INITSAMPLE option to specify that only a fraction of the input data be used. If you do not specify the INITSAMPLE option and the data set has more than 10,000 valid observations, then a uniform random sample of at most 10,000 observations is used for EDF estimation.
In the distributed mode of execution, in which data are distributed across the grid nodes, the EDF estimates are computed
on each node by using the portion of the input data that is located on that node. These local EDF estimates are an approximation
of the global EDF estimates, which would been computed by using the entire input data set. PROC HPSEVERITY does not compute
global EDF estimates. Let denote a quantity that depends on the EDF estimates.
can be either an EDF-based initial value of a distribution parameter or an EDF-based statistic of fit. PROC HPSEVERITY estimates
as follows: First, each grid node
computes an estimate
by using the local EDF estimates that are computed on that node. Then, the estimate
of
is computed as an average of all the
values; that is,
, where
denotes the total number of nodes where the data reside.
This section describes the methods that are used for computing EDF estimates.
Let there be a set of observations, each containing a quintuplet of values
, where
is the value of the response variable,
is the value of the left-truncation threshold,
is the value of the right-truncation threshold, and
is the value of the right-censoring limit,
is the value of the left-censoring limit.
If an observation is not left-truncated, then , where
is the smallest value in the support of the distribution; so
. If an observation is not right-truncated, then
, where
is the largest value in the support of the distribution; so
. If an observation is not right-censored, then
; so
. If an observation is not left-censored, then
; so
.
Let denote the weight associated with
th observation. If you have specified the WEIGHT statement, then
is the normalized value of the weight variable; otherwise, it is set to 1. The weights are normalized such that they sum
up to
.
An indicator function takes a value of 1 or 0 if the expression
is true or false, respectively.
If the response variable is subject to both left-censoring and right-censoring effects and if you have not specified the EMPIRICALCDF=NOTURNBULL method, then PROC HPSEVERITY uses the Turnbull’s method. This section describes methods other than Turnbull’s method. For Turnbull’s method, see the next section Turnbull’s EDF Estimation Method.
The method descriptions assume that all observations are either uncensored or right-censored; that is, each observation is
of the form or
.
If all observations are either uncensored or left-censored, then each observation is of the form . It is converted to an observation
; that is, the signs of all the response variable values are reversed, the new left-truncation threshold is equal to the negative
of the original right-truncation threshold, the new right-truncation threshold is equal to the negative of the original left-truncation
threshold, and the negative of the original left-censoring limit becomes the new right-censoring limit. With this transformation,
each observation is either uncensored or right-censored. The methods described for handling uncensored or right-censored data
are now applicable. After the EDF estimates are computed, the observations are transformed back to the original form and EDF
estimates are adjusted such
, where
denotes the EDF estimate of the value slightly less than the transformed value
.
Further, a set of uncensored or right-censored observations can be converted to a set of observations of the form , where
is the indicator of right-censoring.
indicates a right-censored observation, in which case
is assumed to record the right-censoring limit
.
indicates an uncensored observation, and
records the exact observed value. In other words,
and
.
Given this notation, the EDF is estimated as
where denotes the
th order statistic of the set
and
is the estimate computed at that value. The definition of
depends on the estimation method. You can specify a particular method or let PROC HPSEVERITY choose an appropriate method
by using the EMPIRICALCDF= option in the PROC HPSEVERITY statement. Each method computes
as follows:
This is the default method. First, censored observations, if any, are processed as follows:
An observation that is left-censored but not right-censored is converted to an uncensored observation , where
.
An observation that is both left-censored and right-censored is converted to an uncensored observation , where
.
An observation that is right-censored but not left-censored is left unchanged.
If the processed set of observations contains any truncated or right-censored observations, the KAPLANMEIER method is used. Otherwise, the STANDARD method is used.
The observations are modified only for the purpose of computing the EDF estimates. The original censoring information is used by the parameter estimation process.
This method is chosen when no censoring or truncation information is specified. It is the standard way of computing EDF. The
EDF estimate at observation is computed as follows:
This method is chosen when at least one form of censoring or truncation is specified. The Kaplan-Meier (KM) estimator, also known as the product-limit estimator, was first introduced by Kaplan and Meier (1958) for censored data. Lynden-Bell (1971) derived a similar estimator for left-truncated data. PROC HPSEVERITY uses the definition that combines both censoring and truncation information (Klein and Moeschberger, 1997; Lai and Ying, 1991).
The EDF estimate at observation is computed as
where and
are defined as follows:
, which is the number of uncensored observations (
) for which the response variable value is equal to
and
is observable according to the right-truncation threshold of that observation (
).
, which is the size (cardinality) of the risk set at
. The term risk set has its origins in survival analysis; it contains the events that are at risk of failure at a given time,
. In other words, it contains the events that have survived up to time
and might fail at or after
. For PROC HPSEVERITY, time is equivalent to the magnitude of the event and failure is equivalent to an uncensored and observable event, where observable means it satisfies the truncation thresholds.
The product-limit estimator used by the KAPLANMEIER method does not work well if the risk set size becomes very small. For right-censored data, the size can become small towards the right tail. For left-truncated data, the size can become small at the left tail and can remain so for the entire range of data. This was demonstrated by Lai and Ying (1991). They proposed a modification to the estimator that ignores the effects due to small risk set sizes.
The EDF estimate at observation is computed as
where the definitions of and
are identical to those used for the KAPLANMEIER method described previously.
You can specify the values of and
by using the C= and ALPHA= options. If you do not specify a value for
, the default value used is
. If you do not specify a value for
, the default value used is
.
As an alternative, you can also specify an absolute lower bound, say , on the risk set size by using the RSLB= option, in which case
is replaced by
in the definition.
If the response variable is subject to both left-censoring and right-censoring effects and if you have not specified the NOTURNBULL method, then the HPSEVERITY procedure uses a method proposed by Turnbull (1976) to compute the nonparametric estimates of the cumulative distribution function. The original Turnbull’s method is modified using the suggestions made by Frydman (1994) when truncation effects are present.
Let the input data consist of observations in the form of quintuplets of values
with notation described in the section Notation. For each observation, let
be the censoring interval; that is, the response variable value is known to lie in the interval
, but the exact value is not known. If an observation is uncensored, then
for any arbitrarily small value of
. If an observation is censored, then the value
is ignored. Similarly, for each observation, let
be the truncation interval; that is, the observation is drawn from a truncated (conditional) distribution
.
Two sets, and
, are formed using
and
as follows:
![]() |
![]() |
![]() |
![]() |
The sets and
represent the left endpoints and right endpoints, respectively. A set of disjoint intervals
,
is formed such that
and
and
and
. The value of
is dependent on the nature of censoring and truncation intervals in the input data. Turnbull (1976) showed that the maximum likelihood estimate (MLE) of the EDF can increase only inside intervals
. In other words, the MLE estimate is constant in the interval
. The likelihood is independent of the behavior of
inside any of the intervals
. Let
denote the increase in
inside an interval
. Then, the EDF estimate is as follows:
PROC HPSEVERITY computes the estimates at points
and
and computes
at point
, where
denotes the limiting estimate at a point that is infinitesimally larger than
when approaching
from values larger than
and where
denotes the limiting estimate at a point that is infinitesimally smaller than
when approaching
from values smaller than
.
PROC HPSEVERITY uses the expectation-maximization (EM) algorithm proposed by Turnbull (1976), who referred to the algorithm as the self-consistency algorithm. By default, the algorithm runs until one of the following criteria is met:
Relative-error criterion: The maximum relative error between the two consecutive estimates of falls below a threshold
. If
indicates an index of the current iteration, then this can be formally stated as
You can control the value of by specifying the EPS= suboption of the EDF=TURNBULL option in the PROC HPSEVERITY statement. The default value is 1.0E–8.
Maximum-iteration criterion: The number of iterations exceeds an upper limit specified by the MAXITER= suboption of the EDF=TURNBULL option in the PROC HPSEVERITY statement. The default number of maximum iterations is .
The self-consistent estimates obtained in this manner might not be maximum likelihood estimates. Gentleman and Geyer (1994) suggested the use of the Kuhn-Tucker conditions for the maximum likelihood problem to ensure that the estimates are MLE.
If you specify the ENSUREMLE suboption of the EDF=TURNBULL option in the PROC HPSEVERITY statement, then PROC HPSEVERITY computes the Kuhn-Tucker conditions
at the end of each iteration to determine whether the estimates {} are MLE. If no truncation effects are specified, then the Kuhn-Tucker conditions derived by Gentleman and Geyer (1994) are used. If truncation effects are specified, then PROC HPSEVERITY uses modified Kuhn-Tucker conditions that account for
the truncation effects. An integral part of checking the conditions is to determine whether an estimate
is zero or whether an estimate of the Lagrange multiplier or the reduced gradient associated with the estimate
is zero. PROC HPSEVERITY declares these values to be zero if they are less than or equal to a threshold
. You can control the value of
by specifying the ZEROPROB= suboption of the EDF=TURNBULL option in the PROC HPSEVERITY statement. The default value is 1.0E–8. The algorithm continues
until the Kuhn-Tucker conditions are satisfied or the number of iterations exceeds the upper limit. The relative-error criterion
stated previously is not used when the ENSUREMLE option is specified.
If truncation is specified, then the estimate computed by any method other than the STANDARD method is a conditional estimate. In other words,
, where
and
denote the (unknown) distribution functions of the left-truncation threshold variable
and the right-truncation threshold variable
, respectively,
denotes the smallest left-truncation threshold with a nonzero cumulative probability, and
denotes the largest right-truncation threshold with a nonzero cumulative probability. Formally,
and
. For computational purposes, PROC HPSEVERITY estimates
and
by
and
, respectively, defined as
![]() |
![]() |
![]() |
![]() |
These estimates of and
are used to compute the conditional estimates of the CDF as described in the section Truncation and Conditional CDF Estimates.
The parameter initialization subroutines in distribution models and some predefined utility functions require EDF estimates. For more information, see the sections Defining a Severity Distribution Model with the FCMP Procedure and Predefined Utility Functions.
PROC HPSEVERITY supplies the EDF estimates to these subroutines and functions by using two arrays, x
and F
, the dimension of each array, and a type of the EDF estimates. The type identifies how the EDF estimates are computed and
stored. They are as follows:
specifies that EDF estimates are computed using the STANDARD method; that is, the data used for estimation are neither censored nor truncated.
specifies that EDF estimates are computed using the KAPLANMEIER method; that is, the data used for estimation are subject to at least one form of truncation or censoring.
specifies that EDF estimates are computed using the TURNBULL method; that is, the data used for estimation are subject to both left- and right-censoring. The data might or might not be truncated.
For Types 1 and 2, the EDF estimates are stored in arrays x
and F
of dimension N
such that the following holds:
where denotes
th element of the array ([1] denotes the first element of the array).
For Type 3, the EDF estimates are stored in arrays x
and F
of dimension N
such that the following holds:
Although the behavior of EDF is theoretically undefined for the interval , for computational purposes, all predefined functions and subroutines assume that the EDF increases linearly from
to
in that interval if
. If
, which can happen when the EDF is estimated from a combination of uncensored and interval-censored data, the predefined functions
and subroutines assume that
.