The Interior Point Algorithm

The INTPOINT Procedure

The Interior Point Algorithm

The Simplex algorithm, developed shortly after World War II, was for many years the main method used to solve Linear Programming problems. Over the last fifteen years, however, the Interior Point algorithm has been developed. This algorithm also solves Linear Programming problems. From the start it showed great theoretical promise, and considerable research in the area resulted in practical implementations that performed competitively with the Simplex algorithm. More recently, Interior Point algorithms have evolved to become superior to the Simplex algorithm, in general, especially when the problems are large.

There are many variations of Interior Point algorithms. PROC INTPOINT uses the Primal-Dual with Predictor-Corrector algorithm. More information on this particular algorithm and related theory can be found in the texts by Roos, Terlaky, and Vial (1997), Wright (1996), and Ye (1996).

Interior Point Algorithmic Details

After preprocessing, the Linear Program to be solved is

: min {c^T x}
subject to: A x = b
: $x \geq 0$

This is the primal problem. The matrices of d, z, and Q of NPSC have been renamed c, x, and A, respectively, as these symbols are by convention used more, the problem to be solved is different from the original because of preprocessing, and there has been a change of primal variable to transform the LP into one whose variables have zero lower bounds. To simplify the algebra here, assume that variables have infinite upper bounds, and constraints are equalities. (Interior Point algorithms do efficiently handle finite upper bounds, and it is easy to introduce primal slack variables to change inequalities into equalities.) The problem has n variables. i is a variable number. k is an iteration number, and if used as a subscript or superscript it denotes "of iteration k".

There exists an equivalent problem, the dual problem, stated as

: max {b^T y}
subject to: A^T y + s = c
: $s \geq 0$
where: y are dual variables, and s are dual constraint slacks

What the Interior Point has to do is to solve the system of equations to satisfy the Karush-Kuhn-Tucker (KKT) conditions for optimality:

: A x = b
: A^T y + s = c
: X S e = 0
: $x \geq 0$
: $s \geq 0$
where: S = diag(s), (that is, S_i,j = s_i if i=j, S_i,j = 0 otherwise)
: X = diag(x), and
: $e_i=1 \forall i$

These are the conditions for feasibility, with the complementarity condition X S e = 0 added. c^T x = b^T y must occur at the optimum. Complementarity forces the optimal objectives of the primal and dual to be equal, c^T x_opt = b^T y_opt, as

: 0 = x^T_opt s_opt = s^T_opt x_opt = (c - A^T y_opt)^T x_opt = c^T x_opt - y^T_opt (A x_opt) = c^T x_opt - b^T y_opt
therefore: 0 = c^T x_opt - b^T y_opt

Before the optimum is reached, a solution (x, y, s) may not satisfy the KKT conditions:

Primal constraints can be broken, $infeas_c=b - A x \neq 0$ .
Dual constraints can be broken, $infeas_d=c - A^T y - s \neq 0$ .
Complementarity is unsatisfied, $x^T s=c^T x - b^T y \neq 0$ .This is called the duality gap.

The Interior Point algorithm works by using Newton's method to find a direction to move $(\Delta x^k, \Delta y^k, \Delta s^k)$ from the current solution (x^k, y^k, s^k) toward a better solution:

: $(x^{k+1}, y^{k+1}, s^{k+1})=(x^k, y^k, s^k) + \alpha (\Delta x^k, \Delta y^k, \Delta s^k)$

$\alpha$ is the step length and is assigned a value as large as possible and not so large that a x^k+1_i or s^k+1_i is "too close" to zero. The direction in which to move is found using

: $A \Delta x^k=infeas_c$
: $A^T \Delta y^k + \Delta s^k=infeas_d$
: $S^k \Delta x^k + X^k \Delta s^k=- X^k S^k e$

To greatly improve performance, the third equation is changed to

: $S^k \Delta x^k + X^k \Delta s^k=- X^k S^k e + \sigma_k \mu_k e$
where: $\mu_k=(x^k)^T s^k / n$ , the average complementarity, and
: $0 \leq \sigma_k \leq 1$

The effect now is to find a direction in which to move to reduce infeasibilities and to reduce the complementarity toward zero, but if any x^k_i s^k_i is too close to zero, it is "nudged out" to $\mu$ ,and any x^k_i s^k_i that is larger than $\mu$ is "nudged into" $\mu$ .A $\sigma_k$ close to or equal to 0.0 biases a direction toward the optimum, and a value for $\sigma_k$ close to or equal to 1.0 "centers" the direction toward a point where all pairwise products $x^k_i s^k_i=\mu$ .Such points make up the Central Path in the interior. Although centering directions make little, if any, progress in reducing $\mu$ and moving the solution closer to the optimum, substantial progress toward the optimum can usually be made in the next iteration.

The Central Path is crucial to why the Interior Point algorithm is so efficient. As $\mu$ is decreased, this path "guides" the algorithm to the optimum through the interior of feasible space. Without centering, the algorithm would find a series of solutions near each other close to the boundary of feasible space. Step lengths along the direction would be small and many more iterations would probably be required to reach the optimum.

That in a nutshell is the Primal-Dual Interior Point algorithm. Varieties of the algorithm differ in the way $\alpha$ and $\sigma_k$ are chosen and the direction adjusted during each iteration. A wealth of information can be found in the texts by Roos, Terlaky, and Vial (1997), Wright (1996), and Ye (1996).

The calculation of the direction is the most time-consuming step of the Interior Point algorithm. Assume the kth iteration is being performed, so the subscript and superscript k can be dropped from the algebra:

: $A \Delta x=infeas_c$
: $A^T \Delta y + \Delta s=infeas_d$
: $S \Delta x + X \Delta s=- X S e + \sigma \mu e$

Rearranging the second equation

: $\Delta s=infeas_d - A^T \Delta y$

Rearranging the third equation

: $\Delta s=X^{-1}(- S \Delta x - X S e + \sigma \mu e)$
: $\Delta s=- \Theta \Delta x - S e + X^{-1} \sigma \mu e$
where: $\Theta=S X^{-1}$

Equating these two expressions for $\Delta s$ and rearranging

: $-\Theta \Delta x - S e + X^{-1} \sigma \mu e=infeas_d - A^T \Delta y$
: $-\Theta \Delta x=S e - X^{-1} \sigma \mu e + infeas_d - A^T \Delta y$
: $\Delta x=\Theta^{-1}(-S e + X^{-1} \sigma \mu e - infeas_d + A^T \Delta y)$
: $\Delta x=\rho + \Theta^{-1} A^T \Delta y$
where: $\rho=\Theta^{-1}(-S e + X^{-1} \sigma \mu e - infeas_d)$

Substituting into the first direction equation

: $A \Delta x=infeas_c$
: $A (\rho + \Theta^{-1} A^T \Delta y)=infeas_c$
: $A \Theta^{-1} A^T \Delta y=infeas_c - A \rho$
: $\Delta y=(A \Theta^{-1} A^T)^{-1}(infeas_c - A \rho)$

$\Theta$ , $\rho$ , $\Delta y$ , $\Delta x$ , and $\Delta s$ are calculated in that order. The hardest term is the factorization of the $(A \Theta^{-1} A^T)$ matrix to determine $\Delta y$ . Fortunately, although the values of $(A \Theta^{-1} A^T)$ are different for each iteration, the locations of the nonzeros in this matrix remain fixed; the nonzero locations are the same as those in the matrix (A A^T). This is because $\Theta^{-1}=X S^{-1}$ is a diagonal matrix that has the effect of merely scaling the columns of (A A^T).

The fact that the nonzeros in $A \Theta^{-1} A^T$ have a constant pattern is exploited by all Interior Point algorithms and is a major reason for their excellent performance. Before iterations begin, A A^T is examined and its rows and columns are symmetrically permutated so that during Cholesky factorization, the number of fillins created is smaller. A list of arithmetic operations to perform the factorization is saved in concise computer data structures (working with memory locations rather than actual numerical values). This is called symbolic factorization. During iterations, when memory has been initialized with numerical values, the operations list is performed sequentially. Determining how the factorization should be performed again and again is unnecessary.

The Primal-Dual Predictor-Corrector Interior Point Algorithm

The variant of the Interior Point algorithm implemented in PROC INTPOINT is a Primal-Dual Predictor-Corrector Interior Point algorithm. At first, Newton's method is used to find a direction $(\Delta x^k_{aff}, \Delta y^k_{aff}, \Delta s^k_{aff})$ to move, but calculated as if $\mu$ is zero, that is, as a step with no centering, known as an affine step:

: $A \Delta x^k_{aff}=infeas_c$
: $A^T \Delta y^k_{aff} + \Delta s^k_{aff}=infeas_d$
: $S^k \Delta x^k_{aff} + X^k \Delta s^k_{aff}=- X^k S^k e$

: $(x^k_{aff}, y^k_{aff}, s^k_{aff})=(x^k, y^k, s^k) + \alpha (\Delta x^k_{aff}, \Delta y^k_{aff}, \Delta s^k_{aff})$

$\alpha$ is the step length as before.

Complementarity x^T s is calculated at (x^k_aff, y^k_aff, s^k_aff) and compared with the complementarity at the starting point (x^k, y^k, s^k), and the success of the affine step is gauged. If the affine step was successful in reducing the complementarity by a substantial amount, the need for centering is not great, and $\sigma_k$ in the following linear system is assigned a value close to zero. If, however, the affine step was unsuccessful, centering would be beneficial, and $\sigma_k$ in the following linear system is assigned a value closer to 1.0. The value of $\sigma_k$ is therefore adaptively altered depending on the progress made toward the optimum.

A second linear system is solved to determine a centering vector $(\Delta x^k_{c}, \Delta y^k_{c}, \Delta s^k_{c})$ from (x^k_aff, y^k_aff, s^k_aff):

: $A \Delta x^k_{c}=0$
: $A^T \Delta y^k_{c} + \Delta s^k_{c}=0$
: $S^k \Delta x^k_{c} + X^k \Delta s^k_{c}=- X^k_{aff} S^k_{aff} e + \sigma_k \mu_k e$

then

: $(\Delta x^k, \Delta y^k, \Delta s^k)=(\Delta x^k_{aff}, \Delta y^k_{aff}, \Delta s^k_{aff}) + (\Delta x^k_{c}, \Delta y^k_{c}, \Delta s^k_{c})$

: $(x^{k+1}, y^{k+1}, s^{k+1})=(x^k, y^k, s^k) + \alpha (\Delta x^k, \Delta y^k, \Delta s^k)$

where, as before, $\alpha$ is the step length assigned a value as large as possible but not so large that a x^k+1_i or s^k+1_i is "too close" to zero.

Although the Predictor-Corrector variant entails solving two linear systems instead of one, fewer iterations are usually required to reach the optimum. The additional overhead of calculating the second linear system is small, as the factorization of the $(A \Theta^{-1} A^T)$ matrix has already been performed to solve the first linear system.

Interior Point: Upper Bounds

If the LP had upper bounds ( $0 \leq x \leq u$ where u is the upper bound vector), then the primal and dual problems, the duality gap, and the KKT conditions would have to be expanded.

The primal Linear Program to be solved is

: min {c^T x}
subject to: A x = b
: $0 \leq x \leq u$

$0 \leq x \leq u$ is split into $x \geq 0$ and $x \leq u$ . Let z be primal slack so that x + z = u, and associate dual variables w with these constraints. The Interior Point solves the system of equations to satisfy the Karush-Kuhn-Tucker (KKT) conditions for optimality:

: A x = b
: x + z = u
: A^T y + s - w = c
: X S e = 0
: Z W e = 0
: $x,s,z,w \geq 0$

These are the conditions for feasibility, with the complementarity conditions X S e = 0 and Z W e = 0 added. c^T x = b^T y - u^T w must occur at the optimum. Complementarity forces the optimal objectives of the primal and dual to be equal, c^T x_opt = b^T y_opt - u^T w_opt, as

: 0 = z^T_opt w_opt = (u - x_opt)^T w_opt = u^T w_opt - x^T_opt w_opt

: 0 = x^T_opt s_opt = s^T_opt x_opt = (c - A^T y_opt + w_opt)^T x_opt = c^T x_opt - y^T_opt (A x_opt) + w_opt)^T x_opt= c^T x_opt - b^T y_opt + u^T w_opt
therefore: 0 = c^T x_opt - b^T y_opt + u^T w_opt

Before the optimum is reached, a solution (x, y, s, z, w) might not satisfy the KKT conditions:

Primal bound constraints can be broken, $infeas_b=u - x - z \neq 0$ .
Primal constraints can be broken, $infeas_c=b - A x \neq 0$ .
Dual constraints can be broken, $infeas_d=c - A^T y - s + w\neq 0$ .
Complementarity conditions are unsatisfied, $x^T s \neq 0$ and $z^T w \neq 0$ .

The calculations of the Interior point algorithm can easily be derived in a fashion similar to calculations for when an LP has no upper bounds. See the paper by Lustig, Marsten, and Shanno (1992).

In some iteration k, the affine step system that must be solved is:

: $\Delta x_{aff} + \Delta z_{aff}=infeas_b$
: $A \Delta x_{aff}=infeas_c$
: $A^T \Delta y_{aff} + \Delta s_{aff} - \Delta w_{aff}=infeas_d$
: $S \Delta x_{aff} + X \Delta s_{aff}=- X S e$
: $Z \Delta w_{aff} + W \Delta z_{aff}=- Z W e$

Therefore, the computations involved in solving the affine step are:

: $\Theta=S X^{-1} + W Z^{-1}$
: $\rho=\Theta^{-1} (infeas_d + (S-W)e - Z^{-1} W infeas_b)$
: $\Delta y_{aff}=(A \Theta^{-1} A^T)^{-1} (infeas_c + A \rho)$
: $\Delta x_{aff}=\Theta^{-1} A^T \Delta y_{aff} - \rho$
: $\Delta z_{aff}=infeas_b - \Delta x_{aff}$
: $\Delta w_{aff}=-We - Z^{-1} W \Delta z_{aff}$
: $\Delta s_{aff}=-Se - X^{-1} S \Delta x_{aff}$

: $(x_{aff}, y_{aff}, s_{aff}, z_{aff}, w_{aff})=(x, y, s, z, w) + \alpha (\Delta x_{aff}, \Delta y_{aff}, \Delta s_{aff}, \Delta z_{aff}, \Delta w_{aff})$

$\alpha$ is the step length as before.

A second linear system is solved to determine a centering vector: $(\Delta x_c, \Delta y_c, \Delta s_c, \Delta z_c, \Delta w_c)$ from (x_aff, y_aff, s_aff, z_aff, w_aff):

: $\Delta x_c + \Delta z_c=0$
: $A \Delta x_c=0$
: $A^T \Delta y_c + \Delta s_c - \Delta w_c=0$
: $S \Delta x_c + X \Delta s_c=- X_{aff} S_{aff} e + \sigma \mu e$
: $Z \Delta w_c + W \Delta z_c=- Z_{aff} W_{aff} e + \sigma \mu e$

where: $\zeta_{start}=x^T s + z^T w$ , complementarity at the start of the iteration
: $\zeta_{aff}=x_{aff}^T s_{aff} + z_{aff}^T w_{aff}$ , the affine complementarity
: $\mu=\zeta_{aff} / 2n$ , the average complementarity
: $\sigma=(\zeta_{aff} / \zeta_{start})^3$

Therefore, the computations involved in solving the centering step are:

: $\rho=\Theta^{-1} (\sigma \mu (X^{-1} - Z^{-1}) e - X^{-1} X_{aff} S_{aff} e + Z^{-1} Z_{aff} W_{aff} e)$
: $\Delta y_c=(A \Theta^{-1} A^T)^{-1} A \rho$
: $\Delta x_c=\Theta^{-1} A^T \Delta y_c - \rho$
: $\Delta z_c=-\Delta x_c$
: $\Delta w_c=\sigma \mu Z^{-1} e - Z^{-1} Z_{aff} W_{aff} e - Z^{-1} W_{aff} \Delta z_c$
: $\Delta s_c=\sigma \mu X^{-1} e - X^{-1} X_{aff} S_{aff} e - X^{-1} S_{aff} \Delta x_c$

Then

: $(\Delta x, \Delta y, \Delta s, \Delta z, \Delta w)=(\Delta x_{aff}, \Delta y_{... ..._{aff}) + (\Delta x_{c}, \Delta y_{c}, \Delta s_{c}, \Delta z_{c}, \Delta w_{c})$

: $(x^{k+1}, y^{k+1}, s^{k+1}, z^{k+1}, w^{k+1})=(x^k, y^k, s^k, z^k, w^k) + \alpha (\Delta x, \Delta y, \Delta s, \Delta z, \Delta w)$

where, as before, $\alpha$ is the step length assigned a value as large as possible but not so large that a x^k+1_i, s^k+1_i, z^k+1_i, or w^k+1_i is "too close" to zero.

The algebra in this section has been simplified by assuming that all variables have finite upper bounds. If the number of variables with finite upper bounds n_u < n, you need to change the algebra to reflect that the "z" and "w" arrays has dimension n_u x 1 or n_u x n_u. Other computations need slight modification. For example, the average complementarity

: $\mu=x_{aff}^T s_{aff} / n + z_{aff}^T w_{aff} / n_u$

An important point is that any upper bounds can be handled by specializing the algorithm and not by generating the constraints $x \leq u$ and adding these to the main primal constraints A x = b.

Chapter Contents
Previous
Next
Top