Mathematical Description of LP

If the network component of NPSC is removed, the result is the mathematical description of the linear programming problem. If an LP has variables, and constraints, then the formal statement of the problem solved by PROC NETFLOW is

where

• is the objective function coefficient vector

• is the variable value vector

• is the constraint coefficient matrix for variables, where is the coefficient of variable in the th constraint

• is the side constraint right-hand-side vector

• is the variable value lower bound vector

• is the variable value upper bound vector

Interior Point Algorithmic Details

After preprocessing, the linear program to be solved is

This is the primal problem. The matrices , , and of NPSC have been renamed , , and respectively, as these symbols are by convention used more, the problem to be solved is different from the original because of preprocessing, and there has been a change of primal variable to transform the LP into one whose variables have zero lower bounds. To simplify the algebra here, assume that variables have infinite bounds, and constraints are equalities. (Interior point algorithms do efficiently handle finite bounds, and it is easy to introduce primal slack variables to change inequalities into equalities.) The problem has variables; is a variable number, is an iteration number, and if used as a subscript or superscript it denotes "of iteration ".

There exists an equivalent problem, the dual problem, stated as

where are dual variables, and are dual constraint slacks.

The interior point algorithm solves the system of equations to satisfy the Karush-Kuhn-Tucker (KKT) conditions for optimality:

These are the conditions for feasibility, with the complementarity condition added. Complementarity forces the optimal objectives of the primal and dual to be equal, , as

Before the optimum is reached, a solution may not satisfy the KKT conditions:

• Primal constraints may be violated, .

• Dual constraints may be violated, .

• Complementarity may not be satisfied, .

  This is called the duality gap.

The interior point algorithm works by using Newton’s method to find a direction to move from the current solution toward a better solution:

where is the step length and is assigned a value as large as possible but and not so large that an or is "too close" to zero. The direction in which to move is found using the following:

where , , and is a vector with all elements equal to 1.

To greatly improve performance, the third equation is changed to

where , the average complementarity, and

The effect now is to find a direction in which to move to reduce infeasibilities and to reduce the complementarity toward zero, but if any is too close to zero, it is "nudged out" to , and any that is larger than is "nudged into" . A close to or equal to 0.0 biases a direction toward the optimum, and a value for close to or equal to 1.0 "centers" the direction toward a point where all pairwise products . Such points make up the central path in the interior. Although centering directions make little, if any, progress in reducing and moving the solution closer to the optimum, substantial progress toward the optimum can usually be made in the next iteration.

The central path is crucial to why the interior point algorithm is so efficient. This path "guides" the algorithm to the optimum through the interior of feasible space. Without centering, the algorithm would find a series of solutions near each other close to the boundary of feasible space. Step lengths along the direction would be small and many more iterations would probably be required to reach the optimum.

That in a nutshell is the primal-dual interior point algorithm. Varieties of the algorithm differ in the way and are chosen and the direction adjusted during each iteration. A wealth of information can be found in the texts by Roos, Terlaky, and Vial (1997), Wright (1996), and Ye (1996).

The calculation of the direction is the most time-consuming step of the interior point algorithm. Assume the th iteration is being performed, so the subscript and superscript can be dropped from the algebra:

Rearranging the second equation,

Rearranging the third equation,

where

Equating these two expressions for and rearranging,

where

Substituting into the first direction equation,

, , , and are calculated in that order. The hardest term is the factorization of the matrix to determine . Fortunately, although the values of are different for each iteration, the locations of the nonzeros in this matrix remain fixed; the nonzero locations are the same as those in the matrix . This is due to being a diagonal matrix, which has the effect of merely scaling the columns of .

The fact that the nonzeros in have a constant pattern is exploited by all interior point algorithms, and is a major reason for their excellent performance. Before iterations begin, is examined and its rows and columns are permuted so that during Cholesky Factorization, the number of fill-ins created is smaller. A list of arithmetic operations to perform the factorization is saved in concise computer data structures (working with memory locations rather than actual numerical values). This is called symbolic factorization. During iterations, when memory has been initialized with numerical values, the operations list is performed sequentially. Determining how the factorization should be performed again and again is unnecessary.

The Primal-Dual Predictor-Corrector Interior Point Algorithm

The variant of the interior point algorithm implemented in PROC NETFLOW is a Primal-Dual Predictor-Corrector interior point algorithm. At first, Newton’s method is used to find a direction to move , but calculated as if is zero, that is, a step with no centering, known as an affine step:

where is the step length as before.

Complementarity is calculated at and compared with the complementarity at the starting point , and the success of the affine step is gauged. If the affine step was successful in reducing the complementarity by a substantial amount, the need for centering is not great, and the value of in the following linear system is assigned a value close to zero. If, however, the affine step was unsuccessful, centering would be beneficial, and the value of in the following linear system is assigned a value closer to 1.0. The value of is therefore adaptively altered depending on the progress made toward the optimum.

A second linear system is solved to determine a centering vector from

Then

where, as before, is the step length assigned a value as large as possible but not so large that an or is "too close" to zero.

Although the Predictor-Corrector variant entails solving two linear system instead of one, fewer iterations are usually required to reach the optimum. The additional overhead of calculating the second linear system is small, as the factorization of the matrix has already been performed to solve the first linear system.

Stopping Criteria

There are several reasons why PROC NETFLOW stops interior point optimization. Optimization stops when

• the number of iteration equals MAXITERB=

• the relative gap between the primal and dual objectives is smaller than the value of the PDGAPTOL= option, and both the primal and dual problems are feasible. Duality gap is defined in the section Interior Point Algorithmic Details.

PROC NETFLOW may stop optimization when it detects that the rate at which the complementarity or dualitygap is being reduced is too slow, that is, there are consecutive iterations when the complementarity or duality gap has stopped getting smaller and the infeasibilities, if nonzero, have also stalled. Sometimes, this indicates the problem is infeasible.

The reasons to stop optimization outlined in the previous paragraph will be termed the usual stopping conditions in the following explanation.

However, when solving some problems, especially if the problems are large, the usual stopping criteria are inappropriate. PROC NETFLOW might stop prematurely. If it were allowed to perform additional optimization, a better solution would be found. On other occasions, PROC NETFLOW might do too much work. A sufficiently good solution might be reached several iterations before PROC NETFLOW eventually stops.

You can see PROC NETFLOW’s progress to the optimum by specifying PRINTLEVEL2=2. PROC NETFLOW will produce a table on the SAS log. A row of the table is generated during each iteration and consists of values of the affine step complementarity, the complementarity of the solution for the next iteration, the total bound infeasibility (see the array in the section Interior Point: Upper Bounds), the total constraint infeasibility (see the array in the section Interior Point Algorithmic Details), and the total dual infeasibility (see the array in the section Interior Point Algorithmic Details). As optimization progresses, the values in all columns should converge to zero.

To tailor stopping criteria to your problem, you can use two sets of parameters: the STOP_x and the KEEPGOING_x parameters. The STOP_x parameters (STOP_C, STOP_DG, STOP_IB, STOP_IC, and STOP_ID) are used to test for some condition at the beginning of each iteration and if met, to stop immediately. The KEEPGOING_x parameters ( KEEPGOING_C, KEEPGOING_DG, KEEPGOING_IB, KEEPGOING_IC, and KEEPGOING_ID) are used when PROC NETFLOW would ordinarily stop but does not if some conditions are not met.

For the sake of conciseness, a set of options will be referred to as the part of the option name they have in common followed by the suffix x. For example, STOP_C, STOP_DG, STOP_IB, STOP_IC, and STOP_ID will collectively be referred to as STOP_x.

At the beginning of each iteration, PROC NETFLOW will test whether complementarity is STOP_C (provided you have specified a STOP_C parameter) and if it is, PROC NETFLOW will stop. If the duality gap is STOP_DG (provided you have specified a STOP_DG parameter), PROC NETFLOW will stop immediately. This is also true for the other STOP_x parameters that are related to infeasibilities, STOP_IB, STOP_IC, and STOP_ID.

For example, if you want PROC NETFLOW to stop optimizing for the usual stopping conditions, plus the additional condition, complementarity 100 or duality gap 0.001, then use

  proc netflow stop_c=100 stop_dg=0.001

If you want PROC NETFLOW to stop optimizing for the usual stopping conditions, plus the additional condition, complementarity 1000 and duality gap 0.001 and constraint infeasibility 0.0001, then use

  proc netflow 
     and_stop_c=1000 and_stop_dg=0.01 and_stop_ic=0.0001

Unlike the STOP_x parameters that cause PROC NETFLOW to stop when any one of them is satisfied, the corresponding AND_STOP_x parameters ( AND_STOP_C, AND_STOP_DG, AND_STOP_IB, AND_STOP_IC, and AND_STOP_ID) cause PROC NETFLOW to stop only if all (more precisely, all that are specified) options are satisfied. For example, if PROC NETFLOW should stop when

• complementarity 100 or duality gap 0.001 or

• complementarity 1000 and duality gap 0.001 and constraint infeasibility 0.000

then use

  proc netflow 
     stop_c=100 stop_dg=0.001 
     and_stop_c=1000 and_stop_dg=0.01 and_stop_ic=0.0001

Just as the STOP_x parameters have AND_STOP_x partners, the KEEPGOING_x parameters have AND_KEEPGOING_x partners. The role of the KEEPGOING_x and AND_KEEPGOING_x parameters is to prevent optimization from stopping too early, even though a usual stopping criterion is met.

When PROC NETFLOW detects that it should stop for a usual stopping condition, it performs the following tests:

• It will test whether complementarity is > KEEPGOING_C (provided you have specified a KEEPGOING_C parameter), and if it is, PROC NETFLOW will perform more optimization.

• Otherwise, PROC NETFLOW will then test whether the primal-dual gap is > KEEPGOING_DG (provided you have specified a KEEPGOING_DG parameter), and if it is, PROC NETFLOW will perform more optimization.

• Otherwise, PROC NETFLOW will then test whether the total bound infeasibility KEEPGOING_IB (provided you have specified a KEEPGOING_IB parameter), and if it is, PROC NETFLOW will perform more optimization.

• Otherwise, PROC NETFLOW will then test whether the total constraint infeasibility KEEPGOING_IC (provided you have specified a KEEPGOING_IC parameter), and if it is, PROC NETFLOW will perform more optimization.

• Otherwise, PROC NETFLOW will then test whether the total dual infeasibility KEEPGOING_ID (provided you have specified a KEEPGOING_ID parameter), and if it is, PROC NETFLOW will perform more optimization.

• Otherwise it will test whether complementarity is > AND_KEEPGOING_C (provided you have specified an AND_KEEPGOING_C parameter), and the primal-dual gap is > AND_KEEPGOING_DG (provided you have specified an AND_KEEPGOING_DG parameter), and the total bound infeasibility AND_KEEPGOING_IB (provided you have specified an AND_KEEPGOING_IB parameter), and the total constraint infeasibility AND_KEEPGOING_IC (provided you have specified an AND_KEEPGOING_IC parameter) and the total dual infeasibility AND_KEEPGOING_ID (provided you have specified an AND_KEEPGOING_ID parameter), and if it is, PROC NETFLOW will perform more optimization.

If all these tests to decide whether more optimization should be performed are false, optimization is stopped.

For example,

  proc netflow 
     stop_c=1000 
     and_stop_c=2000 and_stop_dg=0.01 
     and_stop_ib=1 and_stop_ic=1 and_stop_id=1 
     keepgoing_c=1500 
     and_keepgoing_c=2500 and_keepgoing_dg=0.05 
     and_keepgoing_ib=1 and_keepgoing_ic=1 and_keepgoing_id=1 

At the beginning of each iteration, PROC NETFLOW will stop if

• complementarity 1000 or

• complementarity 2000 and duality gap 0.01 and the total bound, constraint, and dual infeasibilities are each 1

When PROC NETFLOW determines it should stop because a usual stopping condition is met, it will stop only if

• complementarity 1500 or

• complementarity 2500 and duality gap 0.05 and the total bound, constraint, and dual infeasibilities are each 1

Interior Point: Upper Bounds

If the LP model had upper bounds ( where is the upper bound vector), then the primal and dual problems, the duality gap, and the KKT conditions would have to be expanded.

The primal linear program to be solved is

where is split into and . Let be primal slack so that , and associate dual variables with these constraints. The interior point algorithm solves the system of equations to satisfy the Karush-Kuhn-Tucker (KKT) conditions for optimality:

These are the conditions for feasibility, with the complementarity conditions and added. Complementarity forces the optimal objectives of the primal and dual to be equal, , as

Before the optimum is reached, a solution might not satisfy the KKT conditions:

• Primal bound constraints may be violated, .

• Primal constraints may be violated, .

• Dual constraints may be violated, .

• Complementarity conditions may not be satisfied, and .

The calculations of the interior point algorithm can easily be derived in a fashion similar to calculations for when an LP has no upper bounds. See the paper by Lustig, Marsten, and Shanno (1992). An important point is that upper bounds can be handled by specializing the algorithm and not by generating the constraints and adding these to the main primal constraints .

Getting Started: Linear Programming Models: Interior Point Algorithm

To solve linear programming problem using PROC NETFLOW, you save a representation of the variables and the constraints in one or two SAS data sets. These data sets are then passed to PROC NETFLOW for solution. There are various forms that a problem’s data can take. You can use any one or a combination of several of these forms.

The ARCDATA= data set contains information about the variables of the problem. Although this data set is called ARCDATA, it contains data for no arcs. Instead, all data in this data set are related to variables.

The ARCDATA= data set can be used to specify information about variables, including objective function coefficients, lower and upper value bounds, and names. These data are the elements of the vectors , , and in problem ( NPSC). Data for a variable can be given in more than one observation.

When the data for a constrained network problem is being provided, the ARCDATA= data set always contains information necessary for arcs, their tail and head nodes, and optionally the supply and demand information of these nodes. When the data for a linear programming problem is being provided, none of this information is present, as the model has no arcs. This is the way PROC NETFLOW decides which type of problem it is to solve.

PROC NETFLOW was originally designed to solve models with networks, so an ARCDATA= data set is always expected. If an ARCDATA= data set is not specified, by default the last data set created before PROC NETFLOW is invoked is assumed to be an ARCDATA= data set. However, these characteristics of PROC NETFLOW are not helpful when a linear programming problem is being solved and all data is provided in a single data set specified by the CONDATA= data set, and that data set is not the last data set created before PROC NETFLOW starts. In this case, you must specify that an ARCDATA= data set and a CONDATA= data set are both equal to the input data set. PROC NETFLOW then knows that a linear programming problem is to be solved, and the data reside in one data set.

The CONDATA= data set describes the constraints and their right-hand sides. These data are elements of the matrix and the vector .

Constraint types are also specified in the CONDATA= data set. You can include in this data set variable data such as upper bound values, lower value bounds, and objective function coefficients. It is possible to give all information about some or all variables in the CONDATA= data set.

A variable is identified in this data set by its name. If you specify a variable’s name in the ARCDATA= data set, then this name is used to associate data in the CONDATA= data set with that variable.

If you use the dense constraint input format, these variable names are names of SAS variables in the VAR list of the CONDATA= data set.

If you use the sparse constraint input format, these variable names are values of the COLUMN list SAS variable of CONDATA= data set.

When using the interior point algorithm, the execution of PROC NETFLOW has two stages. In the preliminary (zeroth) stage, the data are read from the ARCDATA= data set (if used) and the CONDATA= data set. Error checking is performed. In the next stage, the linear program is preprocessed, then the optimal solution to the linear program is found. The solution is saved in the CONOUT= data set. This data set is also named in the PROC NETFLOW, RESET, and SAVE statements.

See the section Getting Started: Network Models: Interior Point Algorithm for a fuller description of the stages of the interior point algorithm.

Introductory Example: Linear Programming Models: Interior Point Algorithm

Consider the linear programming problem in the section An Introductory Example in the chapter on the LP procedure.

data dcon1;
  input _id_ $17.
        a_light a_heavy brega naphthal naphthai 
        heatingo jet_1 jet_2
        _type_ $ _rhs_;
  datalines;
profit            -175 -165 -205  0  0  0 300 300 max     .
naphtha_l_conv    .035 .030 .045 -1  0  0   0   0  eq     0
naphtha_i_conv    .100 .075 .135  0 -1  0   0   0  eq     0
heating_o_conv    .390 .300 .430  0  0 -1   0   0  eq     0
recipe_1             0    0    0  0 .3 .7  -1   0  eq     0
recipe_2             0    0    0 .2  0 .8   0  -1  eq     0
available          110  165   80  .  .  .   .   . upperbd .
;

To find the minimum cost solution and to examine all or parts of the optimum, you use PRINT statements.

• print problem/short; outputs information for all variables and all constraint coefficients. See Figure 7.19.

• print some_variables(j:)/short; is information about a set of variables, (in this case, those with names that start with the character string preceding the colon). See Figure 7.20.

• print some_cons(recipe_1)/short; is information about a set of constraints (here, that set only has one member, the constraint called recipe_1). See Figure 7.21.

• print con_variables(_all_,brega)/short; lists the constraint information for a set of variables (here, that set only has one member, the variable called brega). See Figure 7.22.

• print con_variables(recipe:,n: jet_1)/short; coefficient information for those in a set of constraints belonging to a set of variables. See Figure 7.23.

proc netflow
   arcdata=dcon1
   condata=dcon1
   conout=solutn1;
run;
print problem/short;

print some_variables(j:)/short;
print some_cons(recipe_1)/short;
print con_variables(_all_,brega)/short;
print con_variables(recipe:,n: jet_1)/short;

The following messages, which appear on the SAS log, summarize the model as read by PROC NETFLOW and note the progress toward a solution: