The NLP Procedure

Criteria for Optimality

Subsections:

Karush-Kuhn-Tucker Conditions
Derivatives

PROC NLP solves

$\begin{array}{lll} \min _{x \in {\mathcal R}^ n} & f(x) & \\ \mr {subject\ to} & c_ i(x) = 0 , & i = 1,\dots ,m_ e \\ & c_ i(x) \ge 0 , & i = m_ e+1,\dots ,m \end{array}$

where $f$ is the objective function and the $c_ i$ ’s are the constraint functions.

A point $x$ is feasible if it satisfies all the constraints. The feasible region ${\mathcal G}$ is the set of all the feasible points. A feasible point $x^*$ is a global solution of the preceding problem if no point in ${\mathcal G}$ has a smaller function value than $f(x^*$ ). A feasible point $x^*$ is a local solution of the problem if there exists some open neighborhood surrounding $x^*$ in that no point has a smaller function value than $f(x^*$ ). Nonlinear programming algorithms cannot consistently find global minima. All the algorithms in PROC NLP find a local minimum for this problem. If you need to check whether the obtained solution is a global minimum, you may have to run PROC NLP with different starting points obtained either at random or by selecting a point on a grid that contains ${\mathcal G}$ .

Every local minimizer $x^*$ of this problem satisfies the following local optimality conditions:

The gradient (vector of first derivatives) $g(x^*) = \nabla f(x^*)$ of the objective function $f$ (projected toward the feasible region if the problem is constrained) at the point $x^*$ is zero.
The Hessian (matrix of second derivatives) $G(x^*) = \nabla ^2 f(x^*)$ of the objective function $f$ (projected toward the feasible region ${\mathcal G}$ in the constrained case) at the point $x^*$ is positive definite.

Most of the optimization algorithms in PROC NLP use iterative techniques that result in a sequence of points $x^0,\ldots ,x^ n,\ldots$ , that converges to a local solution $x^*$ . At the solution, PROC NLP performs tests to confirm that the (projected) gradient is close to zero and that the (projected) Hessian matrix is positive definite.

Karush-Kuhn-Tucker Conditions

An important tool in the analysis and design of algorithms in constrained optimization is the Lagrangian function, a linear combination of the objective function and the constraints:

$L(x,\lambda ) = f(x) - \sum _{i=1}^ m \lambda _ i c_ i(x)$

The coefficients $\lambda _ i$ are called Lagrange multipliers. This tool makes it possible to state necessary and sufficient conditions for a local minimum. The various algorithms in PROC NLP create sequences of points, each of which is closer than the previous one to satisfying these conditions.

Assuming that the functions $f$ and $c_ i$ are twice continuously differentiable, the point $x^*$ is a local minimum of the nonlinear programming problem, if there exists a vector $\lambda ^*=(\lambda _1^*,\ldots ,\lambda _ m^*)$ that meets the following conditions.

1. First-order Karush-Kuhn-Tucker conditions:

$\begin{array}{llll} c_ i(x^*) = 0 , & & & i = 1, \ldots ,m_ e \\ c_ i(x^*) \ge 0 , & \lambda _ i^* \ge 0, & \lambda _ i^* c_ i(x^*) = 0 , & i = m_ e+1, \ldots ,m \\ \nabla _ x L(x^*,\lambda ^*) = 0 & & & \end{array}$

2. Second-order conditions: Each nonzero vector $y \in {\mathcal R}^ n$ that satisfies

$y^ T \nabla _ x c_ i(x^*) = 0 \left\{ \begin{array}{l} i = 1,\ldots ,m_ e \\ \forall i\in \{ m_ e+1,\ldots ,m: \lambda _ i^* > 0 \} \end{array} \right.$

also satisfies

$y^ T \nabla _ x^2 L(x^*,\lambda ^*) y > 0$

Most of the algorithms to solve this problem attempt to find a combination of vectors $x$ and $\lambda$ for which the gradient of the Lagrangian function with respect to $x$ is zero.

Derivatives

The first- and second-order conditions of optimality are based on first and second derivatives of the objective function $f$ and the constraints $c_ i$ .

The gradient vector contains the first derivatives of the objective function $f$ with respect to the parameters $x_1,\ldots ,x_ n,$ as follows:

$g(x) = \nabla f(x) = \left( \frac{\partial f}{\partial x_ j} \right)$

The $n \times n$ symmetric Hessian matrix contains the second derivatives of the objective function $f$ with respect to the parameters $x_1,\ldots ,x_ n,$ as follows:

$G(x) = \nabla ^2 f(x) = \left( \frac{\partial ^2 f}{\partial x_ j \partial x_ k} \right)$

For least squares problems, the $m \times n$ Jacobian matrix contains the first-order derivatives of the $m$ objective functions $f_ i(x)$ with respect to the parameters $x_1,\ldots ,x_ n,$ as follows:

$J(x) = (\nabla f_1,\ldots ,\nabla f_ m) = \left( \frac{\partial f_ i}{\partial x_ j} \right)$

In the case of least squares problems, the crossproduct Jacobian

$J^ TJ = \left( {\sum _{i=1}^ m \frac{\partial f_ i}{\partial x_ j} \frac{\partial f_ i}{\partial x_ k}} \right)$

is used as an approximate Hessian matrix. It is a very good approximation of the Hessian if the residuals at the solution are “small.” (If the residuals are not sufficiently small at the solution, this approach may result in slow convergence.) The fact that it is possible to obtain Hessian approximations for this problem that do not require any computation of second derivatives means that least squares algorithms are more efficient than unconstrained optimization algorithms. Using the vector $f(x) = (f_1(x),\ldots ,f_ m(x))^ T$ of function values, PROC NLP computes the gradient $g(x)$ by

$g(x) = J^ T(x) f(x)$

The $\mi {mc} \times n$ Jacobian matrix contains the first-order derivatives of the $\mi {mc}$ nonlinear constraint functions $c_ i(x),$ $i=1,\ldots ,\mi {mc}$ , with respect to the parameters $x_1,\ldots ,x_ n$ , as follows:

$CJ(x) = (\nabla c_1,\ldots ,\nabla c_\mi {mc}) = \left( \frac{\partial c_ i}{\partial x_ j} \right)$

PROC NLP provides three ways to compute derivatives:

It computes analytical first- and second-order derivatives of the objective function $f$ with respect to the $n$ variables $x_ j$ .
It computes first- and second-order finite-difference approximations to the derivatives. For more information, see the section Finite-Difference Approximations of Derivatives.
The user supplies formulas for analytical or numerical first- and second-order derivatives of the objective function in the GRADIENT, JACOBIAN, CRPJAC, and HESSIAN statements. The JACNLC statement can be used to specify the derivatives for the nonlinear constraints.