Regular Inverses

The right inverse of a matrix is the matrix that yields the identity when is postmultiplied by it. Similarly, the left inverse of yields the identity if is premultiplied by it. is said to be invertible and is said to be the inverse of , if is its right and left inverse, . This requires to be square and nonsingular. The inverse of a matrix is commonly denoted as . The following results are useful in manipulating inverse matrices (assuming both and are invertible):

If is a diagonal matrix with nonzero entries on the diagonal—that is, —then . If is a block-diagonal matrix whose blocks are invertible, then

In statistical applications the following two results are particularly important, because they can significantly reduce the computational burden in working with inverse matrices.

Partitioned Matrix

Suppose is a nonsingular matrix that is partitioned as

Then, provided that all the inverses exist, the inverse of is given by

where , , , and .

Patterned Sum

Suppose is nonsingular, is nonsingular, and and are and matrices, respectively. Then the inverse of is given by

This formula is particularly useful if and has a simple form that is easy to invert. This case arises, for example, in mixed models where might be a diagonal or block-diagonal matrix, and .

Another situation where this formula plays a critical role is in the computation of regression diagnostics, such as in determining the effect of removing an observation from the analysis. Suppose that represents the crossproduct matrix in the linear model . If is the th row of the matrix, then is the crossproduct matrix in the same model with the th observation removed. Identifying , , and in the preceding inversion formula, you can obtain the expression for the inverse of the crossproduct matrix:

This expression for the inverse of the reduced data crossproduct matrix enables you to compute "leave-one-out" deletion diagnostics in linear models without refitting the model.

Generalized Inverse Matrices

If is rectangular (not square) or singular, then it is not invertible and the matrix does not exist. Suppose you want to find a solution to simultaneous linear equations of the form

If is square and nonsingular, then the unique solution is . In statistical applications, the case where is rectangular is less important than the case where is a square matrix of rank less than . For example, the normal equations in ordinary least squares (OLS) estimation in the model are

A generalized inverse matrix is a matrix such that is a solution to the linear system. In the OLS example, a solution can be found as , where is a generalized inverse of .

The following four conditions are often associated with generalized inverses. For the square or rectangular matrix there exist matrices that satisfy

The matrix that satisfies all four conditions is unique and is called the Moore-Penrose inverse, after the first published work on generalized inverses by Moore (1920) and the subsequent definition by Penrose (1955). Only the first condition is required, however, to provide a solution to the linear system above.

Pringle and Rayner (1971) introduced a numbering system to distinguish between different types of generalized inverses. A matrix that satisfies only condition (i) is a -inverse. The -inverse satistifes conditions (i) and (ii). It is also called a reflexive generalized inverse. Matrices satisfying conditions (i)–(iii) or conditions (i), (ii), and (iv) are -inverses. Note that a matrix that satisfies the first three conditions is a right generalized inverse, and a matrix that satisfies conditions (i), (ii), and (iv) is a left generalized inverse. For example, if is of rank , then is a left generalized inverse of . The notation -inverse for the Moore-Penrose inverse, satisfying conditions (i)–(iv), is often used by extension, but note that Pringle and Rayner (1971) do not use it; rather, they call such a matrix "the" generalized inverse.

If the matrix is rank-deficient—that is, —then the system of equations

does not have a unique solution. A particular solution depends on the choice of the generalized inverse. However, some aspects of the statistical inference are invariant to the choice of the generalized inverse. If is a generalized inverse of , then is invariant to the choice of . This result comes into play, for example, when you are computing predictions in an OLS model with a rank-deficient matrix, since it implies that the predicted values

are invariant to the choice of .

LDU, LU, and Cholesky Decomposition

Every square matrix , whether it is positive definite or not, can be expressed in the form , where is a unit lower-triangular matrix, is a diagonal matrix, and is a unit upper-triangular matrix. (The diagonal elements of a unit triangular matrix are 1.) Because of the arrangement of the matrices, the decomposition is called the LDU decomposition. Since you can absorb the diagonal matrix into the triangular matrices, the decomposition

is also referred to as the LU decomposition of .

If the matrix is positive definite, then the diagonal elements of are positive and the LDU decomposition is unique. Furthermore, we can add more specificity to this result in that for a symmetric, positive definite matrix, there is a unique decomposition , where is unit upper-triangular and is diagonal with positive elements. Absorbing the square root of into , , the decomposition is known as the Cholesky decomposition of a positive-definite matrix:

where is upper triangular.

If is symmetric nonnegative definite of rank , then we can extend the Cholesky decomposition as follows. Let denote the lower-triangular matrix such that

Then .

Spectral Decomposition

Suppose that is an symmetric matrix. Then there exists an orthogonal matrix and a diagonal matrix such that . Of particular importance is the case where the orthogonal matrix is also orthonormal—that is, its column vectors have unit norm. Denote this orthonormal matrix as . Then the corresponding diagonal matrix—, say—contains the eigenvalues of . The spectral decomposition of can be written as

where denotes the th column vector of . The right-side expression decomposes into a sum of rank-1 matrices, and the weight of each contribution is equal to the eigenvalue associated with the th eigenvector. The sum furthermore emphasizes that the rank of is equal to the number of nonzero eigenvalues.

Harville (1997, p. 538) refers to the spectral decomposition of as the decomposition that takes the previous sum one step further and accumulates contributions associated with the distinct eigenvalues. If are the distinct eigenvalues and , where the sum is taken over the set of columns for which , then

You can employ the spectral decomposition of a nonnegative definite symmetric matrix to form a "square-root" matrix of . Suppose that is the diagonal matrix containing the square roots of the . Then is a square-root matrix of in the sense that , because

Generating the Moore-Penrose inverse of a matrix based on the spectral decomposition is also simple. Denote as the diagonal matrix with typical element

Then the matrix is the Moore-Penrose (-generalized) inverse of .

Singular-Value Decomposition

The singular-value decomposition is related to the spectral decomposition of a matrix, but it is more general. The singular-value decomposition can be applied to any matrix. Let be an matrix of rank . Then there exist orthogonal matrices and of order and , respectively, and a diagonal matrix such that

where is a diagonal matrix of order . The diagonal elements of are strictly positive. As with the spectral decomposition, this result can be written as a decomposition of into a weighted sum of rank-1 matrices

The scalars are called the singular values of the matrix . They are the positive square roots of the nonzero eigenvalues of the matrix . If the singular-value decomposition is applied to a symmetric, nonnegative definite matrix , then the singular values are the nonzero eigenvalues of and the singular-value decomposition is the same as the spectral decomposition.

As with the spectral decomposition, you can use the results of the singular-value decomposition to generate the Moore-Penrose inverse of a matrix. If is with singular-value decomposition , and if is a diagonal matrix with typical element

then is the -generalized inverse of .