Maximum Likelihood Estimation

To estimate the parameters in a linear model with mean function by maximum likelihood, you need to specify the distribution of the response vector . In the linear model with a continuous response variable, it is commonly assumed that the response is normally distributed. In that case, the estimation problem is completely defined by specifying the mean and variance of in addition to the normality assumption. The model can be written as , where the notation indicates a multivariate normal distribution with mean vector and variance matrix . The log likelihood for then can be written as

This function is maximized in when the sum of squares is minimized. The maximum likelihood estimator of is thus identical to the ordinary least squares estimator. To maximize with respect to , note that

Hence the MLE of is the estimator

This is a biased estimator of , with a bias that decreases with .