Maximum Likelihood Estimation

To estimate the parameters in a linear model with mean function by maximum likelihood, you need to specify the distribution of the response vector . In the linear model with a continuous response variable, it is commonly assumed that the response is normally distributed. In that case, the estimation problem is completely defined by specifying the mean and variance of in addition to the normality assumption. The model can be written as , where the notation indicates a multivariate normal distribution with mean vector and variance matrix . The log likelihood for then can be written as

     

This function is maximized in when the sum of squares is minimized. The maximum likelihood estimator of is thus identical to the ordinary least squares estimator. To maximize with respect to , note that

     

Hence the MLE of is the estimator

     
     

This is a biased estimator of , with a bias that decreases with .