← Back to blog

Least Squares Estimation / OLS

When plotting samples onto a graph using a scatter plot, it is visually telling that not all samples fall under one linear line. When plotting a regression, the equation $y_{i} = \beta_{0} + \beta_{1}x_i + \epsilon_{i}$ shows a linear relationship between independent variables, but introduces two beta elements and epsilon to show that the plots do not fall under one line without deviation.

Beta zero and Beta one are known as population parameters, while their true values are unknown, least squares/OLS aims to estimate their true values by minimizing the error (epsilon) of our data set. Rather than taking just the minimum of epsilon, we take the minimum of epsilon squared, this ensures negative values of epsilon do not cancel out positives.

Equation and Derivation of Betas

We start with the following equation which we will aim to minimize

$$\epsilon^2 =(y_{i} - \beta_{0} -\beta_{1}x_{i})^2$$

Since each data point has its own residual (epsilon), we sum all values of epsilon together

$$\sum_{i=1}^n \epsilon_i^2 = \sum_{i=1}^n ({y_{i} - \beta_{0} -\beta_{1}x_{i}}) ^2$$

Now that we have set up epsilon as a non-negative sum of all data points, we can use the standard way of minimizing variables to zero, taking the derivative. However, we have two different betas to derive the equations to, so we'll go one by one.

Deriving Beta Zero

$$\frac{d}{d\beta_{0}} \left(\sum_{i=1}^n \epsilon_i^2 = \sum_{i=1}^n ({y_{i} - \beta_{0} -\beta_{1}x_{i}}) ^2 \right)\implies 0 = \frac{d}{d\beta_{0}} \sum_{i=1}^n ({y_{i} - \beta_{0} -\beta_{1}x_{i}}) ^2$$ $$0 = -2\sum_{i=1}^n(y_{i}-\beta_{0}-\beta_{1}x_{i}) \implies 0 = \sum_{i=1}^n y_{i} - n\beta_{0} - \beta_{1}\sum_{i=1}^n x_{i}$$ $$n\beta_{0} = \sum_{i=1}^n y_{i} - \beta_{1}\sum_{i=1}^n x_{i}$$

And we get our final formula,

$$\beta_{0} = \bar{y} - \beta_{1}\bar{x}$$

Deriving Beta One

Using our derivation of beta zero, we can derive beta one, starting with the same equation.

$$\frac{d}{d\beta_{1}} \left(\sum_{i=1}^n \epsilon_i^2 = \sum_{i=1}^n ({y_{i} - \beta_{0} -\beta_{1}x_{i}}) ^2 \right)\implies 0 = \frac{d}{d\beta_{1}} \sum_{i=1}^n ({y_{i} - \beta_{0} -\beta_{1}x_{i}}) ^2$$ $$0 = \sum_{i=1}^n x_{i}(y_{i}-\beta_{0}-\beta_{1}x_{i})$$ $$0 = \sum_{i=1}^n x_{i}y_{i} - \beta_{0}\sum_{i=1}^n x_{i} - \beta_{1}\sum_{i=1}^n x_{i}^2$$

Substitution of $\beta_{0}$

$$0 = \sum_{i=1}^n x_{i}y_{i} - (\bar{y} - \beta_{1}\bar{x})\sum_{i=1}^n x_{i} - \beta_{1}\sum_{i=1}^n x_{i}^2$$ $$0 = \sum_{i=1}^n x_{i}y_{i} - \bar{y}\sum_{i=1}^n x_{i} +\beta_{1}\bar{x}\sum_{i=1}^n x_{i} - \beta_{1}\sum_{i=1}^n x_{i}^2$$

Use two properties to simplify this further: $\sum_{i=1}^n x_{i} = n \bar{x}$ and combining $\beta_{1}$ like terms

$$0 = \sum_{i=1}^n x_{i}y_{i} - n\bar{y}\bar{x} + \beta_{1} (n\bar{x}^2 - \sum_{i=1}^n x_{i}^2)$$ $$\implies \sum_{i=1}^n x_{i}y_{i} - n \bar{y} \bar{x} = \beta_{1}(\sum_{i=1}^n x_{i}^2 - n \bar{x}^2)$$ $$\implies \beta_{1} = \frac{\sum_{i=1}^n x_{i}y_{i} - n\bar{x}\bar{y}}{\sum_{i=1}^n x_{i}^2 - n\bar{x}^2}$$

Application

Given sets of data points, we can use our derived equations of beta zero and beta one to solve the fitted regression line. To do this, we use the same equation defined earlier, but with a removed epsilon (since it's minimized away)

$$\hat{y} = \hat{\beta}_{0} + \hat{\beta}_{1}x$$

The hats indicate that values are estimated from the provided data, rather than true parameters.

Simple Linear Regression

Given data points: $(1, 2), (2, 4), (3, 5), (4, 7)$

Calculate sample means:

$$\bar{x} = \frac{1+2+3+4}{4} = 2.5, \quad \bar{y} = \frac{2+4+5+7}{4} = 4.5$$

Calculate $\hat{\beta}_1$:

$$\hat{\beta}_1 = \frac{(1-2.5)(2-4.5) + (2-2.5)(4-4.5) + (3-2.5)(5-4.5) + (4-2.5)(7-4.5)}{(1-2.5)^2 + (2-2.5)^2 + (3-2.5)^2 + (4-2.5)^2}$$ $$= \frac{3.75 + 0.25 + 0.25 + 3.75}{2.25 + 0.25 + 0.25 + 2.25} = \frac{8}{5} = 1.6$$

Calculate $\hat{\beta}_0$:

$$\hat{\beta}_0 = 4.5 - 1.6(2.5) = 0.5$$

The fitted regression line is: $\hat{y} = 0.5 + 1.6x$

---

Maximum Likelihood Estimation

Maximum likelihood estimation aims to estimate values of the mean and variance of a random variable that maximizes the likelihood of the observed data. The log-likelihood function is the log of the product across all observable data given a parameter. This allows you to find the "best value" of that parameter.

Since we're finding the maximum of the likelihood function, we take the derivative of it. Rather than being an upward opening parabola (like OLS), log-likelihood functions are concave (curve downward) so the critical point is the maximum.

Equation and Derivation of the Maximum Log-Likelihood

Since the likelihood function of any random variable can be found, we'll cover the derivation of the normal distribution. However, the same steps can be applied across random variables.

The mean, $\mu$

$$\mathcal{L}(\mu | x_1,...,x_n) = \prod_{i=1}^n \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}\left(\frac{x_i-\mu}{\sigma}\right)^2} = (2\pi\sigma^2)^{-\frac{n}{2}} \cdot e^{-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i - \mu)^2}$$ $$\ln(\mathcal{L}) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n(x_i - \mu)^2$$

Take the derivative with respect to $\mu$ and set equal to zero:

$$\frac{d\ln(\mathcal{L})}{d\mu} = \frac{1}{\sigma^2}\sum_{i=1}^n(x_i - \mu) = 0$$ $$\sum_{i=1}^n(x_i - \mu) = 0$$ $$\hat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i = \bar{x}$$

The variance, $\sigma^2$

$$\frac{d\ln(\mathcal{L})}{d\sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^n(x_i - \mu)^2$$

Set equal to zero and solve:

$$-\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^n(x_i - \mu)^2 = 0$$ $$\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n(x_i - \bar{x})^2$$

Bias of MLE and Bessel's Correction

The MLE estimate $\bar{x}$ was calculated to be as close to the data points as possible, which makes deviations systematically smaller than they would be from the true $\mu$. By using the same data to estimate mean and variance, we lose a degree of freedom in understanding the true value of variance.

Rather than using $\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n(x_i - \bar{x})^2$, we apply Bessel's correction, which simply divides by $n-1$ instead of $n$:

$$\hat{\sigma}^2 = \frac{1}{n-1}\sum_{i=1}^n(x_i - \bar{x})^2$$

---

Method of Moments

The simplest of all, Method of Moments estimates parameters by equating sample moments to theoretical population moments.

Mean

For the mean, we set the first sample moment equal to the first theoretical moment.

$$E[X] = \mu$$ $$\hat{\mu} = \bar{x} = \frac{1}{n}\sum_{i=1}^n x_{i}$$

Variance

The second moment about the mean gives variance:

$$E[(X-\mu)^2] = \sigma^2$$

Through this we can find the Method of Moments estimator

$$\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n(x_{i}-\bar{x})^2$$

Note: This estimator is biased for the same reason as MLE — it divides by $n$ and uses $\bar{x}$ instead of the true $\mu$. The unbiased version uses Bessel's correction ($n-1$).