Linear Regression

The link posted by Simon in a comment is really informative. However, for linear regression, minimizing the sum of squared residuals (OLS) still is big, especially in causal modelling. As far as I know, many programmes (R, Stata) still use the (X‘X)^-1 X‘y way to solve linear regression problems.

Often advanced algorithms have a hard time beating OLS. I think OLS often is a good (and fast) benchmark. Also the coeficients (betas) are easy to understand. So you can learn a lot about the problem you have at hand. This can be extremely helpful.

If you have the time, have a look at „Introductory Econometrics: A Modern Approach“ by J. Wooldridge. Also „An Introduction to Statistical Learning with Applications in R“ is a good source.

If you are brave, go for Russell Davidson and James G. MacKinnon: Econometric Theory and Methods.

I see that gradient decent is important. However, having a sound understanding of basic statistics will pay off massively in the long run in my opinion.

The example that you have given is a very trivial case of linear regression but it can still lead to computational problems. Imagine we have sensor inputs and we want to estimate the temperature with them. Imagine that the sensor gives us $100$ measurements per second. What will happen with your parameter estimations (especially the sums) when your system is running for one week?

As you can see the problem of this procedure is that we have to carry out all the computations at each time step. for the general linear regression

$$y_i = boldsymbol{w}^Tboldsymbol{x}_i + varepsilon_i$$

we can determine least squares estimate $hat{boldsymbol{w}}$ for the weights $boldsymbol{w}$ by the following formula

$$hat{boldsymbol{w}}=left[boldsymbol{X}^Tboldsymbol{X} right]^{-1}boldsymbol{X}^Tboldsymbol{y},$$

in which $boldsymbol{X}$ is the data matrix and $boldsymbol{y}$ is the vector of the outputs of all observations. As you can see with an increasing amount of data the data matrix $boldsymbol{X}$ will get an increasing number of rows. Additionally, we have to invert the product which will become more difficult if have a very high dimensional input vector $boldsymbol{x}$. The direct calculation is very nice from the theoretical point of view but very demanding from the practical point of view.

In order to prevent these huge calculations, different recursive methods like the least mean squares (LMS) algorithm or the recursive least squares (RLS) algorithm was invented to prevent such demanding calculations.

In order to see the difference between the direct and the recursive calculations look at the continuous mean calculation

$$bar{x}_N = dfrac{1}{N}sum_{n=1}^Nx_n$$ $$text{direct: }bar{x}_{N+1}=dfrac{1}{N+1}sum_{n=1}^{N+1}x_n$$ $$=dfrac{N}{N+1}left[dfrac{1}{N}sum_{n=1}^{N+1}x_nright]$$
$$=dfrac{N}{N+1}left[bar{x}_N+dfrac{1}{N}x_{N+1}right]$$ $$implies text{recursive: }bar{x}_{N+1}=dfrac{N}{N+1}bar{x}_N+dfrac{1}{N+1}x_{N+1}.$$

If you already calculated the previous mean it is clear that calculating the for the next time step is much easier to calculate with the recursive formula.