AI & ML in Finance - Lecture 8 - Simple Linear Regression

Hello, everyone, and welcome back to our class in Artificial Intelligence and Machine Learning in Finance. And after having talked a lot about data sources, data generation, and our introduction to the realm of ML, we've now finally arrived at the applications in finance. And we'll go through all these topics by, first of all, describing and discussing the statistical methods, and then highlighting the applications in finance. And maybe a little bit surprising, we'll start with simple linear regression, multiple linear regression, because these are the building blocks for more sophisticated models in statistical learning. We'll start our discussion now of these different AI and ML methods. It's a little bit heavy on the machine learning methods, but we'll also see some artificial intelligence methods, and we'll discuss the statistical background, and then the application. Now, all of these methods are examples of statistical learning algorithms, and statistical learning refers to the set of tools used for understanding data and explaining behavior of statistical data. And broadly speaking, we can distinguish two types of models in statistical learning. The first one are supervised algorithms, and in contrast to supervised one, we also have unsupervised algorithms. What are the differences? Supervised statistical learning means that we estimate, or say we predict an output based on input. Very simple example, you want to predict the wage of workers based on their education, their gender, and we would get what we call a regression problem. The second problem is related to this, but it's called a classification problem, because in this case, the outcome variable, the output, is a binary variable. It's a dummy variable that takes on values 1 and 0. For example, if we only want to predict ups and downs of the S&P 500, if we now use, for example, macroeconomic variables, x1, x2, et cetera, and we want to estimate the ups and downs, 1s and 0s, we have a classification problem. Again, same here as with the credit card data, for example, we want to forecast default, no default or default 1 and 0. That's a classification problem. In an unsupervised statistical learning algorithm, we want to group input observations with no output variable. So the example we have here is a clustering problem. We have data on customers, say age, income, education, and we want to group them to see certain clusters of customers which share common properties. For example, one group are the middle-aged men. A second group are young women. Third group is maybe old persons with low income, and we get different clusters. And this is an unsupervised statistical learning problem, because as you can see, we have no output. We have no outcome variable y. We want to predict, and in this case, we have no way to see how these different clusters are different when it comes to an output variable. They only differ based on their inputs. This is a very short introduction. Let's now turn to simple linear regression, which you probably know from your introduction to statistics. Now, linear regression is the foundation for many modern supervised statistical learning approaches, so we are now looking at supervised algorithms, meaning that we have an outcome. And in the setting of simple linear regression, we want to predict a quantitative response or a metric response y on the basis of a single predictor variable x, which means that we have a regression function y, and y is approximated by two parameters, beta 0 and beta 1, and we assume a linear relation between x and y, meaning that y is almost equal to beta 0 plus beta 1 times x. We say that y is regressed on x. This is a little bit different in German. In German, actually, we switch y and x, and we say that x is regressed on y, but it's y that is regressed on x. We have two unknown parameters or coefficients, beta 0, which is called the intercept, and beta 1, which is called the slope. Those are obviously the intercept and slope of the linear regression function. It's a line, and they are estimated by minimizing the residual sum of squares or RSS. What we get as a result is we have an estimate for beta 0, beta 0 hat, and an estimate for beta 1, beta 1 hat, and if we now have input data x, we can use those estimated parameters, enter x, and we get an estimate for y, y hat. Now, how should we estimate this regression function, this linear line? Very simply by minimizing the residual sum of squares. So we'll start with the prediction y i hat. We have a certain number of observations with x i, and we look at this regression function, and we have those estimates, y, our predictions, y i hat. We also have those observations, y i, and by comparing y i and y i hat, we get what we call the i residual. That's the error. That's also why it's called e. It's the error between our prediction, y i hat, and the actual observation, y i, and these are the different residuals from our estimated linear regression function. If we now square those errors to make sure that a negative and a positive error don't cancel each other out, if we square those errors and we sum them up, we get RSS in equation four, which is the residual sum of squares. It's the sum of the squared errors. Now, here you can see one very simple example based on the advertising data summed from the James Whitten-Hasty and Tip Shirani textbook on statistical learning. Now, you see the blue line is the regression line. We have TV and sales, TV being the predictor and sales being our outcome variable, and you see all those different red dots. Those are the estimates, and actually the actual observations, and the blue line is our estimated regression line, and those small gray lines, they highlight the errors, and we now want to choose the blue line in such a way that the sum of the square gray lines becomes minimum. This is the RSS. So how should we estimate this? Well, if you remember your statistics classes, minimization of the RSS in this case of simple linear regression leads to the well-known OLS estimator, the ordinary least squares estimator for beta one and beta zero, X dash and Y dash are the sample means of X and Y. So you take your observations, XI, YI, you take the sample means, enter them into these two equations, five and six, and you get the OLS estimates for beta zero and beta one. Actually, this is done on most modern calculators. They usually have functions for simple linear regression. Okay. Now here, in this case, again, the response variable is sales, the predictor is TV advertising budget, and as you can see here, we have beta zero and beta one on the X and Y axis, and the red dot here, this is a contour plot of the RSS, and as you can see here, this is actually the minimum, so it's very simple to find minimal, this is the minimal value of the RSS for beta zero and beta one. Okay, and this is the three-dimensional plot of the RSS. You can see here, it's a very smooth function with beta one and beta zero on the X and Y axis, and on the Z axis in the three-dimensional case. Here, you have the minimum. Okay. You have resulting OLS regressions, and we can do this, in this case, if we have three predictors, we can use three estimations and look at TV advertising budget, radio advertising budget, and newspaper advertising budget, and we can already see, there seems to be a strong linear relation between TV advertising and sales. It looks a little less linear for radio advertising and sales, and it's probably not a linear relation here, the newspaper advertising and sales, but these are three distinct regression analyses. Again, we are in the case of a simple linear regression, so meaning that we are looking at X, actually Y, the outcome variable, Y being estimated and regressed on X1 in the first plot, and Y being regressed on X2 in the second plot, and on X3 in the third plot, so it's not a multivariate linear regression. It's still simple linear regression. As you remember, the statistics classes, you probably know that we need to look for measures of the accuracy of the coefficient estimates. We need to assess the goodness of fit of our model, and if we assume that we have a regression function, Y being equal to beta zero plus beta one times X plus an error, how should we assess the accuracy of those coefficient estimates, beta zero hat and beta one hat? Well, we have to look at the standard errors of the OLS coefficients of the estimated parameters, because they are estimated from a sample. We do have standard errors, and the standard errors are given by equations seven and eight. As you can see, what we need is, we need sample mean of X, and we also need the variance of epsilon, which is the variance of the residual terms, the error term. Okay. Now, how to estimate sigma squared, because this is the only thing we do not know from our sample. We know X bar, and we know N, the number of observations. The only thing that is left unknown is sigma squared, the variance of the residual terms, and we estimate sigma via the residual standard error. We do not know how sigma looks in the population, but we can estimate sigma based on our sample and on the sample errors. So we take the residual standard error, RSE, and we take the residual sum of squares, divided by N minus two, take the square root, and we get an estimate for the residual standard error. So that's our estimate. And what we do next, we put it into the equation for the standard equation, a standard error of the coefficient estimates, and then we get the standard error of our two coefficients. But what should we do with this? Well, if we have the standard error of beta zero hat and beta one hat, we can actually construct confidence intervals for beta zero and beta one. So the 95% confidence intervals for those two parameter estimates are given by more or less beta zero plus and minus two times the standard error of those coefficients, and we get confidence intervals. And how does this help us? Well, actually, later on, we will look at significance tests, and significance tests usually work like this. You have a hypothesized parameter value. Let's say we want to test the hypothesis that beta one is zero. There's no relation between X and Y, meaning that the coefficient is zero. So we take zero plus minus two times the standard error, and we get a confidence interval. If now our parameter estimate is in this confidence interval, with 95% probability, the parameter is not significantly different from zeros, meaning that we have to reject the, we have to, cannot reject the hypothesis that beta one is zero, and there's no linear relation. So we can use confidence interval for significance testing. This is shown here. For example, the hypothesis is beta one is zero, where this versus H one, beta one is not equal to zero. And now we take the standard error of beta one hat, and we use this t-statistic, beta one hat minus zero, divided by the standard equation. So the t-statistic measures the number of standard deviations that our parameter estimate is far away from zero. If it's far away from zero, there's a likelihood that actually the parameter is not equal to zero. If it's close to zero, then actually we have to reject H one. Now, this is what we do for the parameter estimates, but how should we assess the accuracy of the model itself? Assuming that we rejected the hypothesis beta one is equal to zero. So we cannot reject a linear relation between X and Y, but how can we assess the accuracy of the model itself? We can take the residual standard error and recall that associated with each individual observation is an error term, EI, or epsilon in the population. And even if the coefficients were known, these error terms still exist, and this is why it's not a perfect linear line. If it were a perfect linear line, there would be no stochastic behavior in the data, and we wouldn't need a regression analysis in the first place. But assume that we have a linear relation between X and Y, and the residual term exists, and it's noise simply added to our linear regression. Even then, we do have these error terms, EI, and these error terms prevent us from perfectly predicting Y from X. And thus, we estimate the standard error of epsilon via the residual standard error, which is given by the residual sum of squares divided by N minus two, square root. And then, how should we assess this? Well, the RSE provides an absolute measure of lack of fit of the model, and it is given in terms of units of Y. Problem is, the RSE depends on the scale of Y. So the solution in this case is, we scale it. We take R squared, which you probably know from your statistics introduction, and R squared is the proportion of variance explained by the model. So R squared is equal to one minus the RSS, divided by TSS, which is the total sum of squares, simply YI minus Y bar squared, and then sum it all up for observations. And this gives us the total sum of squares, the total variance, and one minus the residual sum of squares divided by the total sum of squares gives you the proportion of variance explained by our simple linear regression model. Also note, R squared is equal, in this case, to the squared correlation between X and Y. So actually, this is why it's already called R squared. It's the squared correlation between X and Y. Okay, so this is a very short introduction to simple linear regression. We'll come back to this in some instances, but obviously we want to step our game up to multiple linear regression, and we'll look at this in the next video.