Lecture 19 : Regression Modelling-Part 1

[Music] in this lesson we'll discuss the theoretical endings behind the regression algorithm we'll start the discussion with the background and motivation to regression modeling technique next we'll discuss the types of data employed in the regression modeling we'll also discuss in great detail the cause function that is ordinary Lee Square method which is widely employed to fit the regression line next we'll discuss the simple and multiple regression modeling techniques next we'll examine the key assumptions behind the classical linear regression model and implications if these assumptions are violated we'll also discuss the blue that is best linear unbiased estimator properties associated with the oess estimators lastly we'll briefly cover the role of normal distribution in hypothesis testing associated with regression modeling we'll also discuss some nonlinear functional forms that can be suitably modified to linear functional forms in this video we'll provide the background and motivation for the application of machine learning algorithms in business applications many times on Amazon and Netflix we get recommendations for movies and TV series as if some human person has examined and carefully vetted our history of TV or M movie watching and provided these recommendations they are extremely accurate also you would have observed the spam filtering application on your mobile text messages and email applications if you carefully examine these filtered messages you would find almost 95% to 99% accurate messages and emails that are like fishing emails or bulk males which are targeted to bulk audience and not not very genuine mails or messages also if you go to website like Google News you will find news items clustered and put together in groups as if somebody has carefully read these news items and put together similar news items putting together these news items for our ease of reading to summarize this discussion in modern day business applications for example Amazon and Netflix movie recommendations filtering out spams medical prognosis based on previous health records algorithmic trading crate scoring models these applications appear or are driven by computers without being explicitly programmed so here computers are learning without being explicitly programmed for example in recognizing H handwriting natural language processing and so many applications in modern day-to-day and business applications where computers are acting as if some human being is working behind the curtains but obviously given the volume volume and nature of data Big Data being handled a human being cannot done cannot do these things on it on their own and therefore we need or we clearly understand that these are computers working behind the scenes without being explicitly programmed they are made to learn through these machine learning algorithms in this video we'll provide a brief background to machine learning algorithms employed in business applications and finance let us start with a very interesting supervisor learning problem in supervised learning we have a set of input features which are maap to Output labels now please note the critical the important keyword here is that output is labeled so a set of features input features and output which is labeled the algorithm is trained using this input output data and once the algorithm is trained it is employed in future using these input variables or features to predict the output a very simple example to this is house price prediction problem where a set of data that is size of house in square feet which is one of the features and the label that is house price is given and using this data the algorithm is trained this is called regression problem where we try to model this using regression we'll discuss in this lesson we'll discuss the regression problem in great detail but as a simple example using this feature which is the size and output label which is the house price the regression model or algorithm is developed trained and for future when the size of a house is given the algorithm is employed to predict its price let's say some new points are given then this algorithm will predict their price by fit a regression line which we'll discuss in detail in this lesson another interesting supervised learning problem is classification problem where a certain label data is given along with features and then algorithm is trained to classify the new observations across these labels for example credit default scoring problem where we try to classify observations such as loan applications into defaulters for example here defaulters are given in red and non-defaulters or good credit borrowers are given in blue now we have two features the labels are defaulter and non- defaulters while features are net worth and income Now using the given set of data we train this algorithm we train this algorithm and once we have the algorithm trained then as the new data comes in for example new data Maybe here here using this classification problem we try to classify the new observations into defaulter versus non- defaulter using this model we try to predict that we try to classify or segregate defaulters Vis A non defaulters using the algorithm that has been trained in the unsupervised learning the key difference is that the data is not labeled only we have only features for example you may have demographic variables such as taste income age and gender and based on these features you will try to segregate individuals into different groups and this is a very useful application in Market segmentation by consulting firms who employ such data in creating Target segments for targeting them for their products such segments are based on this classification Pro clustering problem where we try to Cluster individuals based on certain features such as taste income gender and so on so this is a very interesting business problem clustering problem of Market segmentation another very interesting clustering problem that we already discussed is clustering of news items depending upon their similarity so uh natural language processing and text processing text analytics applications and algorithms are employed to segregate news items into clusters clusters that are similar based on certain features maybe genre of news or similarity in terms of geography or the topic of news they are clustered together by unsupervised learning clust algorithms that cluster the news items together to summarize this video we discuss supervised and unsupervised learning algorithms in the supervised learning algorithm we have data which is labeled along with the features and we try to develop the algorithm by mapping the relationship between input features and labeled output subsequently this trained algorithm is employed to predict future observations or future units using their features we try to predict their output labels in contrast in the unsupervised learning algorithm in the unsupervised learning algorithm we have only features not labels and we try to map or cluster the observations or units together based on similarity in features and one of the very important and interesting application we saw was that of Market segmentation in this video we'll discuss the types of data being employed with machine learning algorithms to train and test the models in financial markets and finance domain in general in general there are three broad types of data that are employed in finance domain first is observations about multiple individuals or units collected over a single period second observations about a single individual collected over multiple periods and lastly observations about multiple individuals or units collected over various time periods we'll examine this and visualize this types of data in subsequent discussion have a look at this diagram here we have three cross-sections C1 at Time 1 C 2 at time T2 and C3 at time T3 notice if you collect only one individual data about one individual like a over all the three cross-sections that is 1 T1 T2 and T T3 then this kind of data is called time series data you are collecting a Time series of information about individual or Unit A in contrast if you have six individuals or more or less individuals Like A B C D E and F and you collect all these individuals over a particular time let's say T1 then it is called cross-sectional data because you are collecting all the information about these units for a particular time T1 so it is a cross-sectional data lastly if you combine the time series and cross-sectional characteristics that is you collect the information about all the units or individual of interest that is for example A B C D E and F not only for one crosssection but all the desired cross-sections like tal to T1 for time tal to T2 and time tal to T3 then this becomes a panel or longitudinal data that means this panel and longitudinal data comprises or involves the characteristics of Time series as well as cross-section to summarize this video we discuss three types of data first time series data where we collected information about individual unit over a given time series next we discuss cross-sectional data where we collected and analyzed data about multiple individuals or units like a bcde e f over a particular cross-section of time and lastly we discussed panel or longitudinal data wherein we combined the characteristics of Time series and cross-section we collected the information about all the units or individuals of Interest over multiple time periods and we call this panel or longitudinal data panel or longitudinal data in this video we'll introduce simple linear regression model consider a simple linear regression model the way it is presented here yal to Beta plus beta1 X Plus mu this is often referred to as two variable linear regression model or by variate regression model because there are two variables Y and X Y here is the dependent variable or explained or response or predicted or Regent variable while X here is the independent explanatory predictor or regressor variable notice there's an error term or also called mu which is error term or residual term or disturbance term this mu represents the aggregation of those factors that are not explained by why so in this model we are trying to explain the effect of this part this part on why those factors for which we cannot explain or are not part of this model are aggregated or considered to be part of this error term this these are unobserved factors that reflect variables or factors other than x that affect y since mu here is also a random or stochastic variable it is considered that it has some kind of probabilistic distribution often normal distribution is employed to model this error term mu but the idea here is that this is random in nature and it assumes its values assume some kind of probabalistic distribution beta not here is the constant term while beta 1 is called the slope term this is so because this relationship this model can be represented on a two-dimensional space XY space where this model is basically a straight line with this intercept or constant term as beta not and slope of this line is represented as beta 1 so this line represents in visual terms this model so this is a visualization of this model this simple model aims to study the dependence of Y on X that means how X affects y now please note here that regression deals with the dependence of one variable over another it does not imply a simply Cession the idea is that it only establishes the statistical strength of the relation the causation has to be established a PRI by theory for example if instead of Y like this the dependence on y you replace and x with Y and you put it like this even then the regression model the mathematical modeling will give you some kind of statal relationship which obviously may be spous in nature take for example crop and rain now a we already know that crop affects rain sorry rain affects crop my bad rain affects crop this is established by our common understanding Common Sense however even if you put or replace rain and with crop and you try to find some kind of mathematical relationship from crop to rain the regression model the mathematical model will give you that relationship which is obviously purus in nature and therefore a PRI theoretical considerations are needed to to imply the causation now in regression analysis the dependent variable which is why is considered random or stochastic in nature that means it assumes some kind of probability distribution the reason is that it also incorporates the effect of all those unexplained or not considered factor which are aggregated in the error term mu which was stochastic in nature so those VAR those effects or influences will also be incorporated in by and therefore by is random or stochastic in nature in contrast the explanatory variable which is X is assumed to have fixed value that means it is considered to be coming from outside While y is part of this model and therefore it is random in nature X is fixed and taken from outside so the X values are fixed and taken from outside in the model A closely associated concept here is of correlation correlation here establishes the degree of linear relationship between the two variables let's say X1 and X2 in correlation analysis both variables are treated in a similar Manner and considered to be random that means no causality from one to other is implied let's say for example if variable X1 moves by 1% then X2 moves by in the same direction by let's say 0.5% then there's a 50% correlation between these two variables however we do not imply any we do not imply any causality here to summarize in this video We examined a simple linear regression model by variate model where there only two variables the dependent variable y a constant term independent variable X and a slope or coefficient term beta 1 along with the error term y here is the dependent variable which reflects the impact of our model which is beta not plus beta 1 x while those effects or influences which are not accounted for by this model are considered to be aggregated into this error term now please note by definition of this model V here is random and stochastic in nature because it also incorporates the effect of these mu or unexplained part of the model the explained or modeled part this here the variable X is considered to be fixed in nature and taken from outside While y is getting affected by this model itself which includes the effect of mu and mu being random stochastic in nature vse is also random and stochastic in nature this relationship while it is mathematical in nature and statistical significance established through mathematics the theoretical underpinnings or causality from X to Y has to be established a Pari from Theory not the mathematics in this video we'll discuss the role of expectations operator in the context of random probabilistic variable any random probabilistic variable is often represented through expectations operat this random variable can be the error term or the dependent variable like y since these variables attain multiple values for example let's take an example of coin tossing game where there are two possible values and if the coin is fair then there's a 50% chance or odds for any of the outcome whether it is head or tail similarly in regression any random variable like the dependent variable y if it is assumed to be problemistic in nature its expected value is represented as ey for example if there are n probabilities that an event has an outcome of y1 Y2 Y3 YN with possibilities P1 P2 these are probabilities P1 P2 and so on up till PN then the expectation of this variable Y is defined as P1 y1 P2 Y2 and so on that is sigma py p is probability Y is possible outcome Pi I VI I equal to 1 to n where n are the possibilities in the event this is also called probability weighted mean if all the probabilities are assumed to be equal that means many times we do not have any a priori knowledge about possible probabilities so if all the probabilities are assumed to be equal then P1 equal to PN so on equal to 1 upon n and in that case the expectations is nothing but the simple average of all the possible outcomes of yse to summarize in this video we discussed the role of expectations operator for random probabilistic variables or stochastic variables like error term or dependent variable we noted that expectations are nothing but simple probability weed mean of variables in case we do not have any a prior probability assigned to these outcomes Yi then in that case the expected value is nothing but simple average of Y in this video we'll understand the basics of simple linear regression algorithm with a simple example consider the table shown here where family income and consumption expenditures are provided here a population of 60 families is divided into 10 income groups for example 80 80 to 100 100 to 120 and so on up till 260 so there are 10 groups and their consumption weekly family consumption expenditure the Y's are also shown for each level of income X we try to compute the conditional mean that is expected by given X which is 65 let us see how this is computed for example consider an income level of 1880 corresponding to this income level a set of family weekly consumption expenditures that means y are provided so for this given level of X the corresponding y's are provided the summation of these Y is 32 5 and their mean is 65 now this 65 is the conditional mean of Y conditioned upon a given value of x which is 80 and that is why it is called conditional mean of Y expected value of y given X recall that in the previous video we said that if we do not have any a prior probabilities of any set of values we take simple average to compute the expected value and the same thing we did here by taking the average of these values which is 65 for xal to 80 now the average of all these values is 121.2 and since this value which is the average of all the values is not conditioned upon any specific x value this will be called expected value of y which is unconditional mean or unconditional value of y because it is not conditioned on any x value it is uncondition expected value of y please note that such uncond additional value does not account for any income level that means any value of x and it is the prediction or expected value of y or the prediction of y when there is no knowledge of X however one would like to improve upon the value of y by knowing the value of x if one believe that f x affects y so if x indeed affects y then the knowledge of X would definitely improve the prediction of Y so if you have the knowledge of X you can improve your prediction by Computing the conditional mean of Y which is this expected value of y given X and that prediction will be more accurate for example the conditional mean of Y given xal to 260 will be 173 which is different from conditional mean of y when X was 80 now if one ask what is the best prediction or mean value of weekly expenditure for an income of xal to 140 then you would tell him that looking at this table looking at this table you will tell him a value of 41 that means the knowledge of income level that is values of X enables us to better predict the mean value of consumption so our prediction of Y which is the mean improves as compared to the situation when we do not have that knowledge so this is the essence of regression modeling so what we are saying here that for each level of X we have set of V values and the mean of these y values are joined together these means of Y values for each level of X we have expected value of y given X and all these conditional means are joined to get the regression line so this regression line is nothing but the conditional means conditioned upon these X values are joined together to get this fitted line so for example if you have the B click incomes given on x-axis and for each set of income you have a distribution of Y values this is the distribution of Y values and we know that for any distribution if a variable is has a distribution it is stochastic in nature we can compute its expected value as we have already seen using the probabilities we can get the conditional means for each given X we can compute the expected value of y that is expected value of y for each given XI where XI can be 80 90 in this case 8090 and so on and once we have those conditional means we can join them to get the regression line and then in that case our estimate of Y would be conditioned upon XI for any value of XI we can predict the value of y to summarize in this video we said that any regression model of this type y = to B + b1x y = to B + b1x I assumes that some variable XI has some explanatory power over variable Yi and therefore one can improve the prediction of Yi the prediction value of Yi which is Yi hat can be improved by using the knowledge of XI thus regression modeling improves the prediction of Yi by taking a set of values of XI for example if we have the knowledge of XI the corresponding values of Y can be averaged and this average value of Yi for given I is a better estimate of Y since this estimate of Yi that is Yi bar is for a given value which is Xi and therefore often such estimate is called conditional mean of Yi that means conditional value of y given X and this is called conditional mean of Y another type of expectation of Y which is unconditional mean which is which ignores the value of x which does not consider and assumes that we have no knowledge of X is expected value of y which is the average of all the values of Y is respective values of X it is often considered as per aggression modeling that this conditional mean is an improved estimate of Y as compared to the estimate of y when we have no knowledge of XIs in this video we'll introduce the concept of population and Sample regression functions in the context of regression modeling recall in the previous video we said that if we join the conditional means of Y given X these were the conditional means of Y given X if we join these values then we obtain what we call is a operation regression line so we join them for each value of x i X1 X2 X3 and so on we had a set of values on y and we join the mean of these values what we call conditional mean rather for given XI what is the mean of why when we join these conditional means what we obtained is called population regression line which is essentially the regression of Y on X now a population regression curve is simply the locus of these conditional means of the dependent variable which is Y for the fixed or given values of explanatory variable which is X here so this is the basic concept of population regression function the population regression function discussed here can be denoted by this expected value which we have already seen expectation of Y given XI this is often referred to as population regression function fxi here fxi is a linear function of X which is called a population regression function when we say it's a linear function it this linearity means linearity in parameters and this can be more simply written as expected value of y given XI equal to Beta plus beta 1 into XI where beta KN and beta 1 are population parameters these are population parameters because they are for the entire population assuming the population is known which is a more hypothetical construct you never have the entire population but assuming you know the population then these are the population parameters or the True Values of beta not and beta 1 which reflect the relationship between Y and X now when we say this is the linearity in parameters that means for example if this kind of expression is there where the population regression function is beta plus beta 1 s XI this model is nonlinear in parameters and will not be estimated with linear regression modeling that we are discussing here however a model like expected value of y given XI which is equal to Beta plus beta 1 XI squ this model is while nonlinear in variables it is linear in parameters beta not and beta 1 so population parameters are linear and therefore it can be handled under linear regression models however in real life we do not have the luxury of population we often work a much work with much smaller set of data which is called sample and therefore what we estimate is not the population regression function but what we get is the sample regression function and it is denoted by adding a hat symbol this hat indicates that we are working with samples and therefore the estimated values of Y is Yi hat so Yi hat is the estimate of population value which is y so Yi is the population this Yi head is the estimate of Y which is given with this regression sample regression function where beta not hat and beta 1 Hat are sample parameters unlike the beta which was the population beta not and beta 1 which were population parameters beta not hat and beta 1 are the sample parameters or the estimators of population parameter which is beta not or beta 1 so beta not here here is the estimator of beta not we'll discuss some of these estimators beta 1 Hat is the estimator of beta 1 and these estimators are applied on a given sample to estimate or appr proxy the population parameters using these beta not hat and beta 1 Hat what we get is Yi hat which is the estimate of Y which is the estimator of population values based on our sample regression function so srf which is the sample regression function is only a an estimate of population regression function based on a given sample or the samp that is available to us remember we do not work with population we do not have the luxury of population now since this is only an estimate therefore sample regression function can overestimate or underestimate the population regression function or population regression function values for example if you are estimating Yi hats and you get this kind of estimators beta not hat and beta 1 Hat your sample regression function line may look like this this is one example where actual population regression function is this so from this point onwards you overestimate the population values of Y and behind this below this point a before that you underestimate the population values so your estimate is overestimating here underestimating here to summarize in this video we discussed the concept of population regression function which was nothing but the values of Y using population parameters beta not plus beta 1 into X Now Beta KN and beta 1 here are the population parameters please note in real life we do not have the luxury of working with population and therefore this is not observed this is not observed so the set of population is not observed and we only try to map this population or estimate using a much smaller sample which is called sample so you work with samples in your real life through which you try to make estimates or hypothesis or various predictions about the population which is much larger in size when you work with sample you denote this with a hat symbol y hat you do not have actual Yi and XI you have Yi hats which are the predicted values beta not hat plus beta 1 Hat into XI now please note this y i hat is the prediction of y i this beta not is the estim of population parameter which was beta not beta 1 Hat is the estimator of beta 1 which was the population parameter and this is called srf sample regression function because it is estimated on a sample which is available at hand and it tries to estimate the population values population parameters and population VIIs however please note this VII hat is often not very accurate and it may over predict or under predict the actual value values in this video we'll discuss a very important class of estimators called Ordinary least Square estimators recall the sample regression function which we discussed was y equal to Beta not hat plus beta 1 Hat XI plus mui hat here we have removed the Hat because now we are also we have also included the error term earlier we saw this Yi hat equal to Beta not hat plus beta 1 Hat into XI which is nothing but the prediction of Y had the predicted value of y hat once we add the error term in the prediction it becomes the actual Yi now the error term this mui had or often also referred to as mui is important here because this error term which is Yi minus Yi had or Yi minus beta not had minus beta 1 * beta 1 Hat XI this error term is a important very important cause function and any reg any regression fit or any estimator should minimize some function of this error in this particular case the o ordinary Le Square procedure as the name suggests tries to minimize the square of this function and therefore the best cost function to minimize this can be shown as follows this is the appropriate cost function which is the sum of squares of error term which is y i minus y i hat ra to the^ two summation this is our cost function which is minimized here now we can also replace this Yi hat using this expression here so we get y IUS beta hat minus beta 1 Hat into XI ra to the^ two summation is our sum of squares now we minimize these squared residuals question is why not minimize just the residuals or absolute residuals now talking about simple residuals it fails to recognize the fact that large positive and negative errors for example plus 6 and - 6 will tend to cancel each other out so if there are large deviations if there are large deviations large error terms mu1 I and mu J which are very large but on the summation they tend to cancel each other out simple summation of residuals will ignore that similarly if we are minimizing the absolute residuals to an extent that is an improvement because it will recognize that plus 6 and minus 6 or positive and negative residuals should not cancel each other and therefore POS large errors whether positive or negative they are penalized however in case of squad residuals it penalizes large errors more strongly so the penalty on large errors is much more stringent in case of squares as compared to simple absolute residuals to further illustrate and visualize this argument let's have a look at set of observations plotted on this XY diagram these are the actual population actual values that we have rather not population but a sample that we have obtained now we try to fit a sample regression function or sample regression the line corresponding to the sample regression function which is represented as y i hat equal to beta 1 Hat plus beta 2 hat now obviously because we are working with samples these are estimators since these are o estimators we try to minimize the sum of these squared residuals the difference between the actual values and this fitted line so this is our sample regression function the line corresponding to sample regression function this distance is my error in prediction now this error the larger this error the problematic it is for me and therefore a function which minimizes the square of these errors gives higher penalty to large errors as compared to small errors if I would have added simply the absolute residuals it would have definitely penalized errors but a square of residuals penalizes large ER much more than simple absolute summation now in this scheme our final function our final function or Square residual summation is this which we can also write as summation of mu square is a function obviously a function of beta not hat and beta 1 because it is written like this so it must be a function of beta not hat and beta 1 hat which are the key parameters or estimators of importance so these are sample parameters that we are interested in knowing now a very regular simple scheme to minimize this we want to minimize this Square residuals we can set the differential of this partial differential with respect to our betas equal to zero we can set this differential with respect to parameters different parameter estimates beta not hat and beta 1 Hat equal to zero and then set the condition of the double differential to positive for minma condition to obtain the estimates of beta not and beta 1 Hat as we'll shortly see and this is the reason why these estimators are called Le Square estimators because we minimize the square residuals and such scheme or regression model obtained from such scheme is also called gsan standard or classical linear regression model by minimizing these least Square errors now if you recall we said that mu Square can be written as this function which is Yi minus beta not hat minus beta 1 Hat XI summation Square summation we can take the partial differential with respect to Beta not which is since it is square term this is minus 2 * this term which is same and then differential of beta not is nothing but a minus sign so we get this minus 2 * mui hat why because this term itself is Yi hat and Yi minus Yi hat is nothing but mui hat or often referred to as mui itself there we are interchangeably using mui hat or mui here so this is nothing but minus 2 * which is the partial differential with respect to Beta not similarly we can take partial differential with respect to beta 1 one hat which will be very similar only that now because beta 1 Hat is a multiple of XI we have an extra XI here so we have minus 2 * mu hat into XI summation this is the partial differential with respect to beta 1 Hat now we need to set these differential equal to zero and solve the equation a close form solution can be obtained which is shown here this close form solution can be obtained only if certain assumptions are met now in this lesson in a set of series of videos we'll discuss some of these assumptions that are needed and if these assumptions are met then we can obtain a close form solution like this that means we can precisely estimate the values of beta 1 bet not and beta not had so we can obtain these beta 1 hat and beta not hat estimators and these estimators are often referred to as OLS ordinary Le Square estimators because of the minimization of these residual Square scheme these are called Ordinary Square estimators beta not hat and beta 1 Hat and once applied to a given sample we obtain the estimates of population parameter so what the estimates that we obtain the beta not hat and beta 1 had based on these formulas these are the estimates from the sample to summarize in this video we discussed a very important class of estimators called Ordinary leas Square estimators the ordinary Le Square estimators are obtained by minimizing the sum of squares of residuals shown by a formula here to visualize this scheme we plotted all the observations from our sample like this then we try to fit a line which we often called the line corresponding to the sample regression function because we are working with samples so this sample regression function line can be written as Yi hat because these are only estimate not the population values beta 1 head plus beta 2 head XI and then we compute these error terms that means the difference from these observed values to the predicted values which is y I minus y i head this is the gap which is called mui or we are referring to as mui hat or mui interchangeably and then the square of these mui are minimized to obtain the estimators beta 1 hat and beta 2 hat which are called the oils estimators [Music] [Music] [Music]