Lecture 21 : Regression Algorithm: Application (Part 1)

[Music] in this lesson we'll discuss the application of regression algorithm with our programming through a financial Market case study this includes modeling security price data with single and multiple features first we'll train the algorithm using the training data set and examine various goodness of it measures and next we'll test the algorithm on test data set and examine its out of sample prediction efficacy in this video we'll introduce ABC stock price case study which entails forecasting stock prices using regression algorithm please note that stock price prediction or stock return prediction is an attempt to determine the future value of a company based on analysis of factors which impact its price movement there are number of factors that help in predicting stock prices these can be macroeconomic factors like state of country's economy growth rate inflation and so on there are also other factors that are more specific to a stock like profit margin debt to equity ratio sales and so on let's examine the data provided in this case study so we are given data for market prices of a company ABC along with Nifty and sensex these are market indices that represent broad Market wide movement of Securities and we are also given data of dividend announcement and a sentiment index let's discuss this data in more detail now think of a portfolio manager who has to build a model for a particular stock which is ABC in this question and under consideration the manager wants to predict the ABC stock price return for this stock using regression model the data provided to us starts from 2000 and goes till 2019 so we have approximately 19 years of data we can construct daily returns as provided here so we have daily returns of ABC these are provided in this column and next let's call it column B next we have daily return on sensex this is column C and daily return on Nifty in this column please note that sensex and Nifty are two main these sensex and Nifty are two main stock indices used in India they are The Benchmark Indian stock market indices that represent a weighted average of the largest Indian companies so sensex represents average of 30 largest and most actively traded Indian companies similarly Nifty represents a weighted average of 50 largest Indian companies another variable here is dividend announcement which is provided here in this column this is one this variable attains a value of one if the company has announced a dividend on a particular date and zero otherwise so for example if the value is one on this given date of 10th January 2000 because the company ABC announced a dividend on this date and it is zero for other days as we can see when the company did not announce any dividend please remember this is a dummy variable lastly we have sentiment variable in this call it is a sentiment score that quantifies how investors feel about ABC company it can be based upon news analysis or upon option market analysis or based on some survey of investors we would not go into details of the construction of this score here and take it as given a very high sentiment score represents bullish emotions of investors and vice versa to summarize this video in this video we introduced ABC case study problem with respect to stock price forecasting we discuss the background of the case study its relevance and implication for financial markets we discussed the variables and data provided in the case study the interpretation of each of the variable and its implication for the regression modeling in this video we'll introduce the problem statement with the ABC Cas study the problem statement will be demonstrated and implemented with the r programming in the first part of our problem implementation we'll start with the simple linear regression model we'll start the study with data visualization we'll plot the returns and cumulative returns for ABC stock and market indices like Nifty and sensex then we'll segregate the data into test and train data set we'll train and build the model using the simple linear regression algorithm using market index as the independent variable and ABC returns as the dependent variable using the training data set so we'll train the model first then we'll evaluate the model with various good of goodness of fit measures and through visualization thus we'll test the model and also as a part of testing the model we'll evaluate the out of sample model performance on test data set using various quantitative measures and also with visualization in the next set of implementation we'll employ multiple linear aggression algorithm we'll start with training and building the model using multiple linear regression algorithm using Market indes sentiment dividends among other variables with the training data set next we'll evaluate the model with various goodness of it measures and through visualization there are several issues that we'll deal with these include multicolinearity heteros cacity and autocorrelation we'll examine these issues and we'll try to find ways to resolve them we'll answer the question how to resolve these issues in the estimation and then we'll evaluate out of sample performance on the test data set using various quantitative measures and also with visualization to summarize this video we discuss the problem statement that will be implemented through our programming in this video we'll start with our implementation of ABC case study regression problem we start with the ABC case study problem we'll conduct the following steps first we'll load the relevant packages in this case study we are going to use read Excel package this package will help us in Reading Excel files data in the Excel format we'll make use of lubri dat package which will help us in datetime manipulation then we'll use we'll use Ms package for understanding various properties of data then we will load car package car package along with LM test will help us in various regression related aspects and lastly we have sandwich package that will help us in handling the estimated estimation related issues in the regression so with this now we have loaded all the relevant packages as a Next Step we'll set our working directory please remember please remember it is always a best practice to set the working directory in order to set the working directory we'll go to this session button set working directory go choose working directory we have this particular file and folder if I keep put on open Button notice a command appear on my console window this command suggest that now that this command is run appropriate working directory has been set so for reference I'll just copy paste this command on my code so now that my working directory has been set I can read the data file so data I will use read Excel command the file name name is abc. xlx you can save it with any name I I have the file in with the name of abc. LX xlx so now the file is saved you can check a few initial elements of the data file by using head data command notice there's a date variable there's a price variable return variable ABC sensex returns dividend announcement dummy variable 01s sentiment variable and Nifty as we have discussed during the case study problem statement discussion you can also check the dimensions of this data with this dim data command it has 5,153 rows that means 500 5,153 daily observations for seven variables just that we just saw one important issue to be noted you can check the class of various elements for example if I want to check the class of date variable let's see so I put dollar symbol which means I am accessing that variable inside data frame so so it's already a date variable uh also please note there's a time element uh which is a spous kind of uh in spous in nature we need not handle this time element for now we only need date which means month day year so I want to remove this particular aspect I can do that very easily I can simply just run this date data dollar date and I can change the convert type to date earlier it was in posex CT type of class which includes not only date but time element since we are making use of only date element I'll simply convert it to the date part so now if I run this notice the time element is eliminated only date is there I can run this class command again so it's classes date so now it's not anymore the posix CT element that it was earlier now it's only purely a date element so to summarize this video we set the working directory we load the event packages and then we set the working directory we read the data file which is abc. LX in our data variable which is essentially a data frame we can check that also for example you can type the class command and check the data frame it's a data frame then we check the dimensions of this data we converted our date variable from posex C date time object to purely a date object from the next video we'll try to visualize the date with its relevant properties in this video we'll conduct the visualization of data we'll visualize the data we'll examine key variables we'll conduct the visualization of key variables including Returns on ABC and the market variable that is Nifty so first let us plot the ab C return so we'll use the plot command on the x-axis we'll have the date variable on the y- axis we have the return variable we'll give the xais name using xlab command as date so this will give us the xaxis as date and then y AIS label as we'll keep the y axis label empty but we'll give them Central heading we'll use Central heading or main heading as ABC returns so you'll plot the ABC returns along with prices and we would like to see the plotting format in the line form so we'll put type as L now as soon as I run this command we'll notice that a plot will appear on the plot window here we can zoom it you can see the plot you can see there are some extreme Falls here around 2005 and 20 8 and 10 extreme down movements increase in volatility if you want you can export this plot to save as image you can use different image for formats to save this plot for example you have PNG you have jpeg Tiff various formats available to save also you can export it in the form of PDF you can save it as PDF or you can copy it to clipboard and then paste it on Word document and PowerPoint presentation and so on so this is the return variable ABC returns next I would like to plot the cumulative Returns on ABC so how to compute cumulative returns we let's name this variable as c m r so we would like to plot this cumulative return variable the simple way to compute cumulative returns are use data dollar price this will use the price and divide it by the beginning price opening price so any price divided by opening its opening value will give was the cumulative returns here so I need to use the first value of price and then I subtract it from one to reflect the return aspect so this will compute the cumulative return return in my data dollar cumulative return return variable now I need to Simply plot it so I can use the plot again I can simply copy paste the command that I used earlier the same command only that now instead of writing as ABC I can use this new variable C cumulative return variable I can use that and then name the plot as cumulative returns and if I run this the cumulative returns you can see the cumulative returns provided here so it it looks like this there's a sharp fall around here 2005 these are cumulative returns now we'll plot the market returns so we'll plot the Nifty returns Market or Nifty returns very simple I don't have to do any extensive coding for that I'll simply use my plot command that I created earlier I wrote earlier and instead of using ABC I will do a minor change here I'll write Nifty so this will give me the market return Nifty other things remaining the same I'll use the instead of ABC I'll change the central heading to market returns and now if I run this command notice these are my market returns so these are simple market returns next we'll plot the cumulative market returns cumulative market returns I have to just write a very simple command which is data dollar cumulative this time around we are Computing Nifty cumulative so I'll use this and a very simple come sum command cumulative some command to compute the cumulative Nifty returns this will provide me with the cumulative Nifty returns and I will not write the command again so I'll just simply use the same command that I used earlier and I'll just change here cumulative return instead of cumulative I'll use cumulative Nifty returns and in the central heading I'll change the name to cumulative market returns so here I can see the behavior of cumulative market returns easily this is the behavior of cumulative market returns to summarize this video in this video we conducted basic visualization of return variables this included ABC returns and Nifty returns we also plotted the cumulative returns in the next video we'll conduct the summarization of data we'll provide some summary measures we'll also have a look at the normality skewness and curosis of the data and stationarity of the data as well in this video we'll discuss the basic properties of our data we'll start with a very basic summary measure so we'll start by summarizing the data for that we'll use summary command simply summary command and for summary command for ABC returns we can see how ABC returns are distributed the minimum value which is quite High and the maximum value and first quantile which is 25% median value which segregates the data between two halves then mean of the distribution and 75 percentile of the data so these values are provided similarly I can examine the summary of nifty Market index which is this so here also we can see extreme return which is- 14% maximum is around 15.97% and same Matrix that is first quantile median mean and third quantile in The Next Step we'll conduct a very important operation we'll try to see the probability density of our data in R the same can be implemented with this plot command and then density data dollar ABC so this will plot the simply the density of ABC but we want some aesthetic aspects also so we'll type main as we'll type here as density this is a density graph so we'll type density graph and then we'll name the X and Y axis so xaxis we'll just name simply the data also we would like to since we are plotting the density for both market and a BC we will try to give them different colors first we'll plot the density of stock ABC return density in red let's give the line width of two so it will plot a slightly more solid line line type as one so a more solid line will be there and we probably will uh if you do not specify the limits can be extremely large so we'll restrict the limits of xais to - 0.6 to 0.6 although these are are these will change depending upon your final result you may modify and Tinker with that a little bit here and there to improve the Aesthetics next with this lines command I do not want to plot a new graph on the same graph I want to plot Nifty return so I'm using lines command Nifty then I would like to specify the color earlier I have used the red color to PL the ABC return so I'm using the blue color to plot the Nifty returns and line withd again I'll use the line width of two so this will be plotting the NY of nifty and then I would like to add a nice Legend on the on one side so in order to check this first let me plot the okay so this is my uh plot of ABC return then my density plot blue in blue we have Nifty so now there's some empty space on probably the top leftt so I'll use that space to put my Legend here so top left and I'll use C ABC then I'll name the next Legend as Nifty because of this sequence since ABC is in Red so I'll use the red here and Nifty is in blue so I'll use the blue here and now I'll run the Legend So if I run this Legend a very nice Legend will be added use f equal to color and now the legend is added let us enlarge this plot a little bit so notice on this plot plot ABC is plotted in left and notice the behavior of ABC it is extremely left skewed you can see the extreme left tail of ABC while Nifty is rather more balanced and symmetric in blue Nifty returns are more symmetric over the sample period while ABC which is in red are extremely skewed towards the left that means there are extreme negative returns which we also saw in the summary measure so it is sort of confirmation of that to summarize this video we first summarized the Nifty and ABC returns using summary command and then we plotted their density distribution to understand the basic characteristics of our return data for ABC stock and Nifty in this video we'll examine the normality and stationarity of the data now first as a part of normality examination we'll start with the skus of the data we'll write skus data dollar here so we'll compute the qess which is very highly High negative as we already saw in the diagram the skewness of ABC is extremely it's extremely left skewed then we'll examine this sess of nifty returns so the Nifty is also negatively skewed but the SK is not very high but we want a slightly more solid evidence for that so we'll conduct basic atino test of skewness so we'll type data dollar ABC and this will provide a more statistical measure and notice it provides you with a P value which suggests that the skus is statistically significant in fact very significant for ABC returns similarly we can compute the sess of nifty returns so here we computed the qess of nifty returns this is also in terms of magnitude not very high minus 0.177 but its significance is quite high so this is also slightly negatively skewed and low in terms of magnitude but reasonably higher in terms of statistical significance now skewness is a measure of asymmetry on the left side or right side in the remaining part we'll discuss the curosis measure so we also examine curosis measure curosis is a measure of peakness and tail so how Peak or less peaked as compared to normal distribution and how tailed it is whether tails are fat or not so first we'll compute the curtosis of ABC returns the curtois is quite High we'll also measure the curtosis of probably this can be accounted to the extremely fat tals of ABC and pig nature as we saw in the den distribution Nifty also has a very high kosis so we can see that it also has a very high kosis normal distribution has a kosis of three so as compared to normal distribution the curosis levels are very high now we would like to again have a slightly more statistically significant evidence so we'll use the ancom test first we'll compute it for ABC returns dollar ABC for ABC okay so the curosis is not equal to in fact very significantly different from three for ABC which we expected we'll also see the kosis kosis is the measure of pness and tail how fat tails are or how peaked the distribution is as compared to normal distribution so the tails are tails are fat and the pigness is very different from normal distribution here also C is very sently different from normal distribution as a final test we already know although that skus and curosis are very different and as a combination skus and curtois decide whether data is similar to a normal distribution the Bell shied curve that we discussed as a final test we would like to have Hara test har test or it's called Hara test or harest which will give us a sense of its normality although we already know that data is not normal but it's still combined measure of normality which is very very high which is much greater so data is non normal as you can see this is a combination of extreme skewness and Pak nature of ABC although in case of nifty the skewed nature was less it was not as skewed but still as compared to normal distribution it was SK and the taals were fatter on both side as compared to normal distribution same for nift as well Nifty we can see it was symmetric but still it's non normal non normality is high we'll see while working with the regression model we'll see how to deal with these issues but we have seen the summary measures now just to summarize this video we computed the skewness curosis and normality of the DAT overall normality of the data we found that ABC was extremely left skewed and both of the distributions have fat tail as compared to normal distribution and higher Peaks this was statistically examined using agustinos test and anom test and using Hara test we also examined the overall normality of the data we found that both the series ABC return series and Nifty return series are non-normal in nature and the result is stally significant as well in this video we'll discuss a very important property that is stationarity of the data the stationarity of the data refers to the fact that the mean and standard deviation or variance as we have already discussed the concepts these are changing with time and if these parameters such as mean and variance are changing with time estimation is fraught with various econometric issues so before estimating or estimating any kind of relationship it is important to know whether your data is stationary or not there are three very useful tests for that and these are augmented DK Fuller test Philips paron test and kpss test for that these test we need to load a very important B package which is urca unit root test so we'll load this packagea and now we'll start with our first we'll conduct the augmented dck Fuller or often called ADF test the command is quite simple we'll write summary and Ur DF or augmented DK Fuller DF data dollar ABC is our price Series so we'll use ABC and notice the null hypothesis of this particular augmented DK test is that data is non-stationary since notice the test statistic it is minus 51.7 which is much lower and much higher in terms of magnitude as compared to 1% 5% and 10% therefore with a lot of confidence we reject the null and we say that data is stationary please note this is a non-stationarity test that means null hypothesis of this is that data is non-stationary very similar syntax we'll use to check the stationarity of the Nifty returns if the returns were non-stationary any model such as regression algorithm will be difficult to estimate properly so we'll check the Nifty returns also Nifty returns also notice that we reject the null of non-stationarity with a lot of confidence so again the data is stationary but when while testing a stationerity it is very useful to test through a number of tests so for example lentic D Fuller tests were test of non-stationarity here we'll be using test such as PP test also look at the test such as ur. PP and notice here again this is also a test of non-stationarity and we reject notice the test statistic which is very very large in magnitude very negative as compared to the standard statistic given the largeest statistic we can reject the null and again say that data is stationary so here also the null was of stationarity so we reject the null and we say the data is stationary it is very high tested statistics so we are again and again saying that data is stationary another very interesting test and generally it is required to test for stationarity test also so earlier both the tests earlier that we did Philips Peron augmented rcky Fuller they were test of non-stationarity now we'll make use of kpss test which is the test of stationarity so first we'll run the kpss for ABC returns and notice the test has testic is much lower than 1% 5% and 10% level so data is again we are not able to reject the null of stationarity so data stationarity similar test we conduct for Nifty returns also kpss test and here also we find the data is stationary so we have conducted the stationarity test to sumarize this video we conducted test of stationarity and non-stationarity here augmented Dey Fuller and Philips P test were test of non-stationarity where null hypothesis was of non-stationarity and rejection of null indicated stationarity while kpss test was of stationarity and we failure to reject the null we failed to reject the null which indicated again the stationarity of the data in the next set of videos we'll start with training and testing of regression algorithm and prediction of stock returns in this video we'll segregate our main data which is ABC stock return data into training and TR data training data will be employed to train the linear agression algorithm while test data is employed to test the out of sample forecasting efficiency of the algorithm so we'll start the analysis by segregating the data into training and test data as a first step we'll create our train data segment which is equal to main data and we'll tell our that the date should be less than equal to now we'll filter the date with 2017 less than 2017 December so first December 2017 is our cut off point and this is the maximum and it should be great let's say we also filter observations before 2006 so let's say we also want do not want to go too early so we filter those observations that are for which the year is greater than 2006 so we are filtering from 2007 to almost 2017 we are filtering all the observations so this is our train data and let's check this so we'll check the head of train data head of train that that starts from 2007 as we can see here let's check the tail of the train data it ends with 2017 as we can see here in the date segment we can see on the console window it ends with 2017 and if you look at the dimension of the train data it has 2850 observations similarly we'll select our test data our test data is again in a similar manner we'll filter it data dollar date should be greater than and now this time around uh we'll use the exact same date criteria which is greater than s. date and we'll precisely select the date that we selected earlier we'll select the same date in fact so all the observations with higher date number will be taken here so now this test data will contain the remaining observations that are above 1st of December 2017 let's examine this so the head of the test data is starts from 2017 in fact 4th of December and look at the probably second and third are holidays and if you look at the tail of this tail of the test data the tail it ends with 2019 and if I examine the dimension of test data so I have around 47800 of 478 observations to test the algorithm them so to summarize in this video we segregated our data into two segments first the training data which starts from year 2007 and ends at 1st of December 2017 while the test data contains the remaining observations starting from 4th of December 2017 up till 2nd of October 2019 in this video we'll train our simple linear regression model by regressing the ABC Returns on Nifty returns so we'll model simple linear regression model by examining the ABC returns and Nifty returns relationship so in r this relationship examination is done with a very simple LM command so we'll run this LM command which indicates some kind of linear model linear regression model is about to take place and where we are regressing ABC returns here this notation suggest that ABC is the the dependent return variable ABC returns and the independent variable is Nifty our data is train data we are using to train this linear regression model and the output of this linear regression model the train model output the coefficients will be stored in SLR object simple linear regression let us see the summary of this object we'll assign in fact we'll assign the summary to the model object summary SLR let's have a look at the summary variable I can simply extract this by printing model notice the console window on the console we can see the printed output in the out put the following objects to be noted first note the coefficient of nifty which is 0.397 this is also the measure of beta that we have discussed recall the discussion on beta that beta represents the sensitivity of ABC returns here that means if Market moves by 1% ABC stock returns moved by 0.39% notice also notice the T value of the coefficient which is heavily significant remember this T value is computed as the ratio between estimate which is the coefficient that is 0.397 here divided by the standard error now this T value 18.67% level recall our discussion on the significance levels null hypothesis here is that this coefficient Nifty has a value of zero which means Nifty does not affect the movement in Nifty does not affect do not affect the returns of ABC stock however given this result the significance the very high significance of this result at 1% we can say definitely there's a positive impact of nifty that is market movement on ABC returns here it appears if Market moves by 1% ABC stock moves by 0.39% and this relationship is statistically significant at 1% level so we are very conf confident in fact we are more than 99% confident that there's a relationship between Nifty returns and ABC stock returns moreover notice this adjusted R square here and the multiple R square in fact we'll focus on adjusted R square which is a more improved version it is 10.87% they suggest that out of the total variation out of the total variation in ABC return 10.87 variation is explained by Nifty which means and recall our discussion on the market risk and systematic risk that almost 10.87% part of the overall risk total risk if total risk is 100% 10.87% part of the risk is Market risk or systematic risk driven by the market movements in this case Market is proxied here by Nifty the remaining approximately 89 89% 89. 21% around 89% is the edos syncretic risk which is specific to ABC stock the remaining 10.87% is driven by market and explained by market so this is our interpretation of the overall regression to summarize this video We examined the relationship between ABC returns and Nifty returns by regressing ABC Returns on Nifty returns we also examine the output of this regression model we discussed the coefficient its significant the edos syncretic and systematic part of the risk associated with ABC stock return return and we have now trained our linear regression algorithm in fact this is a simple linear regression algorithm because we are using only one variable which is Nifty so now that we have trained our simple linear regression algorithm on the training data set now we will examine we'll examine the efficiency of this model in subsequent videos we'll examine some of the econometric issues that this model May face and how to deal with them in this video we start with the residual Diagnostics of the model Diagnostics first we'll look at the density plot of the error terms so we'll use density plot recall that we have already stored our train model in the model variable or model object so we'll simply use the model residuals so we'll plot the density of this let's examine the density plot of residuals notice that while the shape of this density is relatively similar to normal distribution however its left tail is extremely long which suggests that it is extremely skewed we can also corroborate this by plotting the density plot of studentized residuals for that the command is simple we can just write the same density plot and we type our student but this time we'll pass on the original model as LSR object which is the model object that we created so again it is also very similar so the density plot of residuals and as well as studentized residuals is extremely left disc sked studenti students distribution is essentially T distribution so these are standardized errors and as we can see it is extremely left is skewed another test to examine the behavior of the model and its residual is QQ plot which checks the normality let's examine the QQ plot so we pass on the fitted object SLR which we already fitted we also would like to know some of the extreme observations that may create problems for us so we'll give the number as n equal to 10 so this will identify an extreme observations for us let's see so we'll run the QQ plot command also we would like to see some of the extreme observations so I type ID equal to list and equal to 10 so we would like to highlight 10 extreme observations I run this command and the model is plotted here so we can see the quantile wise distribution of studenti residual so the residuals are plotted and and we can see this number of observations that are extreme observation number 72 152 and 2544 these are some of the extreme observations that depart from the normality these on the fitted model these observations are plotted along the fitted model we can see the observation numbers the student TI residuals are plotted here we can see some of the observations plotted here so this is and we we can see that most of the time the observations the residuals are along the normal line the straight line indicates the normal line the observations that deviate from the straight line indicate deviations from normality and we can see some of the fitted observations on the extreme left and right end these some observations deviate from the nor normal distribution normal distribution here is represented by the straight Blue Line lastly we can also conduct the outlier test now there's a basic outlier test and it will also adjust for Bon fory values adjusted P values let's see them so if I run this notice uh unadjusted P there are lot of very significant unadjusted P values but when once there's bond for any correction one can see uh the significance is lit but still we have uh there are six extreme values and these are appear to be these appear to be very similar to the earlier ones so these are our extreme values if you want to know more about these functions and their theoretical underpinnings you can simply type question mark QQ plot and see the background the entire background of these commands for example this is for quantile comparison plot to check the normality and the outl test also you can check that the theory behind these commands by running the question mark you can read more and more about these commands lastly if you feel that these observations are vitiating these extreme observations are viting your estimation model you can remove them from their model you already have the observation numbers you can remove them from the from the model and then run the model again to improve its estimates to summarize this video we created density plot of residuals we also have examined the studentized residuals on the density plot next we checked the deviations from normality with the help of QQ plot and we also conducted the outlier test to see some of the outlier observations we found that there are a number of outling outlier observations that may be shed the estimation as a future course of action one can remove these observations from the model but please note sometimes these extreme observations also carry useful information probably that may help us model extreme values representing extreme events so sometimes it is not advisable to remove these values so subjective judgment call may have to be taken here [Music] [Music] [Music]