Lecture 25 : Classification Algorithms: Application (Part 1)

[Music] in this lesson we'll apply classification algorithm in security price prediction using our programming we'll revisit the ABC price case study first we'll train linear logic and proit models using our training data set we'll evaluate various statistical good of it measures and visualize the performance of these algorithms these measures include classification and confusion metrics and Roc curve and area under the curve we'll also train a simple algorithmic model to find the best performing algorithm using a simple performance object which is the simple average of model classification accuracy sensitivity and specificity measures in this video we'll recap the ABC stock price forecasting case study and then we'll see the application of classification algorithm in the context of security price prediction recall our discussion about ABC security price prediction case study in the context of regression algorithm in a series of videos we noted that stock price prediction or stock return prediction is an attempt to determine the future value of a company based on the analysis of factors which impact its price Movement we noted that there are a number of factors that help in predicting stock prices these can be macroeconomic factors like the state of the country's economy growth rate inflation among others there are also various factors that are more specific to a stock like profit margin debt Equity issues sales of a company and so on in this particular case we are given the data for a stock market price of ABC company its returns along with the Nifty and sensex returns so we have Nifty sensex and Nifty returns along with the dividend announcement and sentiment sentiment variable is provided sentiment essentially reflects investors view about a market it can be positive or negative as well and then dividend announced for example when dividend is announced here the value one reflects that on this particular date the dividend was announced now consider a portfolio manager who has built a model for a particular stock the manager wants to predict whether in the next period the stock price returns will go up or down that means whether it'll be positive return will be positive or negative data starts from 2007 and goes on till 2019 so we have approximately 13 years of data and we have daily returns for ABC which is basically the change in price here in this column along with that we have variables Market index variables like Nifty sensex sentiment and dividend no as we discussed please note that these sensex and Nifty are two major stock market indes in India they are bench Market stock market indices that represent the weighted average of largest Indian companies sensex represents average 30 largest companies that are most actively traded and similarly Nifty represents a weighted average of 50 largest Indian companies to summarize this video we will perform the following task using R programming first we'll create a dummy variable which is one when stock prices go up and zero when stock prices go down and then we'll segregate the data into training interest data set similarly as we did in the regression model then we'll train and build the model using simple logic proit classification algorithms using market index as the independent variable and up and down dummy variable so remember the up and down the positive negative returns will be represented through a dummy variable like we said so one when there's a positive return zero when there's a negative return so this will act as a dependent variable while market returns like sensex or Nifty will be employed as a independent variable next we'll evaluate insample performance and out of sample performance of these models we'll compute the marginal effects of the independent variable visualize the performance of these models using Roc curve and then examine the classification accuracy of the model and compare it with with a similar linear probability model in this video we'll start with our implementation of classification algorithm with ABC security price study so we'll talk about ABC security price prediction using classification algorithm so we'll conduct the following exercises first we'll set the working directory we'll load the relevant packages and then we'll read the data and then we'll create the up down variable which is one when returns are positive and zero when returns are negative so first we'll set the work working directory here using this session tab choose directory and click on it so notice a set WD command has appeared so for reference for future reference I'll just copy paste this command so whenever I'm working on this file in future I know where the working directory has been set now we'll import our ABC data take I'll import this ABC data so the file has been read and we can see that a view file has appeared now instead of naming it you can change the name also I'll just copy paste this command the full command for future reference I'll use this command and instead of using ABC I'll rather use the name we can change the name here so I'll choose the name data file already there is one read Exel package which has read through this command Library read Excel but we need some more important libraries first lubridate to handle date time pack operations then we need library of moments if we need to do some statistical operation like movements and all computation of movements and so on then Library car we have already seen the application of Library car another important library that we'll be using is LM test Library so these are some of the important library that we are going to use some of them already used in the Reg ation modeling Library tidy ver for any kind of Tidy operations library of margins for examining the marginal effect of logistic regression coefficients for rooc for generating Roc plot we have RR Library we have carrot library for classification Matrix and some various important measures and lastly there's also one Patchwork Library which is for some good PL plotting so these are the these are some of the libraries that we are going to use in our operations we have read the data also now so first we loaded the relevant libraries here loaded the relevant libraries then we read the ABC data another very important operation is checking whether your date often it is important to check whether date variable in your data is actually in the date format or not so sometimes the date variable is WR as character let's see if that is the case so so it is read as the in fact it is read as date time so posex C represents not only date but time as well but we are only dealing with the date part of it so I would to in the interest of clarity we'll I'll just change the class two date this is important so that there's no confusion while we are doing some operations so when I'll run this command I will convert my date from date time to exactly date now if I run again run back this class command notice in the console window you have date it is appearing as date so that time aspect is lastly I need to convert or rather create up down variable which is one so this variable should be one when returns are positive and zero when returns returns are negative so a very simple operation is needed to create this variable notice now we are working with T so it's quite simple I'll use this assignment oper then I start working with this with this symbol I start working in data I request it to create through mutate command I request it to create a new variable it name is up down this variable is equal to and I'll use f Els operation ifil Operation if and remember ABC variable contain the returns so if the returns are greater than zero then I want it to be one and zero if the returns are negative now if I run this command I run this command we would like to check our data so I'll use the head since we are in diverse I can simply type data and notice there's a variable up down created when returns are positive ABC returns are positive this variable is one and when returns are negative as here in red it is zero to summarize this video we started with setting the working directory then we loaded the relevant libraries then we loaded the data and converted its date variable which was actually date time to Pure date variable and then we created a new variable called up down variable which is one when returns are positive and which is zero when returns are negative we created this variable because ultimately we are doing classification with the help of limited dependent variable where this up down is the limited dependent variable or binary response variable or discrete you can say discrete variable in this video we'll create our sample of test and training data sets training and test data sets so as a first step what we'll be doing is we'll be separating only those observations that are after 2007 so we'll first in our data we'll create those observations using this filter year of date is greater than 2006 so notice in the console now only those observations are there where ear is greater than 2006 and we'll assign this to our new data so this will create a new data assign this data to our same name and now our year is starts from 2007 so this is first step now we'll randomly select 80% of the observations as training data set using these 80% observations we'll train our data and remaining 20% will be used to test our Trend algorithm so we'll create an index variable the application of this index variable it will be quite simple so using the sample command we are randomly creating an index variable so notice this with the sample command we are selecting data number of rows with number of rows and here we'll specify that out of these rows only 80% that is 0.8 into only 80% of the rows are to be selected in this index data so the objective of this Index this will act as index in while we are extracting the observations from our data variable notice how we do it so we use train na. omit so we want to so first we'll also sometimes there are missing observations the best way to deal with them is use this na. omit command so all the missing observations are removed and so we save some unavoidable problems now we'll simply select only those observations and see how we use this index variable to extract only those rows so index will act as a index while extracting those rows all the columns notice on the right side of the comma there's nothing that means you want to select all the variables if we wanted to select some specific variables like variables 1 2 3 then we would have used this but we want to use all the variables so I'm simply using index comma and now the train variable will contain all the rows randomly selected 80% of the rows and please remember these are randomly selected now in the test data again we want to use only those rows that were not used earlier so very intuitive and uh interesting use of this minus sign which tells us that only those rows that are excluded in that are not there in the index variable those rows are to be selected and again the same comma operation and we get that in test data so we have our tress data and train data ready now as a practice just have a look at we'll just have a look at these data that we created so see they are not properly ordered train data and test data they are not properly ordered so the next thing we'll do we'll try to order them with time so what we'll do is notice this command train and I want to arrange this with date so I'll use the date command to arrange the train data and now you notice that not train command is with the order of time order with time similarly we would like to order our test data also so with this I'll order the test data with time date basically our date variable indicates the date so we'll arrange this as well with date so now that we have created our test and data train data which are arranged with time as a good check it is important to see whether the distribution of our decus or binary variable which is up down variable has same proportion what do I mean by proportion let's look at so I want to know what is the proportion of ones and zeros in this up down variable in our original data so I'll use this prop. table command and then I'll put put it in tab with table I'll put in table form I have the original data which is data dollar up down and I'll multiply it with 100 to get the number in percentage form so notice it's around 50/50% appro approximately 50/50 in the main data set so I want to see whether this Remains the Same in our test data also so I'll use this first I'll check with the train data and let's see very close it's almost 50/50 and they should not come as surprise for us because we randomly selected our sample was very randomly selected there's no systematic bias so I it should the proportion of up and down variable ones and zeros should be same as the parent data and it's supposed to be indeed the case although in test data is a slight deviation but it almost it is same so it's okay so broadly it's not skewed so ones and zeros are not skewed in any of our s samples test and train to summarize this video first we filtered the observations that are after 2006 we wanted more fresh observations so we started our experiment with 2007 then we removed any na observation not available observations or missing observations then we randomly selected only we randomly selected 80% of the observations as part of our training data set and remaining 20% were assigned to the test data set finally we also tested whether the proportions of ones and zeros is skewed in any of these data sets that is the parent data the training data and test data and we found that broadly the distributions of ones and zeros is fairly similar close to 50/50 although it's slightly different in the test data set which is 4554 but is still it is okay for our experiment in this video we'll train a linear regression model using the training data set and then comput the classification confusion metrics and performance measures like accuracy specificity and sensitivity and see the model performance we evaluate the model performance so in this video we'll do the linear probability modeling and performance evaluation so first as a first step I'll assign it to a linear object I'll create a linear object where I'll assign the train model remember that we use LM command for train model then up down is are the is the variable relevant variable dependent binary variable ones and zeros then we have sensex as market index and the data that we are using is the train data now this trained algorithm is assigned to our linear object if you want to see the summary of this object you can simply type the summary command and linear and notice that this is able to explain approximately 20.3 5% of the model the adjusted R square is approximately 2.35 so we are able to explain that percentage and also notice the coefficient which is 12.34 that means 1 percentage increase 1 percentage increase in Market variable leads to an increase in probability of approximately 12% in return being positive and similarly we can interpret in a VI Versa manner that 1% decrease in Market variable leads to 12% probability in a negative or zero observing a zero that means negative return so 12% increase in the probability of negative return now we cannot simply model like this we need to assign or classify these results into ones and zeros because we only observe what we call probabilities we do not observe these probabilities we observe ones and zeros we always observe ones and zeros so we need to convert these probability results into ones and zeros and a very simple way to do that is through this if Els we'll extract the fitted values so these are our fitted values inside the linear object we have our fitted values and those fitted values let's say we start with the threshold value of 0.4 we'll use three threshold value 0.4 0.6 and 0.8 so if it is greater than 0 .4 we'll assign a value of one and if the probability threshold is less than 0.4 then we'll assign a value of zero so the FED results are put here in the fitted result object in the next step we'll compute the classification or what we call confusion matrix it's very simple confusion Matrix which is facilitated by the carrat package that we installed earlier and here we need to provide our fitted result in the factor form so using as do Factor we'll create these results in the form of factors so there we have our fitted results and then we'll also provide our actual data so then we have actual data as do Factor actual data so we have our actual data so this is our actual data and we'll put our computed results we'll put our computed results in the object named as CM so the CM will carry our fitted objects now I want to compute my performance measures accuracy sensitivity and so on so I'll create this performance performance let scope for this performance object and this will carry I'll assign in the table format I'll create a sort of trible tble threshold so threshold value of 0.4 and accuracy now please note for accuracy we'll assign a name of accuracy but I need to extract this from the accuracy how do I do it I type it accuracy like this and then I also need to extract the sensitivity so I put a name of sensitive sensitivity I assign again I use that CM object CM dollar by class and I put a name of sense and lastly I also want to extract specificity specificity specificity and for that again I'll use this CM Dollar by class and I'll put the argument specificity specificity so here I will get my performance object and notice I can print the performance object very nicely it has for a given value 0.4 whatever the accuracy sensitivity and specificity that are computed now I need not write the code again I want to compute it for a threshold value of 0.6 I can simply create another object performance _ 6 and I'll use a value of 0.6 I'll run this and all the results are stored we later we'll examine the results first we'll create for eight 0.8 also so I'll create the performance object for 0.8 this is we are doing for linear probability models LPM so we are creating the objects next we'll create 0.8 for 0.8 also very simple we'll create it for 0.8 also now lastly we'll combine all these objects for our future purposes we'll combine this linear linear under underscore performance object and we'll type it as rbind there are three objects that we need to combine performance 4 performance six and performance a now I have done that but I will also add another command where I would like to add a class column we are doing it because in future when we are comparing it with logic proit models I'll make use of this new variable class and we'll see the use shortly we'll understand why I'm adding this but for now I'll just print this object linear performance and you notice uh we have threshold levels accuracy levels and sensitivity levels okay so I also need so I by mistake I forgot one step I needed to add so I'll just add this step here I simply needed to compute every time I do that I need to compute again as well I forgot that so I will add 0.6 here and similarly I'll again add here 0.8 here so I just rerun the commands again so that it's done properly so my model is already there I simply need to add run this and once I run this now we have all the threshold values their accuracies and sensitivities and specificity to summarize this video we create we trained a linear probability model using the train data set and then we using these fitted results we converted them into ones and zeros with our threshold values of 0.4 0.6 and 0.8 then we computed the confusion Matrix and subsequently we computed the important measures of performance which is accuracy sensitivity specificity and then we stored these in our linear performance object for future use in this video we'll train our logit and probit algorithms using the train data set and then we'll compute some performance evaluation measures such as pseudo R square we'll also see their marginal effects and then we'll compute their performance evaluate their performance using threshold values of 0.4 0.6 0.8 by using measures of accuracy specificity and sensitivity so we'll first train our logic performance object now this is quite simple we'll use this we'll assign the train object into logic logit object and we need to Simply use glm generalized linear models for glm short for generaliz line models now I'll not write the formula again I'll simply use the same formula that I used in our linear object remember the formula was dependent variable was up down and independent variable was Market variable now again our data set Remains the Same the train data set only we need to specify the family of models that we want to use and this is the binomial family of models which is logic so our train object is assigned inside the logic variable remember the concept of pseudo R square where we wanted to create another restricted or null model sort of null model where we are not using any variable it's a constraint model so the dependent there's no variable we are using simply putting one to reflect that there's no variable only constant term it will be again trained with train data set it'll its only purpose is to check the pseudo R square remember the discussion on pseudo R square in the video topic goodness of it measures in lesson one so family binomial again binomial and logic so this null object restricted object where there are no variables it will help us in creating our pseudo R square measure now the pseudo R square measure is simply remember the Formula 1 minus log log likelihood log I'll select the log likelihood so on numerator we have log likelihood of our logit object and it is divided by our log likelihood of null which is restricted model let's see what value we have so in the sudo R square we are able to now please note this value we have 11% of pseudo R square but please remember on a standalone absolute basis we cannot interpret much it is more of a relative concept so we need to have a competing classification model logit or proit class competing model which can help us compare we can compare with that but on absolute Standalone basis this is not exactly that intuitive as compared to R square or just R square measures also we have margins package we installed earlier where we can check the marginal effect of the coefficient so we had the market coefficient sensex and it seems that its impact is 15.75% that means if Market increases by 1% on average a positive return the probability of a positive return increases by 15.75% and why Versa if Market decreases or Market goes down by 1% then probability of negative return increases by 15.75% now please remember in order to fit our results we need not write the whole code again what we'll do is we'll simply copy paste the codes that we wrote for the linear model it's quite intuitive and simple we'll just simply copy paste these quotes and see how it is done with our it's quite intuitive I'll simply copy paste these codes very minor modifications are needed so we need not write the full quotes again just have a look at this so here I'm using linear instead of linear I'll just simply use the logic I'll replace it with logic so that is first change so I have put it as logic and the second change that I want to do is similarly in the 0.8 I'll again ha change as well logic and lastly the third fitted results where we are converting the probabilities here we are basically converting the probabilities into ones and zeros like we did for the linear model now I'll just change the name of these performance objects to logic logic performance so and then lastly this logic logic performance and again as we did earlier we'll change this name from linear to logic and we'll just run these commands to assign these values so it's quite easy right we didn't have to write the entire code again we simply copy pasted and that is how it works in our so we simply copy pasted the values and we have our measures of sensitivity specificity ready for us now again now we want to compute it for proit and we want to compute the performance evaluation of proit so we'll do that so at this stage I will not do many changes I'll simply replace the logit with proit so I'll simply just copy paste the entire thing and the only thing I have to do is replace loged with proit so I'll simply replace wherever I have written logit I'll just simply replace it with proit so I'll just type proit here proit proit proit I'll just simply replace this logit object with proit object I'll just replace all the all the places where I've written logic I can just copy paste Pro proit everywhere where we have written log we can write proit so everywhere we have written logic we are simply writing proit and with this we are done so we'll start with the implement this one is left so I'll just type okay so now we'll start with training our proit model it's quite simple I'll just run this command so we'll run the proit we'll train the proit model and then we'll also run this null command to get that restricted model as we did earlier we'll compute the pseudo R square pseudo R square is also there it's around similars around 10.5% more importantly we look at the marginal effects of the proit model which is 14.56% increase in market then the probability of a positive return increases by 14.56% and vice versa that is if Market goes down then probability of a negative return increases by 14.56% now we'll run the remaining commands to store our F fitted results in the objects we as we did earlier for the logic model we'll do it for the proit model Al as well uh interesting to note if I compute the correlation the correlation between the fitted logic dollar fitted values and proit dollar fitted values the correlation is very very high that means almost 99% this is intuitive as well because we were expecting this result uh we the distribution of ones and zeros is very similar across the objects 50 almost 50 50% in such scenarios it is expected that logit and proit will give similar results and therefore we also expect that the performance in terms of various measures is going to be similar for logit and proit approaches to summarize this video we trained our logit and proit model objects through logit and proit algorithms we computed we evaluated their performance performance with Pudo R square measure and also We examined the marginal effects of the market coefficient subsequently we evaluated their performance on meas measures of accuracy and sensitivity and specificity using confusion or classification metrix at threshold values of 0.4 0.6 and 0.8 now that we have computed our performance objects in the next set of video we'll compare their performance in this video we'll evaluate the performance of loged probit and linear objects algorithms by Computing the correlation between them and also through visualization so we'll first compute the performance by Computing the correlation and for that let's create the correlation object correlation is simply cbind correlation object is simply cbind we'll compute the data frame which will include the linear fitted object logic fitted object and also the proit fitted object so we have created our saved our fitted values inside this scoree underscore op and now we'll compute the correlation Matrix for this which is very simple in our core underscore of let's see the control in the console window let's see the printed correlations we can see the correlations are very high in fact 99.9 which we already saw in the previous video between proit and logic fitted values for linear values also it's very high on almost 95% so this is expected generally this kind of high correlation expected when the distribution is very symmetric we have 50/50% ones and zeros for our training data set so such high Coalition is expected but still we need to compare whether which one which one of these models linear log and proit is performing well which we'll do shortly for that we need our measures of accuracy specificity and sensitivity so we'll first as a starting point we'll first combine all the performance objects in our performance object we'll combine all of them by creating a table kind of data frame which is easy to process we'll combine all the three performance objects that is first is our linear remember in the previous video we computed Ed the linear performance object logic performance object and we also computed the proit performance object now that we have combined we can see what is there inside this performance object so all the variables threshold accuracy sensitivity along with their class now here the class variable will be important when we'll be doing the plotting exercise so we'll try to visualize them for visualization visual so we'll visualize them now for visualizing we'll make use of ggplot command it's very interesting so first we'll create the P1 plot GG plot and here we'll have the performance object for GG plot we need to provide the Aesthetics so what exactly we want we want threshold levels now threshold levels can be in the numeric form also but we'll make use in the factor form that is category variable so threshold object threshold object and then we also want to plot accuracy but for accuracy we'll use the numerical properties accuracy so this is our accuracy then we would like to print because these are simply points so we'll like to make use of Gom Point while plotting them so Gom point will ensure that we have a plot of and now here notice now in this Aesthetics we are providing color to be segregated as per the class and that is why class variable that we added this class variable that we added is important because this will help us segregate the plots in terms of their different classes so this is first object in fact we can print it right now and we can see that P1 object you can see there accuracy is printed three objects notice the accuracy while we can examine them later on but just to give us a flavor you can see how they are plotted but we'll compare them later on then similarly like P1 I can also plot P2 I have P2 object I can create the graph object P2 where instead of accuracy I can use the variable Sensitivity I can simply just copy this sensitivity variable here so this is my sensitivity I'll run store this in P2 and the last one is the specificity P specificity so here I'll use the specificity object so here P3 I'll use this specificity object I'll just copy it one more thing I need to adjust for the X and Y axis so I need to add those x file access lab commands so for example I need to have let's say xlb in my accuracy object I'll have the xaxis always remains the threshold and Y AIS here y AIS let me use next line for this so that will be better make it more clear so Y axis I'm using as s accuracy because this is P1 is for accuracy so I'll use accuracy here I'll just copy paste this sorry I'll just copy paste this and make use of in the next P2 we have sensitivity so instead of accuracy I'll replace it with sensitivity similarly for P3 I have specific spe it so I'll add specific City now I have all the three plots I'll again just run these commands to ensure that we have proper P1 P2 P3 now it's time to plot them and very intuitive very nice because we had one Patchwork Library if you remember we added which will help us plotting these objects like this P1 P2 P3 they'll be very nicely plotted and notice they are there very nicely lined in rows and now we can compare the performance so just notice the logic object which is the green window is always most of the times it is higher on the higher side while the values are very close to each other which which is which we also saw in terms of the correlation number the correlation was very high in the fitted values nonetheless the green one the logic seems to be doing better most of the times the green one you know especially on the accuracy level it is particularly High sometimes there is change in the sequence but it's still most of the time the values of the logic for accuracy and sensitivity is reasonably High except a few scenarios for example here the linear one is doing well in terms of sensitivity of 0.6 and so on but mostly it seems the logic model is doing well to summarize this video we computed the correlation across all the three linear logic and probit models fitted value then we plotted them for the three threshold values of 0.4 0.6 0.8 we noted that while the performance on all these measures are very close for all of them nonetheless logic model seems to be doing marginally better than the other fit models we were expecting this close performance this close Callum performance because the correlation across all the three linear logit and profit fitted value seems to be very close to each other [Music] [Music] [Music]