Lecture 26 : Classification Algorithms: Application (Part 2)

[Music] in this video we'll compare the performance of linear logic and proit algorithms with the help of Roc curve recall our discussion about Roc curve in the lesson one video topic Roc curve characteristics R curve so first we'll create a performance object with prediction object prediction and we'll start with the linear algorithm so linear model that we fitted fitted values and also we need to supply the train training data up and down value values so this will create a prediction object and from this prediction object we'll create we'll create our performance measurement object with with the help of performance command PR and we need as a measure we need true positive rate or sensitivity on as our y- axis and we need on x-axis we need false positivity or one minus specificity as we have discussed in the ROC characteristic curve so we'll run this command so now I'll plot the ROC command it is a very interesting measure to compare the performance or evaluate the performance so I'll plot with the help of this plot prf so when I plot this notice this is the plot I'll again clarify it's slightly smaller so we need to have proper PR plot so this is our prf plot now a good evaluation of this prf plot can be done with the help of Center Line which is 45 Dee line which gives 50/50 probability so I'll use this 01 which is basically 0 to one line and remember this if you recall this Roc curve essentially measures different values different coordinates of tpr true positivity rate and false positivity rate at different values of threshold different threshold values so it gives a complete view of the performance so we have line type two and not is the line type now we can see that this model is somewhere between 50% line as we saw and the complete 100% accuracy it is somewhere in between but in order to know the exact accuracy number we need to compute the area under the curve so for that we'll use our performance object PR and the measure is area under the curve so we'll assign this Au object now with the help of this Au object we can simply access the area under the curve with this command and is around 72% so this is our area under the curve which visual through visualization we Al we can also see the performance let us also calculate this for the logit object we'll visualize also so the same thing we'll do for our logit object let's do that so for logit object we get the plot looks like this again very similar plot in fact it appears to be very similar to the linear object but we can compute the numbers again very same values we are getting very similar values which was expected because of their High correlation almost 95 to 99 correlation let's do this for proit also now so for proit I'll do the same thing so Pro proit object let's plot again almost identical graph we are getting so if I compute the AOC again the same very similar values notice that we observed exactly identical value of are at the curve so could it be the case that we have done some mistake in Computing our fitted values are these some result of some computational error let's check the summary of these fitted values so I'll compute the summary linear dollar these are fitted values so just compute the summary of linear model then we'll do for it logit also logit logic dollar fitted values and we'll see the distribution the numbers are very close in fact to the extent of four four digit they are close but they are not identical as we can see they are not identical they are very close to each other we can see for proit also so you can see there they are not identical but they are so close they're so close in terms of their distribution the minimum first quantile median and mean and this was also born out by the fact that the correlation between them is almost from 90 95% between linear and loged proit and between loged proit almost 99.9% so because of such closeness such high correlation the classification is also the ones and zeros would be very similar and that is the result that different thresholding values the area under the curve the accuracy of performance is almost identical so area under the curve Roc curve is almost identical area for Roc curve area under the curve is very identical quickly to just summarize the the video we plotted Roc curve for all the three linear logit and proit objects and we noted that area under the curve for OC curve is almost identical this is ascribed to the fact that the correlation between all these the fitted values of all three algorithms that is linear algorithm and logic proit algorithms the correlation is very high and because of that the area under the curve the performance measure is almost identical in this video we'll develop a simple machine learning system which will help help the computer learn how to select the best classification algorithm across a class of algorithms so we'll develop a simple machine Learning System very often we would have heard of a cost function to be minimized in the context of machine learning algorithms in this particular case we'll talk about a simple performance function which we'll try to maximize the idea here is to develop a performance function a Well Suited performance function to the algorithm to these classification algorithms and see which one of these algorithms help us maximize the performance function so let's call this performance function as op op for optimize and let's define the parameters of this function so so in this function we'll try to simulate some numbers we'll use our F values we'll use the actual values and we'll use different threshold values so we'll simulate different fitted values for a given threshold compare them with the actual values and try to create a performance object which will be maximized across different algorithms so first we need to use these fitted uh values we need these fitted values so for that as we did earlier we'll use the very simple if else function the idea is to using these threshold values create ones and zeros so actual fit values are in the form of probabilities if you remember so we need to convert them into ones and zeros using our performance evaluation function once we have that recall that we computed confusion or classification Matrix object and in that we we'll create that object again it's a very simple object so we'll just use this command cm equal to confusion conf Fusion Matrix in fact we can simply copy paste this remember we have already typed this so we need not type this again we can simply use this we already have computed this so I can type this command so this command will help me extract all the three relevant objects that is accuracy specificity and sensitivity so we'll use this also because we are using a number of zero and one as for to the threshold we need to tell that there are two possible levels to this variable so two possible levels that are 0 comma 1 so because it may give us error from my past experience I know that it will give error if you do not specify levels because for 0o and one there's one extreme case where all the fitted values are either Z or one so in that case we may get an error to avoid such error we simply Define the levels in advance now with the help of the CM object I can obtain all the three relevant measures assuming that specificity accuracy and sensitivity are the relevant measures we'll Define our complex combined performance object and this can be anything you can Define you can give any weights or any complex function for your performance Matrix of accuracy specificity and sensitivity but to keep the elaboration simple we I simply use this average simple average of all these three Matrix what are these three Matrix these three metrics are of overall performance are accuracy sensitivity and specificity so I'll just use these three metrics so it's one is accuracy one is accuracy I'm simply using their averages just to demonstrate the point you can think of you can imagine any complex performance function depending upon your taste choice and preference but to keep the demonstration simple I'm using their simple averages just the average of three measures which is accuracy sensitivity sensitivity and specificity so I'm using the simple average to test my train algorithms and tell my machine or the machine learning process that this function has which the algorithm whichever maximizes this function need to be considered so specification C now the idea is through this function I'll simulate a large number of threshold Valu I'll simulate a large number of tow or threshold values and then I'll generate number of performance measures for different algorithms and then I'll see which one of these algorithms is able to maximize or provide the maximum level of performance across competing algorithms to summarize this video we created an optimization function where the arguments that is fitted value actual value and threshold values will be passed and this will give us our performance or objective function which is simple average of accuracy sensitivity and specificity the idea is to simulate a set of threshold values and through which we'll generate a number of performance measurement values for different Computing algorithms in the next set of videos we'll generate these performance values for our three competing that is linear logit and probit train models we'll generate these performance values and we'll see and compare and contrast which of these models are able to generate better performance in the previous video we defined our performance objective function in this video we'll simulate thousand threshold values and then calculate the value of our performance objective function for for each of the classification algorithms that is linear logit and proit models so we'll compute the performance objective performance function as a starting point let's start with the linear performance object performance object of the linear model so how do we do it first first and foremost we we have a remember the function we have the fit object we'll assign the linear dollar fitted values so these are our fitted values then we also need to supply our actual values which are trained dollar up down so these are actual values we'll start with the threshold value of zero so that is our starting point and we'll create a performance linear object so this is our performance linear object where we'll assign the values here we are putting we are putting the values of to so TOA we have as a sequence from 0 to 1 so our to values are from 0 to 1 in the sequence so minimum is zero maximum is one and of we need to give the length so length is 1,000 so th000 is the length of our assigned values in fact we can remove to equal to0 from here we already defined tow from 0 to one and then the performance object we initiate the performance object with a value of zero later these values will be filled when the value is calculated as we'll see shortly so we have created a performance linear object now in the next set of quotes we'll run a for Loop a very simple for where we'll simulate the values and for each value of threshold we'll compute our performance object value so notice a simple follow Loop it starts from 0 to th000 now the code is very simple here it's a very simple code just notice so I have my performance object performance undor linear object and in this performance for each I remember the I moves from 0,000 in each I I'll assign the value computed from my optimization function where I Supply in the optimization function here I'm supplying the fitted values fit that are the fitted values by the linear object then my actual values and then here I provide this threshold simulated threshold value so notice this I moves from 0 to th000 and it is divided by th000 so I Supply th000 values simulated values which will be used to compute the value of performance object now I run these values so we'll run our for Loop here and notice now that the loop has run it will simulate a set of values you can check also you can run this you can print the simulated values in the performance linear object so we can check the head of our performance linear object we can also check the TR tail of performance linear object since they are varying from around 50% from smaller to larger values but we'll compare with the actual thing so this is how you compute the performance object for the linear algorithm similarly very similar manner we'll compute it for logit object for the logic module now I'll not write the code again I'll simply just change the small line of code here to ensure that these are my logic fitted values and then here instead of performance linear I'll type performance logit I'll run this function and just replace the performance logit here and I'll run this Loop so this will assign in the performance logit function it will will assign the values similarly I'll create a performance proit function where I'll assign the proit performance proit model so I'll use this proit model here I'll change the names from proit here and proit here so I'll just run these codes proit I'll create a probit object here also when I'm simulating the threshold values I'll run the for loop with proit objects to summarize this video we created three performance object first is the linear logic and probit using our optimize function which we created for measurement of our performance we simulated thousand values of threshold and for different values for each object we computed the performance objective function value and assigned them into a performance object of linear logic and provit in the previous video we computed the values of our performance object for all the three classification algorithms that is linear logistic and profit using our simulated threshold values in this video we'll try to compare the performance through visualization so we'll compare the performance through visualization as a starting point let us plot the values of the performance object for logic object so we'll plot the logit object first it's tow values threshold then the values of the performance for different threshold values we'll plot as a typ line color red line withth of two seems to be appropriate we'll not name the y axis we'll keep the y axis as empty x-axis is common so we can give a name of threshold and the central heading is performance so the central heading is performance let's plot this object so we can see the object that is plotted now we'll superimpose other performance objects for first for the proit again same we can use the same commands only that we change the color to Green to make a Extinction other things remain same line width and so on now we'll just copy paste this and instead of probit we'll use linear here and the color will change to Blue to make a distinction let's superimpose these now it would be nice if we can add a legend also so we'll add a legend on the probably top left seems to be empty so we'll use top left corner to add the legend so that a user can see the distinction Legend So we have three items logit proit let's make get Capital log it proit linear then we need to add the color also through fill fill feature red logit is in red proit is in green and linear is in blue so with this we add the legend finally with this our plotting will be complete now notice the performance objects here for all the values in fact for most of the values the red one seems to be offering in terms of performance higher value so the red one attains Higher values for most of the threshold values and red one is logic next is the proit which is green one although they are very close but still for most of the cases logit is higher linear is the lowest one they are pretty close as we said this is expected because the distribution of ones and zeros is pretty symmetric 50 50% but nonetheless logic performs better then second number we have proit and third we have linear lastly if you want to extract a particular value for example we would mostly be interested in the maximum performance value and what threshold it is to train our algorithm so we'll be extracting it very easily in our notice the command so I'm extracting that performance value which is equal to Max so the maximum value we are extracting here just taking the maximum value and we'll extract that so along with its threshold as well so the maximum performance is 0.672 performance object value at a threshold of 0.53 536 so uh so an algorithm which is trained through this manner would probably be better place to have a threshold level of 0.536 and at that level the performance object will give a value of 0.6 72 to summarize this video we computed various performance object values for different classification algorithms that is linear logit and proit for our thousand simulated threshold values we plotted all these performance values and superimpose for all the three classification algorithms and we noted that for most of the values logic algorithm performs the best next proit and finally the linear although the performances are close to each other nonetheless for most of the threshold values logic seems to be the best algorithm for classification in this video we'll start with out of sample fit performance evaluation recall that in the discussion around linear regression modeling on the video topic out of sample fit performance evaluation we have already covered a certain number of measures such as error measures like rmsc RSC and so on so we'll not repeat that discussion here we can refer to that video so as a first step we will using our trained algorithm on the linear object linear performance object will create the test object first remember this was our linear object it was simply computed as linear equal to LM LM up down variable and it was modeled using sensex with the training data set so this was our linear object if you recall so we run this command this was our linear object now we'll use the same Trend so this is our Trend object linear and we'll now work out with the test data set the command is simple predict we'll use our linear object but this time the data that we'll use the new data is test data so this is our predicted object the values are contained in predicted linear object next we'll train our logic performance object logic performance object again remember we computed the logic object with the simple glm M glm we'll use the same formula formula for the linear one so we'll extract the formula from there and we use the train data set to train our algorithm family of binomial models under logic so this was our logic model train model using train data and now we'll compute the test object using test data PR logic and again we'll use the predict command this is our Trend object which is logic and we'll use the new data which is test data again the type is response this is for logic probit type of variables sorry the response type is equal to response for binary variables so this is our predicted logic variable next we'll generate our proit object proit performance object again similar command prop bit we'll first train our algorithm in the similar manner glm so in fact I'll just copy paste the previous commands and just make few spelling changes and save time so I'll just copy paste it here notice is here now I'll change it to proit the object will be change to proit and in the family I'll change it to probit here so I'll run this command again we'll change here the predicted object the trained object is proit so we'll change it to proit here also we'll change it to proit and remaining things will remain same so we'll predict this is our predicted proit object as a summary aspect we can check the summary of predicted object for example we can check the summary of of predict linear or if you want you can check the summary of predict proit and so on lastly we'll just combine them combine all the predicted object let's call it table cbind data frame we'll combine our predicted objects linear predict logic predict proit so we'll combine them in the Consolidated table all the three objects that are there in fact you can check the correlation also if you want you can just check the correlation across all the three objects it's pretty high between proit and logit we can see very high correlation that means we can expect similar results and similar performance results with logit proit the correlation with linear is less still it is high at 91% but still much less as compared to between and profit in this video we computed the predicted values for all the three classification algorithm using our trained object so using the trained object we predicted the values for all the three algorithms and we noted that in the predicted values using out of samp test data which is out of sample fit the correlation between all the three algorithms predictions are pretty high in fact for logit and probit it is close to 99% while with with linear object it is 91 to 92% this is ascribed to the fact that in our data ones and zeros that is positives and negatives are pretty symmetric that is 50/50% both in test train and original data set therefore we expect the overall performance of logit and probit to be pretty similar and that of linear also to be close to log and proit models in the previous video we predicted the values using the trained model with the test data set in this video using these predicted values for all the three algorithms we'll compute the ROC curve Roc curve and compare and visualize performance we'll also compute area under the curve to compare the performance of three algorithms please note that a detailed discussion on R curves have already been take done in lesson one video topic Roc curve also a detailed R implementation of these Roc curves has been done with the trained data so we'll not have a detailed discussion we'll keep the discussion brief and we'll focus on the our implementation part so we'll not have a detailed discussion we'll keep the discussion brief and we'll focus on the our implementation part so we'll first create a prediction object as we have already done similarly in the with the training data set for the first for the linear object and we'll compare it with the test data Act ual up down values so this is our prediction object using this we'll go to the performance same as we did earlier PR and we use on x-axis we have Pro positivity rate tpr on sorry on Y axis which is also our sensitivity and on x-axis we'll have false positivity rate which is our 1 minus specificity so this is our performance object we can plot this performance object very well so we can plot it as we can see here along with that we can also draw the 50% area line as we did earlier so this is very simple using a line command 0 comma 1 line width of two U maybe line type of two we can give it a color of maybe red and this is the line one can also compute area under the curve very simply Au equal to Performance and will supply our prediction object PR along with measure which is area under the curve so this will store in AU I can simply extract the area in the curve with this Au y do values which is around 71.5% now I'll not write the quote again I'll simply copy paste these and this was created for the linear object linear performance object similarly I'll create cre it for the logic performance object logic performance object only I'll replace from linear I'll replace it to logic and I'll just compute the performance now we have the plot here uh this is theine the values are expected to be pretty similar given the high correlation we are expecting High values similarly I can compute it for proit as noted earlier we may expect a very high similarity predict proit so this time we'll be using proit object and I'll compute so again pretty similar value this is this is ascribed to the fact that the prediction the probabilities may be different but ones and zeros will be almost identical so we are expecting a very high similarity given the very high correlation in the fitted predicted objects we saw the correlations were in fact more than 90% so we are expecting very similar performance as we can see here around 71.5% area under the curve so as per Roc the performance is quite similar but we'll create our complex more complex performance object to see how they are comparing with each other and contrasting with each other to summarize in this video first we computed the RC receiver operator characteristic curve and then we also computed area under the curve we find that given the high correlation across the fitted objects the area under the curve and performance as per Roc characteristic curve is very similar in the next video we'll use these values our Trend algorithms and then compute our objective performance function for which we have already written the user defined function and we'll say using the simulated threshold values we'll compare the performance for all the three algorithms linear logic and it recall that we have already set up a performance object which was simply the average of accuracy sensitivity and specificity Now using our predicted values for all the three algorithms we'll compute various performance object values for thousand threshold values that we'll simulate using the user defined function that we have already created so we'll start with the performance object for l model this quite simple we have already seen that in with the train data so we'll not repeat the discussion in great detail we'll simply execute it so this is our fitted values predicted values using the test data and trained algorithm these are our actual values test dollar these are actual values and then our performance object let's use this name performance linear we'll keep the same name and we'll we'll assign a table which is nothing but a sort of data frame object tow will be a sequence remember the tows here will be sequence of 0 to 1 and we'll keep the length th same as earlier length is th000 so th000 threshold values will be created from 0 to 1 at equal intervals and we'll initiate the performance object with the value of zero so this initiation value is zero so this will be our linear performance object now recall we'll start with the for Loop where I is the counter and 0 to th000 so we'll start with 0 to th000 so we include 0 and one so that 0 and one both are included in the threshold and now performance underscore linear in this we'll assign the values of perform we have initiated a value of Zero by default so by default there's a zero value each time this for Loop will run one new performance value we assigned and this op user defined function we have already created linear dollar fitted values in fact I can simply use the fit we have already assigned to the predicted values into fit object actual values are also assigned here and then I upon th000 so this will be our performance object I'll run this so this performance linear object will carry my performance object values through simulated threshold values in a similar manner I need to create it for the logit object so I'll just copy paste this code I'll not change it much so I'll use it performance object for the logic model only thing I need to do here is I simply need to just change this to logic and rest of all it will also this one as well logic this one also needs to be changed other than that okay this one also needs to be changed so now I'll run this entire segment so this will also run in the meanwhile while the loop is running I'll also create my proit object so this is for proit again I'll make the same changes proit here here the performance object is proit and here also it is proit so now I have computed the values of my performance object for the simulated threshold values for all the three classification algorithms to summarize in this video we computed the value of our performance object using the predicted values for all the three algorithms with simulated thousand threshold values in the next video using these performance object values we'll comp compare the out of sample performance of the all the three algorithms and also visualize the performance in the previous video we have simulated thousand performance object values for our test data in this video we'll compare the performance of all the three classification algorithms using these simulated values since we have already done this exercise for training data set we'll keep the discussion brief and focus on the implementation so we'll compare out of sample performance first we'll plot in the similar manner we'll plot the performance for the linear object to Performance linear object for the performance again same commands we'll keep the type as L line color as red line width of two seems to be appropriate here y lab access y label will keep empty again X lab will keep as threshold name which is a appropriate and the central heading will keep as performance so this will be a with this we'll start plot and we can see that performance plot next we'll to this we'll add robit performance object so just to maintain consistency I'll use logit here next we'll start with the proit object again I'll I'll use the same set of commands performance proit performance other commands we keep same type l column Red Line bid two we'll change the color to Green we'll maintain the consistency here for consistency we we'll run this as well and next we'll post linear so linear and I need to change this to logit as well sorry here take linear I'll keep the color blue so I'll again replot because there was some error so I'll replot it so this is the these are the plots let me add the Legends here as we did earlier so we'll add the legend again the top left seems to be empty so I'll create the legend at top left names of Legend we give the name same as earlier logic proit and linear color scheme Remains the Same so we'll use that again fill logit is with red proit is with green and linear is with blue so this is my Legend So the legend command the legend has appeared now notice very identical to our train data set test in test data set also we knowe that the red one is the most outer one that means the performance values which is the average of accuracy sensitivity and specificity is highest for most of the probability region most of the sorry threshold region it remains higher for logit in red proit in green and linear in blue so for logit it is highest proit is very close but is still lower but very close to it which is expected because of the very high correlation in the fitted values linear is the lowest linear the blue one is the lowest although relatively it's not that lower also but still it if one has to choose one will choose the logic and next profit lastly if one wants to finally select the best train model one can simply as we can see the logic is the most optimal model here from our projected object so and if you want to select that particular observation and threshold value as well as we did earlier we'll simply choose this logic dollar performance and equal to maximum value so we'll select the maximum value for our performance object and this will help us extract the corresponding threshold value along with the performance object so I'll just run this and again we can see at tow of 0.569 which is the threshold value we get a performance object value of 0.659 so this is the maximum performance value which we obtain for the logit object and logit is supposed arguably the best classification algorithm at to value of 0.569 to summarize in this video we plotted our simulated performance object values we observed that for most of the threshold region logit model seems to perform best as compared to other two provit and linear it is closely followed by proit and then linear we also extracted the best performance threshold value for logic object which was around 0.569 and the performance object has a value of 0.659 which is simply the average of three accuracy sensitivity and specificity values for the model to summarize this lesson we modeled ABC stock price up down measure that is a dummy variable which is one for up and zero for down movements using linear logic proit classification algorithms first we trained the model using the training data set and examined various measures of model performance evaluation these included parameters of accuracy sensitivity and specificity derived from confusion classification Matrix we also visually examined the model performance using Roc curve and computed the area under the curve next we created a simple algorithmic system by simulating thousand thresholding values and computed a comprehensive model performance object using parameters namely accuracy sensitivity and specificity with the test data to examine the out of sample prediction efficiency of the model overall the logic model seems to have provided better fit and model efficiency followed by probit and linear models we also computed the marginal effects of the independent variables on the dependent variable for logic proit class of models and discuss the interpretation of the same [Music] [Music] [Music]