Lecture 11: Missing Data Handling

Hello friends welcome to the course multivariate procedure with r now in this lecture I am going to introduce A New Concept this is about how to handle the missing data what is missing data whenever we are trying to conduct any experiment in which we are trying to record some data then it is possible that some of the data may get missed that may happen during the experimentation or in some cases after the experimentation also for example if an experiment is conducting a clinical trial in which Suppose there are 20 patients which are given some medicine and they have to report to the clinic every day in every day they will be giving a dose and next day their recordings will be uh recorded in the clinic now suppose uh there is one patient which starts taking the medicine on first day second day third but on fourth day the patient does not report in the clinic and the patient uh comes again to the clinic on the next day in this case what will happen the data on that particular day will not be available and this may happen on multiple days and with multiple number of patients in another example suppose uh we are trying to conduct a crop cutting experiment means in some agricultural field some crop is there and we want to experiment uh in that field for that we have to cut some of the crop or we want to estimate that how much crop is going to be produced after the season and suppose one day some cattle some cow comes and the cow eat it'ss some of the part of the field now what will happen there is no CR in that part of the field but the remaining part is available so in these types of example what is happening that uh some part of the data sets gets missed now you have two options either you try to ignore all sets of data and you try to conduct the whole experiment once again in some cases it may be possible in some cases it may not be possible but it will always always be an expensive Affair which will take more cost more time more energy more label more labor so now what to do the next option what we have that whatever data we have we should use it and try to ignore the missing part well in statistics we have got a concept of missing data models in which uh we try to estimate the missing value on the basis of given set of data and we try to imput it back that means whatever the value we we are trying to know on the basis of given set of data that is replaced back in the sample Now sample looks complete as if there was no missing value and then we try to uh conduct our Staal analysis but definitely we are not going into that aspect in this lecture but our main objective in this lecture is that how to handle the missing values in the data and how we can handle them how we can compute our statistical quantities when there is missing data so what I will do I will try to take uh different types of example in this lecture I'm going to explain you how are you going to hand the missing data then in the next two lecture I am going to repeat the same concept what we have discussed earlier on mean and variance or the measures of central tendency and measures of variation that I will try to show you how you can compute them when we have the missing data now I will try to show you here in the case of univ various setup but without any problem these things can be extended to a multivariate also so my OB here is to give you basically the idea that how are you going to handle it what are the commands and the same commands can be rep replicated when you have more than one data vectors or matrices or data frames or data in any other format and wherever we are trying to use the multivariate procedure okay so now let us begin our lecture and we try to understand how we can handle the missing data in the our software okay so now about this missing data handling so in our software the data which is missing that is indicated by capital n capital a n a right different softwares may have different types of symbols and notations to indicate it but in R we use the notation here capital and capital A and A the the first question comes whenever data set come to you usually as a statistician you may not always be be compil fing or collecting the data but there will be somebody who will be conducting the experiment and they try to bring the data to us in some format maybe in some CSV file Excel file Etc so now our first step is the how should we know is there any missing data because if there is no missing data and if there is some missing data then our Command for the statal procedure they have to be modified so in order to know in order to DET whether there are any missing values or not we have a command here is do na a is do na all in lower case alphabets so whatever is my vector or the data set that we try to write inside the parenthesis and we try to write is do Na and inside the par the data vector and it outcome will be a logical Vector that means the outcome will come in terms of true or false right and in all the places where the data is missing that means there is value n a this is represented by true right for example just to give you an example here suppose if I try to take here one value here x which I'm taking it as Na and if I try to use my command is do n ax then it will come out to be here true so now whatever is the data Vector wherever I am getting the value here true that means the data is there right so you can see here that when I'm trying to use here xal to na this is assigning na to a variable X and why I'm trying to operate this is do na it is giving me here true so is it uh it is actually trying to know is any data missing is it missing right so now let me try to take here data vector and try to show it suppose if I take a data Vector of four values say like 11 133 and two values here Na and na they are missing I'm trying to intensely take it so that I can show you the outcome so that you can understand it easily now when I try to operate here is do na inside the parenthesis X you can see here this is my ha outcome false true false true and whever it is here true you can communicate it to the na value available in the data vector and variable we have here no missing value you can see here 11 is indicated by here false and this 13 is indicated by here and this is here the screenshot right so you can believe at the moment that okay it is going to be correct so this is how we try to take the data Vector now let me try to show you what are the consequences if you have missing data that will give you an idea and this concept can be extended to any statistical measure I'm just trying to illustrate it through a very simple command mean mean we understand that arithmetic mean so if I try to take this dat and if I try to find out the arithmetic mean so basically we are trying to find out 11 + na + 13 + na / 4 and this will come out to be here the value na a when we try to operate it in the r software but do you think that this is what you want certainly not because if there is a data Vector which has got missing values and if you don't know and if you try to compute their mean it is going to give you the value Na and then it will be used in further in the programming and you will not come to know that what is really happening but if you try to understand what do you want suppose you want that okay whatever are the missing values they should be removed from the data Vector for example this and this I am removing it and whatever are the values available in the data set we would like to find out the arithmetic mean of those values so in order to to execute it we try to use here an option n a do RM is equal to true we can say that n a for missing value and RM is used to remove that means do you want to remove the missing values I'm trying to say true yes please remove so now if you try to compute the mean of X with using the command na. RM is equal to true then the outcome will come out to be here 12 which is obtained here 11 + 13 / 2 which is equal to here 2 so now you can here that how it is being operated yeah as I explained you in the beginning in the statistics we have some procedures which are under the missing data models in which what we try to do suppose this is my data Vector C 11 and a 13 na a then we try to use some imputation methods and then we try to estimate missing values based on on available data then we replace them so what will happen here now the new X star will become here something like a C1 now there will be some value here not Na and then 13 and then there will be another value here in place of Na and these are going to be some numerical values values obtained through computation procedures one common imputation procedure is to find out the arithmetic mean and we try to replace all the missing values by the arithmetic mean so for example in this case you can see that the missing Val value is Na and mean of the remaining values here is 12 so one option is that I can use here in place of this missing value I can use here 12 and 12 now this new data Vector X becomes here C 11 12 13 12 right but and then I can use my mean over this thing but but then definitely as I said earlier I explained you this thing because in case if you want to use such missing data models they are available in the art software and you can use it also but here I'm not going to use it my objective is simply to explain you how you can diagnose if there are missing values and how you can find out their position their details and how you can modify your hair thatal computation in the art software right so let me try to this illustration if I try to take here x equal to Na and if I say here is do na X you can see this is coming out to be here true and simly if I try to take here another Vector here C 11 13 14 you can see here the mean of X comes out to be here 12.5 on the other hand in case if I try to remove here two values and if I make it here Na and na even one value also it doesn't make any difference then if you try to find out here mean of X this will come out to be here na but now in the same command if you try to use here na. RM is equal to true then you can yeah there is one typographical mistake here like this you can see here that the mean of X is coming out to be a 13 which is here 12 + 14 this is 26 divided by 2 which is here 13 right so you can see here if you try to make any slip or mistakes then R gives you an error that it cannot move forward on the other hand if there is a warning message that mean it will move forward anyway so if you try to see this is how you can handle this missing values and then yeah if Staal command will have this option which you have to give in the parenthesis and my advice to you all will be that whenever you are trying to use any statical procedure please try to look into the help menu and try to see that how the missing values are going to be handled when we are working in the r software then we have two concept one is is here Na and another here is null right and they are somehow in some common sense I can say that they are trying to indicate that the values are not available values are not available means whether they are missing or not available there is some difference right so in R you will see that both This na and n n l upper case alphabets both are available and sometime you will get outcome as null so let's try to understand what is the difference between na n n right actually this outcome null is written by some of the functions and expression when you try to execute it but any and null they are not the same any is a placeholder for something that exists but it is missing whereas null stand for something that does not exist that never existed at all for example if I try to take an example and then you will understand it clearly Suppose there are two persons person number one and person number two and both of them appear in the admission test in a school now person number one gets selected and person number two is not selected now if you try to look in the attendance register of the school whose name will be there person number one or person number two the attendance register in the school will have only the name of person number one and person number two will not be there because that person number two is not admitted in the the school so now on a particular day if during the attendance this person P1 is absent then the attendance will be marked as here and here and when we try to take the attendance of person number two P2 this will be marked here as a null because person number two does not belong to that school or that college so this is what I meant that null stand for something that never exist exed at all so person number two never existed in the school so the name of person number two will not be present in the attendance register so it will be marked as null so this is the basic difference between Na and null right okay now let me try to address another aspect of missing data that whenever we have some missing data then we would like to know what is the location of the missing values or we want to identify the location of an a for that we can use the command which wh I and inside the parenthesis we have to write is do Na and then inside the parenthesis data Vector for example if I try to take here the beta Vector X like this C 11 na 13 na so these two values they are here missing so now if I try to operate this command here which inside parenthesis is na inside parenthesis X so now you can see here what are the locations on which we have a missing data if you try to look into this data Vector here x then if you try to write down the position of different value this is in position number one this is position number two this is position number three and this is position number four 11 is at position number one and a is at position number two 13 is at position number three and na here is at position number four now if you try to execute this command it is giving you here 2 4 what is this 2 and four that is what we have to understand this two is indicating and this four is indicating this four so the command which is NX is trying to indicate that the missing values are at second and fourth places right this is how you can understand and this type of information can be used further right now suppose if you want to operate any procedure or any command over this thing suppose if you want to find out the sum and you want to know that how many missing values are there for example so in order to count the number of missing values that is number of Nas we use the command here sum with inside the parenthesis using the command is do na so if I try to write down here sum and inside the parenthesis is not na then you can see here what will happen let me try to take the same data Vector 11 na 13 na so this is this thing so now if you try to see at how many positions the missing values are happening first position and this is here the second position with to so total number of missing values are of missing values they are here two so now if you try to see here sum and inside the parenthesis is do any inside the parenthesis X then it will give you here the two so this is indicating this two that there are two missing values in this data set right so let me try to show you these commands on the R console here so that you can see here so let me try to copy the command so that I don't make any mistake in the typing okay so now you see here let me and this is your here like this if you try to see here which is so now if you see here that the missing values are occurring at first and third position if you try to enter you will get here right and so similarly if you want to obtain here the sum of the these values that how many values are here missing then if you try to see here this will is giving you here 1 three now similarly if you try to take here means if you try to increase here some more vales here na comma na if you try to see here and if you again try to find out here this sum it will come out to be here four because there are one 2 three and four there are four values in the data Vector which are missing right so this is how we try to find the count now beside those things there is another objective which you would like to achieve suppose you want to know the complete cases that mean what are the values where the data is available me up to now we were trying to find out the places or the data which are missing now I am trying to concentrate on the data which is available right so for this I can use the command here complete dot c o m p l e TS e do CES and inside the parenthesis you have to write down the data Vector so this will return a logical vector and it will try to identify the rows which are which have the complete cases and wherever we have the complete cases it the outcome will be a logical Vector it is something like true and if the value is missing then it will come out to be false so let me try to take the same example C 11 na 13 Na and if I try to operate here complete this is inside the parenthesis X then this outcome comes out to be here like this so if you try to understand this thing in the data Vector X there are two places where the data is available and when you try to see the outcome of complete dot is this 11 is indicated by here true and this 13 is indicated by here true and similarly if you try to take here the This na value So This na is indicated by here false and this na is indicated by here false right so by looking at this outcome one can see that the values at the first and third places they are available they are not missing right and now after this I want to go one step further there is another way out to handle the missing value suppose I want to ignore all the missing values and I want to consider only the values which are complete yes yes so up to now what we have considered suppose we have a data Vector which has missing values so we use the command na. RM is equal to true and after that whenever we are trying to execute any statistical measure or any function of r it is trying to remove the na values and it is trying to give the outcome now our objective is this we want to create a new Vector new data Vector after removing the missing values from the original databas so now how to do it so for that we have here the function n. n a.it and inside the parenthesis you try to write down the data Vector it Returns the objected the object with listwise relution of missing values that is it will drop out any rowes with the missing values anywhere in them and forgus them forever let me try to explain you this thing with this data set I try to take the same data Vector X C1 na 13 na like this now I try to create here another data set y by writing n a do omit X so what will happen now this Na and na will be removed from this x data vector and we will get here only 11 and 13 which is here like this after that we have here some action is trying to tell us that this na values are available at second and fourth position and it class is omit but anyway we are not going into that part but now in simple words we have removed the missing values from the data Vector X and we have obtained here a new data Vector y now if you want to make any operation on this data Vector y this should not be a problem for example now if I try to find out here the arithmetic mean so we have here x which has missing value we have here Y which has no missing values right now if you try to find out here mean of X and mean of Y then you can see here that mean of X of this x comes out to be here Na and mean of this 11 and 13 comes out to be here 12 like this here right so let me I to show you all these things in the r software also so that you can understand it very easily let me try to show you here this here xal to here C 11 say n a 12 and here n so now if you try to see here complete cases in X we have complete cases 11 and 12 which are available at first and third position so in this outcome first and third position have true and second and fourth position have here false so that is indicated by this thing now in this data Vector 11 and 12 are present they are available so it is here right and now if you try to find out here mean of here x this will be here Na and if you try to find out here mean of Y this will be 11 + 12 / 2 which is 11.5 right so now I have given you this sufficient examples to explain you uh the concept of missing data that mean how to handle the missing data in the art software well I have taken here a univariate case with only three or four values which I can illustrate and whenever we are trying to think about the multivariate procedures the multivariate procedures you will see that they are combined with the univariate procedures so now whatever I have told you here for handling the missing data this concept will continue to the multivariate procedures also whatsoever we are going to take there will always be an option that how are you going to handle the missing value in that contacts such contacts my advice is that if you want to know the best place is to look into the help of this of that function suppose if I want to know that how to handle the missing value in means uh suppose I want to know how to handle the missing value in variance or quantiles Etc the best is to look into the help menu because in some cases it may work but in some cases it may work in a different way or it can be a combination of this for example in the next lecture I will try to show you that whenever we are trying to find out the length of a data vector and suppose the data Vector has some missing values then na. RM equal to True will not work but there I have to uh to work with na do omit so these uh minor shufflings minor changes will be there in different concepts uh but the basic fundamental basic concept will remain the same so my advice to you all is that back you please try to take now different examples try to take simple data Vector try to combine them in metrix data frame Etc and try to give some missing value yourself and try to see whatever missing values you have given at a given place are they being diagnosed by the r software and the more important part is that you try to look at the outcome of the software and try to look into backward what it is trying to indicate for example if the outcome is 24 that means you have to understand that in your data Vector the values are missing but what this this two and four are indicating is it second place fourth place or the values which are two and four so this type of understanding if you develop with the r software that is surely going to help you whenever you are trying to do with the multivariate procedures for multivariate procedure our thought process starts with the univariate and that is the reason I'm trying to emphasize first on the univariate procedure and all these concepts are going to be used later on whenever we are trying to do the hardcore multivar procedures like mpal component discri analysis cluster analysis Etc so you try to practice it and and I will see you in the next lecture where we are going to understand that how this missing values are being handled in the measures of central tendency so my request to you all is that please have a quick revision of the lecture where we had considered the measures of central tendency and then we will see how they can be modified when there is a missing data and how are we going to handle them how are we going to modify those commands that will help you in better better understanding so you revise and I will see you in the next lecture till then then [Music] [Music] goodbye [Music]