noc21_ma35 - Lecture 54

[Music] [Music] hello friends welcome to the course essential of data science with r software one in which we are trying to understand the basic fundamentals of probability theory and statistical inference so now you can see we have now understood about the role of probability in computing different types of tools for the statistical inference so now i ask you a very simple question in case if somehow it is becoming difficult to compute the probability from a given pdf or pmf what do you do can you recall that when we had discussed the normal distribution then we had computed different types of probabilities how just by taking the random variable minus its mean divided by standard deviation and we have seen that that was a good approach to find out different types of approximations for the probabilities which are coming from different distribution either discrete or continuous but they can be very well approximated by the normal distribution so now in this class today in the lecture today i will be working on this concept and we are going to talk about a very important result this is central limit theorem well if you come to the pure statistics there are several forms of the central limit theorem which are defined for different types of conditions here in this case i am going to discuss the most simple form of the central limit theorem and i will try to illustrate that how much it is useful in real data applications where you are trying to find out the value of approximate probabilities using the normal distribution so let us begin our lecture ok so what is this central limit theorem in a very simple language i can say that the central limit theorem tells that the sum of a large number of independent random variables has approximately a normal distribution right and it gives us a very simple method for computing approximate probabilities for the sums of independent random variables you can recall that at couple of places i have shown you that okay if two random variables are independent then their sum also has got a probability mass function or probability density function with some specific parameters and then in case if you want to find out the probabilities of their sum that means you have to first compute the their joint probability density function of the random variable x 1 plus x 2 and then you have to find out the probabilities from there which is many times difficult so the central limit theorem helps us and explain the fact that the empirical frequencies of so many natural population exhibits a bell shaped normal curve right thats a very useful information for those who are working in real data applications right so there are various forms of the central limit theorem and we consider here a very simple form right so this theorem says that let x1 x2 xn be a sequence of independent and identically distributed random variables and each of this variable is having a mean mu and variance sigma square i'm not assuming any distribution or even i'm not saying that whether x1 x2 xn are say discrete or say continuous now for large n the distribution of x 1 plus x 2 plus x n is approximately normal with mean n times mu and variance n times sigma square right then it follows from the central limit theorem that x 1 plus x 2 plus x n minus its mean and mu divided by its standard division sigma into root n this will approximately follow a normal distribution with mean 0 and variance 1 that is your standard normal distribution and now you can see here if you can just write down this thing as say x 1 plus x 2 plus x n divided by here n minus here mu then what will happen this result will be converted into a sample mean x bar n so now both the options are there and i will try to explain you so now at in case if you want to find the probability that x one plus x two plus x n minus n mu divided by sigma into root n this is less than some quantity say x then finding out this probability might be difficult but you can very well approximate it by the normal zero one so this can be approximated by probability that z is smaller than x where your z is standard normal variate following normal 0 1 right so this result can also be expressed as the distribution of say here zn which is x bar and minus mu divided by sigma by root n approaches to normal 0 1 as n approaches infinity right so now you can see here in this result nowhere i have assumed that x 1 x 2 x n are continuous or discrete but we are saying that this result is valid for actually both right so this x 1 plus x 2 plus x n minus n mu divided by sigma into root n or say x bar and minus mu divided by sigma by root n their distribution is always going to be continuous which is normal zero one and normal zero one is a continuous distribution right and beside this thing if you have observed it nowhere i am saying that x one x two x n they are coming from binomial or poisson or exponential or geometric we have not talked about it so nothing is said about the form of the original density function the only thing what we are trying to assume that we are assuming that the variance is finite so whatever the distribution function be provided that prohibited only that it has a finite variance the sample mean will have approximately the normal distribution for large samples yeah that is the condition that for finite sample the approximation may not be good but as you try to increase the value of your sample size or n this approximation will become better right now in case if you ask me that you are assuming here that the random variable should have finite variant do you think that is it a very difficult assumption whatever distributions you have done up to now means you have seen that they had finite variance and similarly if you try to look for other probability distribution then most of them have finite variance there is only one distribution cauchy distribution which has this problem that it does not has the finite variance but for all other thing there should not be any problem so the condition that the variance be finite is not a very critical restriction so far as we are concerning the applied statistics because in almost any practical situation the range of the random variable will always be finite and in that case the variance must must necessarily be finite right so this is not a very difficult con so it is not a very stringent condition that cannot be satisfied in real life so we need not to worry that much and it will give us a good result so now let me try to demonstrate that how this result will look like by taking an example of exponential distribution now you know what is exponential distribution so let x1 x2 xn be a sequence of iid random variables from exponential distribution whose pdf is this as and is having a parameter lambda so each of this x i has mean 1 upon lambda and variance 1 upon lambda square so now i can say that if i try to consider here this quantity here x 1 plus x 2 plus x n minus n times lambda divided by the standard deviation that is square root of n upon lambda then for large n this will have approximately a normal 0 1 distribution so now ah i will try to demonstrate this result using the r software so that i can show you that how it look like so what i am going to do here that i will try to generate the sample from this exponential distribution with say lambda equal to 2 and then i will try to compute the sum x 1 plus x 2 plus x n and i will try to compute this quantity also that x 1 plus x 2 plus x n minus and lambda divided by square root of n divided by lambda and now i will try to repeat this experiment and every time i will compute these two quantities for a given sample and then after that i will try to create a density plot of both the values which are obtained for this and this and we also try to find out their mean and variance right so it is not difficult i am now giving you here the entire program you have the slides also so you can simply copy and paste this program and you can just do it so you can see here this program is pretty simple we have only here the number of observations and the number of times we want to repeat it and then i am trying to generate the observation from this exponential distribution using the command and lambda that you know and then i am trying to compute their sum and then sum minus n lambda and then square root of n divided by lambda right and then whatever is the outcome i am trying to use here in plotting the density curve and this is the screenshot right so well my basic objective is to make you convinced and to show you that how these things are happening so now we try to generate the number of observation as say ten hundred one thousand ten thousand from exponential with lambda equal to two and then we change them up to an observation plus rep rep means repetition so this n is going to be increasing right so we start with n and then continuously the n is going to be increase and every time these two quantities their sum and then this quantity sum minus and lambda divided by square root of n lambda this is going to be calculated right and then means i have plotted all this curve clearly here so now you see that's the outcome so when i try to take out here this can observation then the plot of the sum that means x 1 plus x 2 plus x n this looks like this one you can see here the curve and if i try to plot the this central limit theorem that sum minus its mean divided by standard deviation it looks like this but now in case if i try to increase the sample size right i try to increase the samples from 10 to 100. you can see here now i try to compare this curve and try to compare this curve what is really happening this is becoming smoother but this is now becoming more symmetric and in both the cases if you try to see here that means ah and variance are obtained here like this right and here even you can see that the mean is coming out to be say minus 0.22 which is say means quite close to 0 but still that is not actually 0 and variance is not actually here 1 right but if you try to increase the sample size the mean and variance are going closer to 0 and 1 respectively now in case if you try to increase the sample size to 1000 and then to 10 000 then the density plot of the sum will look like this or this but in case if you try to plot the sum minus its mean divided by standard deviation that is clt it will look like this and you can see here that as in going to infinity this curve is becoming more symmetric and more similar to the normal density curve and even the mean and variance they are approaching towards 0 and 1 respectively ok so this is the basic idea of the center limit theorem and you can see here these are the results which i have reported you here right now do you remember that when we did the normal distribution then we also had discussed this aspect continuity correction and there i had explained you that when we are trying to approximate a discrete distribution by a continuous distribution then there is a need of continuity correction and i had explained you what is this how it is obtained so now i will not explain it again but i will try to use it here because in the central limit theorem also when you are trying to approximate the probabilities for a discrete random variable then you need to apply the continuity correction also right so and we approximate the probabilities for discrete distribution we incorporate the continuity correction also right so let x1 x2 xn be a sequence of iid but discrete random variables and each having a mean mu and some variance say sigma square suppose we want to find out this probability that x1 plus x2 plus xn they are lying between two numbers x1 and x2 so now i have to do the same thing that we discussed earlier that i have to subtract 1 by 2 in x 1 and i have to add 1 by 2 in x 2 and then i will try to compute the probability after standardization right what we try to do we try to simply write down here x 1 plus x 2 plus x n minus n mu divided by sigma root n and the same thing i try to do on the both the sides and this probability can be obtained the probability of here this quantity minus probability of here this quantity very straight forward algebra that now you know and this is nothing but this can be obtained by the cdf of normal zero one and this is also the cdf of normal zero one right so both these probabilities can be obtained at these two point x two plus one by two minus n mu divided by sigma into root n and x one minus one y two minus n mu divided by sigma into root n right so now ah let me try to give you here one more example from the poisson distribution where we try to apply this continuity correction so let x1 x2 xn be a sequence of iid ah random variables from poisson distribution each having a mean say lambda is equal to 0.125 and variance is equal to 0.125 because in the case of poisson distribution the mean and variance both are the same and its probability mass function is given by this quantity so for example if i try to say here therefore n equal to 65 suppose we want to know probability that some of this x is equal to 10 is equal to 0.099 right that we can obtain in the r now we try to approximate the same probabilities using the central limit theorem so what i try to do here that i try to write down here summation i goes from 1 to n x i equal to 10 and this can be we obtained by applying the continuity correlation that you try to say subtract 1 by 2 and add 1 by 2 on the left and right hand limits and then you try to standardize it that just means summation x i minus its mean divided by standard deviation and you get here this expression and this probability can be can be written because now this is here your z that is the standard normal variate following a normal 0 1 distribution and this probability can be written in the form of cdf as cdf at this point and the cdf at this point right and if you try to compute this probability from the r software this will come out to be 0.108 this is not difficult at all right so now in case if you try to take here suppose here n is equal to 96 and you try to compute the probability that summation x is equal to 10 that this will come out to be 0.105 but if you try to use it on the basis of central limit theorem this will come out to be here 0.101 so you can see here this is more closer to 0.105 than the case when we had n is equal to only 64. right so you can see that as you are trying to increase the sample size the probabilities are becoming more clear okay so now we try to come to one more result where i am trying to give you the approximate distribution of the sample mean because now in the last lecture we have seen that sample mean is playing a very important role when we are trying to draw different types of stealth in currencies right so let this x1 x2 xn be a sample from a population having a mean mu and variant sigma square now the central limit theorem can be used to approximate the probability the distribution of the sample mean say x bar n which is equal to 1 upon n summation i goes from 1 to n x i now we know we already have actually proved that expected value of x bar n is equal to mu and variance of x bar and is sigma square upon n which depends on this here n and we also know that this x bar n this is based on a linear combination of the normally distributed random variable so when sample size is large then x bar and minus mu upon ah sigma by root m this will has this will also have an approximately of normal distribution with mean zero and variance one that is the standard normal distribution normal zero one right so now let me try to give you an example of this result ah suppose the height of the university student was measured and and it was found that it has got a mean of 167 centimeter and standard deviation of 27 centimeter now uh sample of 36 workers is chosen from that population and we want to find the approximate probability that the sample mean of the height lies between one sixty three and one seventy that is what we want to find out so now using the standard notations let z be a normal zero one random variable and it follows using the central limit theorem that x bar is approximately normal with mean 167 and standard deviation 27 divided by the square root of 36 which is the value of here n equal to 36 and this will come out to be here 4.5 now in case if you want to compute this probability that x bar is lying between 163 and 170 you can just standardize the sample mean by writing sample mean minus its mean divided by standard deviation and you try to do the same operation on both the sides and you can simply solve it after this this will become a standard normal variate and this probability will be equivalent to finding out the probability between minus 0.089 and plus 0.089 so you have done a result that probability that z is lying between minus a and plus a right then this can be written as say twice of probability that less z is less than a minus 1. so i can use that result directly here and can find out the value of the cdf at this point from the r software so that you know how to find out the value of the cdf in the standard normal distribution so this value comes out to be here 0.625 and so on so you can see here it is not difficult to obtain such complicated probabilities without any problem now let me try to address the last topic of this lecture that many times in real life we always have a question that what should be the sample size means how many observations should we draw so that we can take a reasonable conclusion so now the question i am trying to phrase here is that how large the sample size and needs to be for the normal approximation to be valid for example in all these cases we are trying to approximate the probabilities using the normal distribution and we are saying at least theoretically that as n goes to infinity this will work but in practice somebody will always like to know what is this n for which it is really going to work so this answer depends on the population distribution of the sample data that is there that you have seen means i was always trying to do an experiment and i was trying to show you the results after every probability density function and probability mass function that i will try to generate the observation from that distribution and then try to compute its mean and variance and then i was trying to show you that as n is increasing the estimated value of mean and variance are converging towards the theoretical mean and variance right so now for example if i say the population distribution is suppose normal then the sample mean will also be normal regardless of the sample size then there is no issue a general rule of thumb is that one can be confident of the normal approximation when the sample size is n is greater than 30. now you know what is the reason means this reason will become more clear as we move forward but do you remember the discussion in the t distribution lecture try to think about it so in such a case even if the underlying population distribution is not necessarily normal the sample mean of a sample of size at least 30 will be an approximately normal right and the normal approximation is valid for much smaller sample size in most of the cases but it depends on say different types of parameters which you are going to involve or what is the probability density function probability mass function and so on right so now let me come to an end to this lecture that was a very short and simple result but what i am trying to address here that is very important because in data science you are dealing with the complicated distributions and sometimes there can be a combination of discrete as well as continuous random variables also and you are always interested in computing different types of probabilities well without probabilities you cannot do the statistical modeling that we have understood so now whenever you are trapped whenever you want to find out the value of the probability yes you cannot find out the exact probability but if you try to use this central limit theorem possibly you will get a very good approximation and this is going to work very well in practice now regarding the sample size issue my experience is that the sample size required depends on many things for example even if you are trying to take the normal mu sigma square in case if the sigma square value is very very low possible you will need a smaller sample size but in case if your sigma square is very high then you will need a much larger size so the same thing can happen in other type of probability density functions also also in real life whenever you are trying to deal with the real data it may be difficult many times to know exactly what is the probability density function of probability mass function so you are simply trying to approximate or assume that this is the distribution which is going to represent the true data up to a great extent so thats a sort of approximation so under those type of situation this clt is going to help you a lot so now i will stop here and i will request you that you please try to think try to look into books and you will find different versions of the central limit theorem under different types of condition for example here we have assumed that the random variables are identically and independently distributed suppose they are not independent then these this are not going to be valid so for that you have to look into the books and find out the correct version but now i am sure that after so much of training it should not be a very difficult task for you to learn and understand those things so you try to revise have a look and i will see you in the next lecture with more topics till then good bye [Music] [Music] [Music] you