Lecture 13

[Music] recall that we found that the sample mean xar was 2.2 PPM and we want to validate whether this mean of the sample will actually be the mean of population now we understand one thing for sure to find out the population mean exactly from on the sample mean with zero error is near impossible we'll obviously make some error that is at the best we can say that the population mean should be 2.2 plus minus some error so we might be able to conclude that the population mean will be 2.2 PPM plus minus 0.1 or 2.2 PPM plusus 0.2 or something like that but that is still useful to us the aim of the problem at hand is to be able to tell whether the amount of lead in a noodle packet is greater than or less than 2.5 PPM so even if we can say that the mean of the population is 2.2 PPM with an error of plus- 0.2 we would know for sure that the mean lead content will be between 2 and 2.4 which is definitely less than 2.5 now in order to see if the sample is indeed a true representation of the population let's do a simple experiment let's go back to our noodles example where we already have a population whose parameters are known to us consider that you have the complete set of 30,000 modal packets that this company has manufactured we are given that capital N is equal to 30,000 now each data point in the CC file is the lead content in PPM for that modle packet now if we check the mean of this data we get it as 2.99 and the standard deviation comes out to be 0.132 these are population mean and standard deviation values which are represented as Mu and sigma respectively but for now let's take a small sample size of five and see what its parameters are so we randomly choose five noodle packets out of 30,000 and we find out that the mean of their lead content is 2.45 PPM but this is a little far away from the actual population mean which was 2199 it might have happened that we coincident choose noodle packets with lower lead content let's take another example this time the mean comes out to be 2.27 PPM which is quite higher than the population mean again it look like we coincidently choose Nole packets having higher L content so to normalize this effect of getting bias samples let's choose 100 samples instead of two this number could have been 70 80 or 90 but for now let's move with 100 samples note that we are talking about number of samples here and not the sample size sample size here is still five if we plot these sample means on a graph we find out that the distribution of these sample means looks like a normal distribution and if you see the center point of this distribution it is around 2.2 which is in fact very close to our population mean this certainly opens up many Avenues we saw that when we took a large number of samples and plotted the mean of each sample on the graph we got a distribution that was quite close to a normal distribution as we keep on increasing the sample size the sample mean will keep approaching the population mean now when I say increasing the sample size I mean the number of noodle packets in each sample and not the number of samples that we took so earlier our sample size was five if we increase it to 10 keeping the number of samples the same that is 100 the distribution becomes thinner and closer to a normal distribution in fact if you keep increasing the sample size the distribution keeps getting narrower and the distribution seems to look more and more like a normal distribution in fact when you increase the sample size to 30 it becomes almost perfectly normally distributed with the mean centered around the population mean what does this mean for us this experiment that we performed right now is the basis for Central liit theorem the centrer limit theorem states that when you take a large number of samples the mean of the sampling distribution thus formed will be approximately equal to the population mean the second part of the theorem states that the standard deviation of the sampling distribution will be equal to Sigma which is population standard deviation divided by the square root of n where n is the sample size just remember that the sampling distribution standard deviation would be Sigma by root of n finally the center limit theorem states that if the sample size that you take is greater than 30 the sampling distribution will become normally distributed these three findings in short constitute the center limit theorem let us first look at a few terms and notations the whole bunch of packets that is 30,000 packets is called the population and the small collection of packets that we select from it is called a sample if you look at this table you can find all the notations and formula for both the population and the sample the population size is denoted by capital N its mean by mu and its standard deviation by Sigma the variance in turn is Sigma squ the formula of the sample have a small twist here the sample size is denoted by lower case n and the mean by xar which is nothing but the sum of all the observations in the sample divide by the total number of observations which is the standard way of computing the mean so there's nothing different about calculating the mean for the sample this xar after measuring the lead was found to be 2.2 PPM parts per million but now if you come to the sample variance s squ you can see in the denominator you have n minus one instead of n to concretize your understanding of the center limit theorem let's try and visualize the center limit theorem we plot means sampled from a non-normal population with 100 samples for different sample sizes the four plots corresponding to our four sample sizes are 1 5 10 and 30 examine the four density plots for the sample that are colored in blue and compare with the normal dens distribution in red here we start with a sample size of one the resulting curve deviates a lot from the red curve as we keep on increasing the sample size from 1 to 5 10 and 30 the blue curve comes closer and closer to the red curve this graph is now starting to look like a normal distribution and the reason for this is the centrer limit theorem and what this States is that if you plot the sampling distribution then this distribution approach as a normal distribution regardless of what your parent population is and this is especially true when the sample size is above 30 so here you can see that even with a sample size of 30 the distribution almost approximates a normal distribution if you look different means from different sample sizes they appear to be similar you see that this is also the mean of the population and then you come to the third graph and here you see that this is the mean of the sampling distribution and you can see that both these means are almost equal the standard deviation of sampling distribution is the standard deviation of population divide by square root of sample size as you keep on increasing the sample size the Curve will start to look like normal distribution now as you increase the size of n you are bringing the N value closer to the population size value you'll start seeing that the sampling distribution of sample means will start looking more like a normal distribution and you see that the mean is BEC more equal to the population mean what about the standard deviation of the sampling distribution compar the standard deviation value of the population which is 28.8 with the standard deviation of the sampling distribution the standard deviation of the sampling distribution approaches closer to the standard deviation of population divided by the root of n for example the standard deviation of sampling distribution for n equal to 30 is 5.3 which is closer to 28.8 upon < tk3 which is equal to 5.26 also see that the normal distribution became thinner as the sample size increases this is because the standard deviation decreases remember here the standard deviation is Sigma by root n so obviously Sigma byoot 5 is going to be higher as compared to Sigma byun3 which is why this graph seen earlier becomes more packed and now what do you think will happen if I further increase the sample size say from 9 N = to 30 to Nal to 50 the normal distribution will become even narrower right so this is your Center limit theorem statistical inference two introduction to confidence intervals let us go back to the noodles example and see if we are able to derive conclusions about the population using the sample if you recall we took a sample of 100 packets and find out that its sample mean was 2.2 PPM and standard devision was 0.7 PPM making use of what you learned about sampling distributions we can directly assume that the sample mean of 2.2 PPM that we have calculated belongs to one of the infinite possible sample means that is present in the probability distribution and this probability distribution is nothing but our sampling distribution remember that the sampling distribution is nothing but the distribution of all the possible sample means that can be generated from this population and this means that our sample mean which was 2.2 PPM is definitely one of the sample means present in this distribution as per the center limit theorem if the sample size is greater than 30 then we can assume that the sampling distribution is normally dist distributed with mean equal to the population mean which is unknown in this case and standard deviation equal to Sigma by root of n where Sigma is unknown and N is the sample size so with this we can conclude that our sample mean which was 2.2 PPM belongs to a normal distribution with mean equal to the population mean and standard deviation equal to Sigma by root n again this distribution is called the sampling distribution and the reason we know that this sampling distribution will be a normal distribution is because of the center limit theorem so the sampling distribution is like this and our sample might lie somewhere in this distribution so let's start with the estimation right now we know that the mean of this normal distribution is equal to Mu which is the population mean but we don't quite know what this mean is so let's come back to that later this distribution standard deviation is Sigma now remember that Sigma is the population standard deviation but wait we don't know what the population mean is how are we supposed supposed to know the population standard deviation now there may be some cases where prior to sampling you were able to obtain a good estimate about the population standard deviation in that case you can use the value of Sigma as it is but what if that is not the case in fact in most practical scenarios such as our noodle example Sigma is usually unknown in such cases we'll need to substitute the sample standard deviation in place of the unknown population standard deviation Sigma the sample standard deviation is represented by by S and which we found out to be 0.7 hence replacing s in the formula and replacing n with 100 which is the sample size we get the standard deviation of the normal distribution as 0.07 note that here we are using the sample standard deviation as a substitute for the population standard deviation this is something we do since in most cases we will not know the population standard deviation now come back to our example we know that the distribution is normal and thus we can make use of the different properties that we have learned about the normal distribution recall that one of them was the empirical rule also known as 1 23 rule so let's quickly revisit that rule the probability of X lying between mu minus Sigma and mu plus Sigma is around 68% 1 two the probability of X Ling between muus 2 Sigma and mu + 2 Sigma is around 95% and three the probability of x l between muus 3 Sigma and mu + 3 Sigma is 99.7% using this rule we can say that the probability that the sample mean which is 2.2 will Li between muus 2 standard deviation to Mu + 2 standard deviation will be equal to 95% right note that mu is the population mean and it is the same as the mean of the sampling distribution which we are making use of standard deviation is something we just found out using Sigma by root n which we calculated as 0.07 so going back to our problem we said that 1 23 rule states that 95% of the values in a normal distribution lie between mean plusus two standard deviations we do know that the standard deviation of this normal distribution is 0.07 which we just find found out hence this equation after replacing 07 becomes probability of mu minus 0.14 is less than 2.2 Which is less than mu + 0.14 = to 95% that is probability of 2.2 lying between muus 0.14 and mu + 0.14 is 95% now using this equation can you tell what will be the value of mean mu lying between 2.2 - 0.14 to 2.2 + 0.14 think about it it's the same thing you simply just rearrange the terms of the expression probability of muus 0.14 less than 2.2 less than mu + 0.14 so you got the expression probability of 2.2 minus 0.14 less than mu Which is less than 2.2 + 1 0.14 what we did was we simply carried over mu to this side and 2.2 to either sides so since both of these are the same equation you can say that probability of 2.2 - 0.14 is less than mu and mu is less than 2.2 + 0.14 equal to 95% or in other words probability of mu being lying between 2.06 to 2.34 is 95% hence you can say that with 95% probability that the mean will Li between 2.06 to 2.34 and you found out this value just by using the 1 12 3 without even knowing the value of mu so before moving further let's get some terms straight the probability associated with this claim is called the confidence interval in this case since we are concluding about the population mean with 95% probability we can say that the confidence level is 95% in many cases instead of confidence level you may also be given the level of significance which is denoted by Alpha the significance level is given by one minus confidence level so if the confidence level is 95% or 0.95 then the level of significance will be 1 - 0.95 which is 5% or 0.05 similarly if you given the level of significance or told that Alpha is equal to some value say 0.05 then you can say that the confidence level which we want to conclude is going to be 95% next you have the margin of error which is the maximum error made in the sample mean in this case margin of error is 0.07 into 2 which is 0.14 and finally the final interval of values that you get is called the con confidence interval in this case our confidence interval is 2.06 to 2.34 the upper bound of the confidence interval that is 2.34 we can conclude that the noodles that are being manufactured on average do not contain higher than the accepted threshold of lead which is 2.5 PPM and we did all of this using just a sample of you noodle packets which is amazing right hi welcome to the statistical inflence 3 confidence interval construction if you look back at everything you learned till now it was all about solving a simp Le problem calculating an interval for our population mean and in order to do this you learn something about sampling sampling distributions Center limit theorem and many other things so in this discussion let us actually generalize this approach so that if we give us any large population and ask us to give an estimation about the population mean we know how to solve this problem and come up with an unbiased confidence interval first you need to decide whether it is practical to actually obtain all the values in the population so that you can calculate the mean accurately because in such a case your problem is already solved but suppose getting this population data is not actually feasible at all which is usually the case this is where your influential statistics knowledge comes into play there is no way to know the actual population mean and standard deviation so for this population that we have with us weide its mean Is mu and standard deviation is Sigma and the objective of this problem is to get an estimate of the population mean mean that is Mu also most important of all this population can follow any distribution it need not be a normal distribution it can be a uniform distribution or any random distribution that comes to your mind so how do you start this problem you start by collecting a sample right you have even learned multiple sampling techniques such as simple random sampling and stratified sampling so it is up to you to choose an appropriate sampling technique such as that whatever sample you have can be considered a proper representation of your population so let's say you have selected your sample with a size of small n now for the sample calculating various parameters such as mean and standard deviation are possible so let's calculate those let's call the sample mean as xar sample standard deviation as s and sample size as n now remember that we know the values of xar s and n and using these values we need to somehow find an interval for our population mean mu to solve this problem you learned that we can make use of central limit theorem let's recall this theorem the center limit theorem is based on a probability distribution known as sampling distribution so firstly what is the sampling distribution let us say we have our population and we start collecting different sample of size n and let's say you collect your first sample calculate the sample mean call it xar one then calculate your second sample calculate the sample mean call it xar 2 and so on now you can continue this process an infinite number of times and you'll be left with an infinite number of sample means that are xar 1 xar 2 xar 3 and so on all the way up to xar Infinity now suppose you consider your random variable X such that it is the sample mean then the possible outcomes of X will be xar 1 xar 2 xar 3 and so on up till xar Infinity then the probability distribution that this random variable X follows is called sampling distribution you will also hear some people saying it sampling distribution of sample means which is the same thing so we have the sampling distribution now our Center limit theorem says that this sampling distribution that you have generated is going to be more and more like a normal distribution as you increase the sample size n in fact we assume any sampling distribution as a normal distribution if the sample size is greater than 30 which has a mean of mu and standard deviation of Sigma by rooot n we have selected our sample whose sample mean is xar and Sample standard deviation is s and the sample size is n Now using these values you need to come up with an interval for our population now if we consider a confidence level of y% and if we apply the center limit theorem we can derive that our population mean lies in the range xar minus Z into s by root n plus xar + z into s byoot n where Z is the critical value associated with y% confidence interval for example if the confidence level at which you are looking to estimate the population parameters is 90% then the value of Zed will be 1.65 if the confidence level is 95% then the value of Zed will be 1 .96 and for 99% it will be 2.58 these are the standards at values for confidence levels traditionally taken in the industry you can choose the confidence level according to your problem for example in our noodles problem we want to be highly confident in our results since the entire company's business depends on this and hence in such cases choosing a confidence level of 95% or 99% makes more sense on the other hand if you are even 90% confident that making a change in your homepage will result in generating more leads you can probably go ahead with it and make the change so based on different scenarios you choose different confidence levels to make it easy for us to remember there are five step approach that you can use for estimating the mean of a population first you collect the sample of size n greater than 30 from a population in step two then you compute the sample mean xar and Sample standard deviation in step three now you can assume that your sampling distribution is a normal distribution with mean equal to population mean mu which is unknown and a standard deviation Sigma by root n which can be approximated by s/ ro n step four you select the confidence level at which you want to estimate the population mean for example 95% 99% or maybe 90% it can be lower or higher than these numbers based on your requirement finally compute the confidence interval given by xar minus Z into s by < TK of n to xar + z into s by < TK of n now that you know how to determine the confidence interval for a population we'll take a look at some other scenarios that you might come across and learn how you can calculate the confidence interval for these scenarios statistical inference four interval estimation for small samples so far you had the luxury of collecting lots of data points which meant your sample size was at least 30 which allowed you to apply the center limit theorem hence you need a minimum sample size of 30 to be able to use the Z distribution for calculating the confidence interval however in real life you'll often come across situations where you have to work with small samples for example assume you are working for a Pharma company you need to compute an interval for the effect of a medicine on patients and you only have 15 Volunteers in such cases it is better to use the T distribution when the population standard devation is not known instead of using the Z distribution where we estimate the population standard deviation to be equal to the sample standard deviation so in this discussion you will see how to calculate the confidence interval using a t distribution when your sample size is less than 30 so first let's understand what this T distribution is the T distribution is centered at zero but its standard deviation is proportionally larger compared to the Z distribution the only difference is that the T distribution has a shorter Peak and wider Tails consider the figure shown here the exact shape of T distribution depends on the size of sample for smaller sample sizes the T distributions are flatter and for large sample sizes as you can see here how the sample size relates to the T distribution as sample sizes increase degrees of freedom increase and as they approach 30 the T distribution approaches to normal distribution which is in Black so let's take an example where you can understand how you can estimate the interval for smaller samples imagine you work for a Pharm company and are testing the effects of medicine on 15 volet years hence your sample size is 15 now you find that medicine increases the presence of a particular hormone XYZ in patient's blood by 10.38 micro units so this value becomes our sample since the population standard deviation is not known to us we estimate this using the sample standard deviation let's say that our sample standard deviation comes out to be 0.072 so how do we solve this problem recall the general approach you learned for estimating the interval using Z distribution we will use this exact same approach except for a few changes so let's go over each step that you have learned the first step was to collect a sample size n here n is equal to 15 since the sample size is less than 30 we'll go ahead with the T distribution let's move to step two now step two was to compute the sample mean and standard deviation since the population standard deviation is known to us we are estimating this with the sample standard deviation again the values of sample mean xar and a sample standard deviation s is 10.38 and 0.072 coming to step three earlier you learned that step three was where we assumed our sampling distribution was a normal distribution however since we are dealing with a smaller sample hence in our case the sampling distribution will follow the T distribution now like the normal distribution there's an entire family of different T distributions each T distribution is a distinguish by what statis called degrees of freedom which are related to the sample size of the data set for a sample size of n the degrees of freedom for the corresponding T distribution would be n minus one for example for a sample size of 100 the T distrib would have 100 minus 1 that is 99° of Freedom denoted BF d99 this is why for smaller sample sizes the T distributions are flatter than for large sample sizes in fact as the sample size increases the degrees of freedom also increase this makes the D distribution look more like a standard normal distribution or the Z distribution the point where they become very similar to each other is about the point where the sample size is 30 hence for any sample size that is greater than 30 the T distribution can be approximated to a normal distribution so in our case the sample size is 15 this means that the degrees of freedom should be 15 minus 1 which is 14 hence the distribution that we'll use to calculate the confidence interval will be a t distribution with degrees of freedom equal to 14 now that we know our sampling distribution for step three let's move to step four if the step four was to select the confidence level at which you want to estimate the population mean let's assume that the confidence level we want to estimate is 95% let's move to step five this is the step where we calculated the final confidence interval using the formula xar minus critical Z * s byun n the only difference in the formula will be that in place of Z we use T value the confidence interval for our population mean will thus be given by xar minus critical T * s byun n to xar + T critical * s by otk n so what is the T critical value to calculate this value we'll use either the T table or you can also compute T values on T table easily using our software so we are going to search for 0.05 significance value and two tail test for now we do not have to worry about what is one tail and two tail we'll select the critical T values from the table corresponding to the degrees of freedom for example in this case degrees of freedom is 14 so we'll find the critical value as 2.45 the lower bound of the interval will be xar minus t critical into s by root of N and the upper bound of the interval will be xar plus T critical * s by root of n so let us substitute the values we have and then we get 10.38 - 2.45 which is our T critical times s which is 0.072 and then divide by square < TK of 15 this comes out to be 99.998 similarly the upper bound will be what 10.38 + 2.45 * 0.072 divide < TK of 15 and this value comes out to approximately 10.07 7 so we can say that our confidence interval for the presence of that hormone lies in the range of 9.9 98 and 10.07 7 with a 95% confidence interal so let me quickly recap what you have learned in this video you must use the T distribution bin the number of data points that you that is a sample of size of less than 30 and the T distribution is also preferred when the population standard deviation is unknown you also learned that the T distribution depends on an additional parameter called degrees of freedom that is DF the degrees of freedom is calculated as a sample size minus one as a sample size increases it means that the degree of freedom is also increasing and as a result the T distribution tend to become narrower and narrower at sample size greater than or equal to 30 the T distribution is essentially indistinguishable from a normal distribution which is why if you tried solving the noodles problem using the T distribution you'll get the same result as you got with the Z distribution so going forward we can see that the flow chart that you can use for deciding between Z and T distribution if the population standard deviation is unknown and the sample size is greater than or equal to 30 then that distribution is perfect over T distribution this is so because T distribution approximates the variance using the sample size as you saw when we increase the degrees of freedom but this is not the case with Z distribution however if the sample size is less than 30 then even if the population standard deviation is known it is best to use the T Test as it is idly suited to dealing with a small sample now if you use the T distribution then there is the formula for for calculating the confidence interval the lower bound of the interval is given by xarus t s by root n and the upper bound is given by xar plus T critical Time s by root n Now using Z distribution and T distribution you can estimate the confidence interval for any sample whether the sample size is greater than 30 or less than 30 statistical inference part five internal estimation for proportions here we have assumed that all the values in our sample are numerical values but what if the values are categorical in nature a very common example of this is the exit polls that are conducted during the election the idea of these polls is to estimate the number of people who voted for a certain candidate or party suppose you sample a few voters and calculate the proportion of Voters that voted for a certain party now how will you extrapolate this value to the entire population in this discussion we'll dis show how to solve such problems and as an example we'll take up this exact problem in this discussion we will show how to solve such problems when the data is categorical in nature and in such cases we calculate the proportion instead of mean let's see this example if you're working as a part of political science company that specializes in water pools and Designs surveys to keep political office Seekers informed of their position in a race this is done through telephone surveys where they ask registered voters who they would vote for if the election were held at that day now Suppose there is an election campaign going on and through these interviews you found that 220 registered voters out of 500 contacted favor a particular candidate now your company wants you to develop a 95% confidence interval estimate for the proportion of the population of registered voters that favored that candidate as you can see the data in this problem is categorical in nature either the voter voted for that candidate or did not vote for that candidate for this reason will work with the proportion of Voters that voted for the candidate hence the proportion of Voters voted for the candidate is 220 divid 500 which is 0.44 now the objective here is to compute a confidence interval for the sample proportion value which is 0.44 the idea is to compute an interval around 0.44 for example 0.43 to 0.45 let us call this sample proportion as P bar P bar equal to 0.44 as we mentioned earlier the approach for solving any interval estimation problem remains always the same let us go back to our fstep approach that we discussed earlier at and understand how we can solve this problem step one here is to collect a sample of size and as you can do that in our case sample size is 500 step two is to calculate the sample mean and standard deviation however since our data is categorical in nature we only have data on Sample proportion we found the sample proportion to be 220 by 500 which is 0.44 step three was to assume our sampling distribution as normal distribution now as you have learned for any interval estimation problem we der derive the interval from the sampling distribution when we use the normal distribution we first generated the sampling distribution of sample means which was xar and we found that the sampling distribution was a normal curve with parameters mu and sigma by root n in our example since we want to find the interval estimate for the population proportion if you recall the condition of approximating the sampling distribution of sampling means it was that the sample size should be greater than 30 similarly we have a condition for being able to apply the sampling distribution of sampling proportion the condition is that first the sample size which is n times the population proportion P should be greater than 5 and second the sample size n * 1us population proportion P should be greater than five so let's see if these conditions are satisfied now since the best estimate of the population proportion that we have is the sample proportion hence we'll substitute the sample proportion in the equation since we know n equal to 500 and P is approximately 0.44 we already know that NP equal to 220 similarly if you calculate n * minus P it will come out as 280 since both these values are significantly greater than five we can say that our sampling distribution follows a normal distribution and we can go ahead and use the formula for the confidence interval of the population proportion as we'll see shortly so let us move to our step four for now a step four was to select our confidence level it has been given to us that the company wants us to develop a 95% confidence level for the proportion finally we come to step five which is where we compute the final conf confidence interval for calculating the confidence interval we'll use the formula given by P bar minus critical Z * the root of P bar into 1 - P bar by n as shown here so this is the lower bound of our interval and the upper bound of the interval will be P bar plus critical Z * under root P bar into 1 - P bar by n just by looking at the formula you can say that the sampling distribution that we have used is a normal distribution with mean as the sample proportion and the standard deviation as the root of P bar * 1 - P bar upon n you already know that P bar is equal 0.44 and the sample size n is 500 we also have to calculate Z but you have already learned that for 95% confidence interval Zed will be 1.96 so let's put these values so if I just look at the margin of error Zed or critical Z * root of P bar into 1 - P bar by n and putting in the values we'll get this values 1.96 * root of 0.44 into 1 -44 upon 500 which is equal to 03965 overall the value will give us 0.44 - 0.435 equal to 03965 which is the lower limit and 0.44 + 0.435 equal to 0.48 35 so our confidence interval is 03965 to 0.48 35 thus we can conclude that there are 95% chances that the proportion of all voters that favor the candidate is within range of 03965 to 0.483 5 so now let us quickly go over the approach for solving the interval estimation for the population proportion wherever the data is categorical in nature we calculate the proportion instead of mean so the aim of the problem is to be able to estimate an interval around the sample proportion P bar P start of these are n * P bar should be greater than 5 and n * 1 minus P bar should be greater than 5 if both these conditions are met then we can use the formula for inference this formula is given by quite simply P bar minus Z * root of P bar into 1us P bar divid by sample size n here Z is basically your critical Z value that will depend upon the confidence level that is provided to us [Music] [Music] [Music]