noc21- ma35 - lec19

[Music] [Music] hello friends welcome to the course essentials of data science with r software and now you can see that we have learned the concepts of probability theory in different ways simple probability conditional probability and we have used different types of theorems in the form of some results additive theorem multiplicative theorem and so on and you have seen that these theorems results concept they are going to be applied in different types of conditions situations and you can draw various types of conclusions from there that depends on your capability how much you can dig out the information through these tools now continuing on the same lines today i am going to talk about a very important result in the probability theory this is bayes theorem believe me this base theorem is very important and this is the foundation of the bayesian inference and you know that whenever we are trying to compute some probability there are two possibilities that the event is going to happen for the first time or the event has already occurred in the past for example if i ask you that how much time do you take from going from your home to your college if you are going for the first time then you might not be knowing but now since you are going to the same college for a long time so you will say ok i will reach in say 20 minutes or so how do you come to this idea or if i in fact you can also say if there is no traffic then 15 minutes if there is heavy traffic then 30 minutes if and if there is a reasonable traffic possibly 20 minutes and so on now if you try to see if i ask you a simple question today that how much time are you going to take then will you compute the probability only on one observation or you will try to use all this information somewhere inside your mind somewhere in the background you will try to capture that information and somehow you will try to compute the probability and you will tell me that what is the probability for example if i ask you that how much time i am going to take going from my hotel to the airport or railway station in a new city means i am going there for the first time but you are staying there for a very long time so you will ask me at what time you would like to go i will say okay this is at five o'clock in the evening so that is going to be office time people will be coming back so it is going to be crowded so you will so whatever is the time you will give me some more time more time in the sense the time taken at four o clock in the morning when there is no traffic usually what is that you automatically computed that time taking into account the prior information that prior information was there with you and that information was collected either from you or your friends or somehow you came to know that at five o clock there is going to be a traffic and it may take longer time and then you are trying to tell me that how much i am going to take so under these types of events if you try to compute various types of probabilities they can also utilize this type of prior information and once you are trying to use the prior information things are expected to improve because you are trying to incorporate more information into the system if you try to incorporate more information well the information has to be correct then you expect that your statistical inferences are going to be improved so with this point of view let me try to explain you here the concept of base theorem and then i will try to take couple of example to make you understand better i will try my best and you also try your best so with this objective let us begin our lecture okay so now you already have learnt one thing what was the law of total probability and this rule says that if a 1 a 2 a m are the event cell that a 1 union a 2 union a m is equal to capital omega and a i and a j they are pairwise disjoint that is a i intersection h is equal to phi for all i not equal to j one to m and probability of a i is positive for all i that is obvious then the probability of an even can be calculated as probability of b which is equal to summation i goes from one to m probability of b given a i into probability of a i right so now if you try to see i would try to use this theorem in getting the bayes theorem the base theorem gives a connection between the two conditional probabilities properties of a even b and probability of b having a right and in order to understand what this base theorem tries to give you what type of probabilities it gives us let me try to take a very interesting example and try to show you that how it is used and what type of information it is going to give you and how it is different from the simple probability right so suppose there is a blood test and that blood test claimed that it can check the absence of presence of a rare disease right and there will be some probabilities this i will tell you and there are two possible events related with this claim the first event can be that the outcome of test is positive and the event d we are trying to define as person has disease right now i try to do here one thing that suppose i have some information that the probability that the person has disease and test is positive that will probability of a condition on d that is 0.999 right and we have one more information that probability that person do not has disease and test is negative that is probability of a compliment condition on d complement and this is also 0.999 now as a common man if you try to see you are trying to say that in case if the person has disease and test is also saying that the person has disease that is the test is positive this probability is 0.999 that is almost 99.9 percent and on the other hand in case if the person doesn't has any disease then that test is saying negative this probability is once again 99.9 percent so obviously you will assume or you will feel that the test is excellent test is very good now what we try to do we select a person and we make a test right and now this person who is collected for the test this person may or may not have that disease so we assume that the probability that the person has a disease is probability of d so usually this probability of d is going to be small right why should i expect that every person has a disease and suppose this probability is very small 0.0001 right now we want to know based on the outcome that whether the test is good or bad so now you know we can mean the condition is this you already know that probability of d that is the probability of disease is very small now you try to compute this probability probability of d condition on a that a has already occurred or a is true so now this probability can be written as a probability of d into probability of a 7 d divided by probability of d into probability of a 7 d plus probability of d complement into probability of a given d complement so now we have all these values and we try to substitute it here right we have this probability of d to be 0.0001 and the probability of this test is 0.999 so we try to substitute it here and then probability of d complement will be simply 1 minus 0.0001 and probability of a given d complement will be 1 minus 0.999 and if you try to compute it this probability comes out to be 0.091 what do you think now you are saying that test is nearly 9.1 percent that the test is good or bad now if i you are saying that the test is nearly nine percent reliable only what do you think the test is good or bad and if you try to see what you have said earlier in these two probabilities here and here you said the guess is 99.9 percent good but when you are trying to compute the probability that whether the test is good or bad given that the probability of d is very small this probability is coming out to be very less nearly 9.1 percent so that means test is not dependable so what you have seen here that if you are trying to use some prior information some prior probability then this is going to change the outcome this is going to change the result right on these types of probabilities which are known to you a priory they are called as prior probabilities and up to now we were looking only on the prior probabilities and the probabilities which you have now computed they are posterior probabilities they are called as posterior probabilities so now the model of the story is do not rely only on prior probabilities but also look for the posterior probabilities and this bears theorem will help us in the computation of such prior probabilities right so this base theorem gives us a connection between the conditional probabilities of a given b and b q a and for two events a and b with probability of a greater than zero and probability of b greater than zero we know that probability of a given b is probability of a b divided by probability of b and now i can just multiply and divide this probability of a in this expression and you can see here what is this thing probability of a b upon probability of a this is simply your here the conditional probability of b given a right so now you can see here that probability of a given b is equal to probability of b given a into probability of a divided by probability of b well you also had found such a result earlier by computing the probability of a b right so now in case if i try to extend it then suppose we have here m events a1 a2 am like this so that even union a2 union am that is equal to sample space capital omega and a i and a g are pairwise disjoint that is i for not for i not equal to j goes from 1 to m a i intersection aj is phi right and probability of a i is is positive for all i and suppose b is another event then a then in case if i want to find out the conditional probability of a j given b then using this result which was only for two events this can be extended to m events and we can write down here this probability of b given a j into probability of a j divided by summation of i goes from one to n probability of b given a i into probability of a i and this formula is known as as the base formula or this is the called the bayes theorem and this is actually based on the name of english philosopher thomas base who had given this formula and here these probabilities what we have written here this probability of a is they are called as prior probability probability of a i given b they are called as posterior probabilities and probability of b given a i they are called as model probabilities so now we take some examples to understand the application of this law of total probability and base theorem so the first example is like this suppose someone rents books from two different libraries and sometimes it happens that the book is defective due to missing pages so here we consider the two events a1 and a2 so even is indicating that the book is issued from the library 1 and a 2 is indicating the event that the book is issued from the library 2 and suppose b is the event that the book is available and is not defective right so now we assume that we know that probability of a 1 is 0.6 probability of a 2 0.4 and probability of b given a1 is 0.95 and probability of b given a2 is 0.75 and based on this information suppose we are interested in the probability of probability that the rented book from the library is not defective i want to know this probability so now if you try to apply here the law of a total probability then we can use the formula here that probability of b is going to be probability of b given a 1 into probability of a 1 plus plus probability of b given a2 into probability of a two and if you simply try to substitute all these values over here you get here the value to be 0.87 so you can see here that this law of total probability is helping you in computing this type of probabilities and we may also be interested in the probability that the book was issued from library one and and is not defective so this type of probability can be computed by computing the probability of b intersection a one which can be expressed in terms of conditional probability such as probability of b given even into probability of a 1 which is 0.95 into 0.6 which is 0.57 and now suppose we have a non-defective book issued and suppose we want to know what is the probability that it is issued from the library 1 right so this can be obtained by computing the probability of a 1 given b so that is equal to probability of a 1 given b divided by probability of b and this can be computed here is 0.65 after substituting these values so you can see here from this example i am trying to give you different types of events that can be computed and similarly if you assume that we have a defective book somebody has got a defective book that is b complement event occurs now we want to know that the suppose the probability that a book is defective given that it is from library 1 is given by probability of b complement given a1 is 0.05 and probability and suppose the probability that a book is defective given that it is from library 2 this is probability of conditional probability of b complement given a2 so this is 0.25 for library 2 right now we can now calculate the conditional probability that the book is issued from library 1 given that it is defective as follows right so you can see here that we want to find out the probability of even given b complement so this can be written using the base theorem that probability of b complement given a 1 into probability of a 1 divided by probability of b complement given even into probability of a 1 plus probability of b complement given a two into probability of a two and if you simply try to substitute all these respective probabilities over here then you get the value here zero point two three zero eight now and similarly if you want to know the result about the probability of b complement that we have used couple of places then you can use this result directly probability of b complement is equal to 1 minus probability of b and you can obtain the value here 0.13 no issues so now after giving this a couple of example through this example one let me try to consider one more example suppose at a certain stage of a criminal investigation the inspector in charge is 60 percent convinced of the guilt of a certain suspect that somebody has made a crime and the corresponding police officer is 60 percent convinced right that possibly that person has done this so suppose now there is a new piece of evidence and that shows that the criminal has a certain characteristic such as the criminal is left handed means he or she works from the left handed bodiless that the the number of hairs on the head brown hair etc that type of information somehow becomes available from somewhere and if 20 percent of the population is possessing this type of characteristic we want to know that how certain of the guilt of the suspect should the inspector now be if it turns out to be that the suspect is among this group so how to compute this type of probability that you will see that is a very interesting example and you will try to see that how this base theorem helps so we try to consider here two events g and c so g is the event that the suspect is guilty so g is coming from the g of this word guilty and c is the event that he possess the characteristic characteristic of the criminal so the c is indicating the criminal so that is easy for you to remember so now ah we have that probability of g given c can be computed by probability of g intersection c divided by probability of c and so probability of g intersection c can be written as probability of g into probability of c given g which is equal to 0.6 into 1 which is equal to 0.6 so now we have suppose that the probability of the suspect having the characteristic if he is in fact inducent is equal to 0.2 so now to compute the probability that the suspect has the characteristic we condition on whether or not he is guilty and we try to find out probability of c which is the probability of being criminal that can be obtained as probability of c condition on g into probability of g plus probability of c condition on g complement into probability of g complement and if you try to substitute all the respective probabilities you get here the value 0.68 now using these two values probability of c and here the probability of g c what you have obtained here you can obtain the probability of g given c and this comes out to be here if you try to substitute all the values 0.882 so now we can conclude that the inspector should now be 88.2 percent or nearly 88 percent be confident of the guilt of the suspect you can see that how base theorem has helped police personnel to investigate a crime right now let me try to take here one more ah example to make this base theorem and its application to be more interesting suppose a plane is missing and it is pre-assumed that it was equally likely to have gone down in any of the three possible regions yeah may god forbid that this may happen but if it happened then people are trying to have a such operation so so this is based on those things so in these cases we have the probabilities actually which are the constants which are p i they are called as overlooked probabilities because they represent the probability of overlooking the plane and these probabilities are generally attributable to the geographical and environmental condition of the regions right so this one minus p i is going to indicate the probability the plane will be found upon a search of the ith region and when the plane is in fact in that region that means suppose the suppose we suspect that the plane has crashed in this region and we want to know it so in such a case we want to know the conditional probability that the plane is in the ith region given that a search of region one is unsuccessful right you can see i mean these types of things are very common whenever there is an accident people try to look for such a site and they try to investigate from starting from some place so they want to compute this type of probability they will be very helpful to them so we try to use the base theorem to compute such probabilities and let us try to see what happens so let this r i means r 1 r 2 r 3 these are the three possible regions so this r i be the event that the plane is in the region i and suppose this e is the event that the search of region 1 is unsuccessful that is given to us so now using the base formula we can obtain probability of r 1 given e probability that the plane is in region 1 given that the search of the plane in region 1 is not successful so now this can be obtained as probability of e intersection r 1 divided by probability of e and now using the base theorem you can rewrite this probability here as like this a simple extension and now i try to write down here this values that this the probability p 1 into 1 upon 3 and i try to substitute here all the conditional probabilities and another probabilities of r 1 r 2 r 3 here and we try to compute this thing this comes out to be p 1 upon p 1 plus 2 right and similarly we can also find here probability of r 2 given e and probability of r 3 given e also so that will also give us the probability of the event that the plane is in region 2 given that the search of region 1 is unsuccessful and similarly the plane is in region 3 given that the search in the region one is not successful so he can compute it exactly on the same line the probability of r j given e is equal to probability of inter section of e and r j divided by probability of e and if you try to simply substitute all the values as you have done earlier and try to substitute all the value this will give you here the values 1 upon p 1 plus 2. so for example if i say if the value of probability 1 that is the p one is zero point four then the conditional probability that the plane is in region one given that the search or the region did not uncover it is simply one by six you simply have to just put this value out there right so you can see here that this type of application will indicate that how you can apply them in different type of situation which you are trying to handle in the data sciences right now i come to the last example on bayes theorem and that is also an interesting example that many students try to do it well they have a very different type of belief and now i am going to show them that their belief is right or wrong you know that whenever we are trying to attempt a multiple choice question in exam then we then sometime we try to make a guess and based on the guess we try to choose the correct answer so this example is related to that one right so in answering a question on a multiple choice test a student either knows the answer or the student makes a guess so let a small p is the probability that the student knows the answer and one minus p is the probability that the student makes a guess that means he the other student doesn't know it as simple as that so assume that a student who makes a guess at the answer will be correct with probability 1 upon m where m is the number of multiple choice alternative you know that as a student that sometime if there are four alternatives and then you say many times that if there are hundred question and and if you take at random to them you will get 25 correct answer approximately so these types of statements are very popular among the students right so now you will see what really happens right so this is the same statement that what if a question has four alternatives then we are assuming that a student who makes a guess at the answer will be correct with probability one upon four right so now we want to know the conditional probability that a student knew the answer to a question given that he or she answered it correctly right so now that is a very interesting problem for you i am sure so now suppose there are two events c and k so c is the event that the student answered the question correctly and k is the event that he or she actually knows the answer now we try to obtain here the conditional probability of k given c that means the conditional probability that he or she actually knows the answer given that the student answers the question correctly so now i can write it down here probability of k intersection t divided by probability of c and using the base theorem i can write down here probability of c given k into probability of k divided by probability of c given k into probability of k plus probability of c given k complement into probability of k complement and if i try to substitute all the values over here which we have assumed we get here this answer so now if you try to see here that if you try to assume that there are four alternatives and and if you try to substitute here m equal to 4 and suppose p is equal to 0.5 then the probability that a student knew the answer to a question he or she correctly answered is simply 4 upon 5 right so now you can see that possibly this is a different result than what you expected so that is the main use of this base theorem right and now let me give you here one more example and suppose a laboratory blood test is 95 percent effective in protecting a certain disease when the disease in fact is present however the test yields a false positive results for one percent of the healthy percent tested what is the meaning of false positive that means if a healthy person is it tested then with probability 0.01 the test result will imply that he or she has the disease right so if you assume that 0.5 percent of the population actually has the disease then we want to know what is the probability that a person has the disease given that the test result is positive so this is the same example by which we started and now we are trying to come to this thing right so now in this last slide or the last example we assume that here there are two even d and e so d is indicating the even that the tested person has a disease i mean d means disease here this is how i have defined it and e is the event that the test result is positive so now the probability that we want to compute is given by probability of d given e and using the base theorem we obtain it probability of d given e as probability of d intersection e divided by probability of e and using the base theorem we can write it like this and we try to substitute here all the values what are given to us and we try to compute the probability this comes out to be 0.323 approximately so it is indicating that only 32 percent of those person who test result are positive actually have the disease right so we expected that this figure should be much higher because blood test seems to be a good blood test good in the sense that if a person has a problem it will say yes there is a problem or if not the bloods this will say no there is no problem but there are 32 percent chances which is not a small value right so now i come to an end to this uh lecture with this bayes theorem well i have tried to give you the statement of the bayes theorem and i have taken couple of example to make you convinced that how this bayes theorem is very useful in real application now if you try to see either this is conditional probability or base probability anything the entire probability function is going to be a function of this individual probabilities right so that is why it was very important for me to give you the information on different types of probabilities and now you can see that as you are going into deeper and deeper you are getting a better results now if i ask you that for a given incident if you have to choose simply the simple probability or the base probability or the conditional probability you know that how the results are going to be changed and how they are going to impact the outcome so now i will say try to look into your assignments books and try to take some example try to understand them and more important part for you to understand is how this individual values are coming how are you going to translate that what is a and what is a complement what is a given b and what is b given a that is the job of a data scientist the data will not raise its hand and say ok in this case you please find out the conditional probability of a given b or conditional probability of a given b complement and so on now one thing more that this conditional probability is just like simple probability which is having some nice properties and all those rules about this conditional probabilities are there but definitely if i try to take all the topics over here means we will miss some other important topics so i will request you why don't you take up a book and try to compensate that part which i am unable to do here in this course my objective here is something else so i am doing it from that point of view but definitely by saying this i cannot say that the topic was hundred percent complete so that depends on you how much you now want to learn the statistics so that you can use it in a much better way in a data science so you try to practice it and i will see you in the next lecture till then good bye [Music] [Music] [Music] you