Lecture 05

[Music] in this lesson we'll discuss probability distributions and different types of probility distribution namely binomial distribution and normal distribution probility distributions such as normal distribution are often employed to draw inferences from the sample data for example how does one make the following inference what is the probility that a data scientist in the US earns between $97,000 to $115,000 we'll try to explore such questions in this lesson we'll start the discussion with basic rule of probability such as single and multiple events rule of addition and multiplication and joint and marginal probabilities we'll also discuss conditional probability and base theorem in the context of probability models we'll discuss random variables expected value of random variables variance and cumulative probabilities these building blocks on probability models will also help us understand more advanced topics related to sampling statistical inference and hypothesis testing we'll be introduced to important probability Concepts such as intersection and Union of events mutually exclusive events and Complex events in this video we'll understand probability models probability models Empower stats to draw inferences from sample data and make statements like the average salary of a data scientist in the US is $97,000 plusus $39,000 with 95% of confidence or the exit polls for this election show us with 95% confidence that party X will win 53 plusus 3 seats in the Senate how do we get to these conclusions in this video we'll discuss the theoretical underpinnings regarding basics of probability probability distributions and statistical inference we'll also discuss the formula and an understanding for using the prob driven influential statistics in everyday life we'll also discuss in detail the theoretical Foundation to be able to understand these Concepts in the section of basics of probability we'll go through the building Concepts probably already heard or have a good sense of the general formula for probability but you will need to learn some more advanced concepts to be able to estimate probability for more complex cases for example estimating the probability of multi events can be more complex than just a single event let's say the probability that a data scientist has a master's degree is relatively easy to calculate but consider the following questions what is the probability that a data scientist has a master's degree and earns more than $120,000 what is the probability that a data scientist either has a master's degree or earns more than $120,000 are these same or not if not then are they at least related and if yes in what way okay what kind of information do we need to calculate these numbers our discussion of single and multiple events Associated rules of probability that is the addition and multiplication rules and joint and marginal probabilities will enable us to take these calls moreover events can interact in other ways too for example given that a data scientist earns more than $120,000 what is the probability that they have a master's degree is it the same as the reverse condition that is given a data scientist has a master's degree what is the probability that may earn more than $120,000 are the two at least related if yes then in what way such conditional event relations will be covered in the section on conditional probability and the topic of Base theorem will give us a very powerful mathematical formula for calculating them all of these building blocks will help us move on to the next building block which is probability distributions probability distributions as the name implies help us visualize and quantify how probabilities are distributed across the range of possible values for example what is the probability that a data scientist earns more than $997,000 and less than $115,000 what is the probability that done between $115,000 and $133,000 as you can see from this distribution even though both the intervals are about $188,000 long the probability of earning $115,000 to $133,000 is is much lower that is because of how the probabilities are distributed and in the section of probability distributions we'll Deep dive into these concepts of discrete variables and continuous variables Having learned all these building blocks you'll be able to proceed to the final section on statistical inference which is the one that will give you all the tools to draw inferences from samples for entire populations we'll start out with the primer on samples and sampling going through the nuances of how to sample and ensure that our sample for example 2,000 data scientists whether it is a good representation of the entire population which here would be all data scientists in the US or not once you have the samples we'll go through the process of making sampling distributions from this data and then using the center limit theorem to extrapolate from it and make inferences for the entire population in other words we'll use the sample data to estimate the average data scientist salary which will be in the form of of an interval estimate for example $97,000 plusus $39,000 which is margin and $97,000 is average and with an Associated degree of confidence we will learn to estimate the confidence intervals as well in the industry there are a lot of tools that can be used to directly apply different statal techniques some of these tools are like python R SAS SPSS Excel and M tab however the most common tools used in the industry are Python and r and this is true for a variety of reasons both Python and R provide a wide variety of statistical techniques such as linear and law Newar modeling Etc also you can easily create quality plots and include mathematical symbols and formula with these languages unlike many other tools these Python and R are free to use in particular R is very popular among statisticians so in this module and in the course we'll be using r as a primary tool for demonstrating different concepts random events and sample space at some point in life you would have heard that the probability of something happening for example a stock giving returns above certain level is 80% or a particular party winning election is 50% but what is this probability and what do these numbers mean to us let us try to understand this with a simple example in most of the sports that you watch whether it is Cricket football soccer or basketball tossing a coin is a very common feature in cricket the coin might be tossed to decide which team gets to bat first or ball first while in soccer it might be to decide who kicks off in the first half now in these coin tosses are you always sure whether it is going to be a heads or tails well no assuming that it is a Fair coin you may not so let's take an example of a fair coin a Fair coin is a coin that will have the structure or the make identical for both the heads and tail ta sides so that when you toss it the chance of getting head is same as the chance of getting Tails now let's go back to the example where a coin toss is happening between the Liverpool team and Manchester United Team in the English Premier League the captain of Liverpool calls for heads what do you think will be the chance that the Liverpool Captain wins let's try to calculate it you can have only two outcomes heads or tails since both are equally likely to happen as in the case of a fair the chance that the Liverpool Captain wins is calculated like this number of favorable outcomes divide by total outcomes which is equal to one head divide by two head and Tails which is equal to 0.5 the chances of something happening like this is known as probability in other words in this particular case you can say that the probability of winning the coin toss is 0.5 or 50% th probability is a language of uncertainty now let's formalize some of these terms related to probability the coints is known as an experiment here the set of possible outcomes that is head and Tails is known as the sample space sample space can be mathematically represented as omega symbol which is equal to two possible outcomes that is heads and tails in our example the probability of the Liber pool Capital winning is known as the event here let us try to reinforce our understanding of probability and other terms related to it with the help of some more examples let's consider rolling a dice that has has six faces what is the random experiment here it is the rolling of a dice what are the possible outcomes they can be 1 2 3 4 5 or 6 hence we can write our sample space as follows Omega = to 1 2 3 4 5 6 all six events are possible now what if I ask you what is the probability of getting one in the rolling of a dice you can easily calculate this with the formula number of favorable outcomes divide by total outcomes which is equal to 1 upon 6 that is 1x 6 what about the probability of getting an even number here the number of favorable outcomes are three that is 2 4 or 6 and the probability becomes 3x 6 that is 0.5 in this example rolling a dice is a random event or random experiment that is rolling one or rolling an even number here are the events for which we find the probability a random experiment is a process that leads to one of several possible outcomes an event is just a subset of all the possible out comes of an random experiment I hope this helps us differentiate between a random experiment and an event now what if I say that the formula of probability that we have discussed till now that is the number of favorable outcomes divide by total outcomes may not apply at all such instances let us consider the marks scored out of 100 by a student in a math test here the random experiment is that the student scoring marks in a math test what can be the outcomes of the test the possible outcomes can be 10 20 99 or even 0 or 100 essentially any number between 0 and 100 let's consider that the evaluation happens only in whole numbers and not in fractions so that the sample space becomes the set of whole numbers between 0 and 100 which is 0 1 2 and so on up to 100 that is 101 possible outcomes now if I ask you what is the probability of the event that a student scores 80 in the math test will you apply the probability formula that we have just seen that is number of favorable outcomes divide by total outcomes let us try it out according to the formula that will be 1 upon 1001 will be the probability of a student scoring a team that test do we think this is right may not be so because not all the experiments are designed to make all outcomes equally likely just like you can have an unfair coin you can have a test that is designed such that the people at least get a passing grade or at least a certain set of minimum grade so how would we calcul the probability in that scenario well we can always use pass data to find out empirically for example if I tell you that there were 50 students in the class who took the math test out of which 10 students scored 80 10 students scored 90 and the rest scored 60 then what is the probability of a student scoring 80 marks if you use the formula that is number of favorable outcomes divide by total outcomes that is 1 by 3 because the sample space becomes 608 90 this would be incorrect hence in this case the formula has to change a little bit and you can consider it as frequency of favorable outcome divide by total number of outcomes frequency essentially the number of times an experiment is repeated with the outcome in our case the experiment is repeated for 50 students and the frequency of favorable outcome is 10 hence the probability becomes 10 upon 50 which is 0.2 now there are certain important things that we should note here the formula that is number of favorable outcome divide by total outcome in the case of coin tossing and rolling a die experiments because first the different outcomes of experiments do not depend on what happened in the past in the case of dice and coin second different outcomes for example heads and tails or in the case of coin toss and 1 2 3 4 5 6 in the case of D roll are equally likely the chance of one coming up is same as any other number in a d roll similarly the chance of heads is equally likely as the chance of getting tails in a coin TOS however this is usually not the case with most real life experiments as noted earlier this is also not the case with the math exam which is designed in a way that one outcome that is scoring zero probably and most likely is not same as scoring 100 or 60 or 80 probability definition by now you have understood that we calculate probability when we want to understand the chances of an event happening in a random experiment probability is associated with events that have unsure outcomes in fact most things in life have unsure outcomes for example it might or might not rain on a given day hence there are two possible events rain and no rain and each of these two events have a certain probability associated with them probabilities are always between zero and one let's look at a formula to understand this better probability equal to frequeny of favorable outcomes divide by total frequency of all outcomes now the frequency of favorable outcomes can be zero for example getting seven in a die roll or maximum equal to the total frequency of all possible outcomes for example if you define the event as getting 1 2 3 4 5 or six in a d role the event will have a total probability of one hence the probability of an event in a random experiment can only be between zero and one a probability of zero means that with a certainty that event will not occur a probability of one means that an event is certain and it will occur definitely for example the probability that a day will follow the night is always one unless some Cosmic event occurs where the sun disappears let's hope that does not happen at times probability is also represented as percentages so zero correspond to 0% and one corresponds to 100% you may have watched The Weather Channel and they will tell you that there is a 20% or 30% chance of rainfall on any given day someone may give you an investment advice and they may tell you that an investment has a high probability of succeeding or not succeeding in medicine you have certain probabilities you may have a certain probability for a surgery being a success in business you may have a certain probability of customer turnout a probability that a customer may pay a loan back or default on the loan if you are in sales or marketing you may assign a certain probability to a customer purchasing a product and there are various ways of calculating probabilities there are so many examples as if life itself is a game of probabilities isn't it combining two or more experiments let us say that instead you flip two coins what would be the sample space here let's figure it out the experiment of tossing two coins can be thought of as two step in which Step One is the tossing of the first coin and step two is the tossing of second coin now if you use H to denote a head and T to denote a tail then H comma H indicates that the experiment outcome with a head on the first coin and another head on the second coin so if we route down all the possible outcomes we will get HH HT th and TT and therefore the total possible outcomes would be four that is 2 cross two what do you notice two is the sample space of first coin HT and also for the next two it is the sample space of second coin toss as well in step two that is H HD now if we are to generalize this for a multi-step experiment we can say that if an experiment can be described as a sequence of K steps where N1 is the possible outcomes in the first step N2 is the possible number of outcomes in the second step and so on until we have the K step which is NK then the total number of possible experimental outcomes would be ns1 * N2 * N3 and so on up till n k in such multi-step experiments we can represent this in the form of a t diagram step one corresponds to the first coin toss and step two corresponds to the second coin toss for each step there are two possible outcomes head or tail on the right end of the tree you can see all the possible outcomes of this experiment you would notice some key points here first one is that experiments need not be only a single step they can be multi-step consequently the sample outcome can be a complex representation of the multi-step experiment and it need not be just numbers now let's extend our learning to a more real life use case consider that you are working in a data scientist role and you are starting a project that involves two stages one stage is the data preparation stage and the other stage is the model development stage your task is to estimate the possible completion time for the project here is some more information about the project an analysis of similar projects in the past revealed possible completion times for the data preparation stage stages 5 6 or 7 weeks which is equally likely and possible completion times for the model development stages second 2 3 or four weeks again equally likely what is the random experiment here it is the two-step process as part of the machine learning project which involved the data preparation and model development stage what do you think can be different events here for example can completing the data preparation stage in 5 weeks be an event here yes why not can completing the machine learning project in exactly 8 weeks be an event yes of course can completing the model development stage in 3 weeks is an event yes it is now let's try to find out the possibilities and probability of the occurrence of these events for that let us create a decision tree like we created in the coin tossing example discussed earlier here in the last column we added the numbers in the sample outcome to get the completion time of the whole project as seen earlier the number of sample outcomes the number of outcomes Step 1 into number of outcomes step two is 3 into 3 which is equal to 9 now let's find the probability of completing the data preparation stage exactly in five weeks you can see that the frequency of favorable outcomes are three the first three hence the probability is 3x 9 which is 1x3 similarly what do you think is the probability of completing the model development stage in exactly 3 weeks again since there are three possibilities that is 2 3 and 4 weeks out of which one is 3 weeks the answer would be 1x3 then what is the probability that the project will be completed in exactly 8 weeks we can see that in two of the nine experimental outcomes the project gets completed in 8 weeks th the probability is 2x9 or 22.22% similarly we can also compute the probability that the project will completed in 9 weeks or less types of events in this video let us compute and estimate the probability of finishing the project for example in this particular case if we are estimating the probability that project will be finished in 9 weeks or less we can see that there are six possible outcomes out of total nine outcomes that satisfy this condition that the project is being completed in 9 weeks or less hence the probability of project being completed in 9 weeks or less is 6id 9 that is 66.67% now what is the probability that work will be completed in more than 9 weeks also is this question somehow related to the previous question there are three possible outcomes out of the total nine possible outcomes where this condition is satisfied and therefore the probability works out to 1x3 now if I can subtract this 1x3 from Total probability 1 we get 2x3 which is 66.67% this is precisely the probability of getting the project completed in 9 weeks or less how did we do this we could do this because because between these two events let's say the project being completed in 9 weeks or less we call it event a and project being completed in 9 weeks more is event B we can call them complimentary events in other words the sample outcomes or sample points that are not in event a that is 10 11 12 are the sample point of event B and vice versa hence we can write event b as event AC or a complement to event a this can very well be represented as a v diagram as you can see in this diagram the area enclosed within the rectangle is the complete sample space for the experiment and the area shaded in the yellow represents event a thus the complement of event a that is event a which in our case is also event B is represented by the area that is not in a which is the wide portion within the rectangle since the probability of complete sample space is one we can say that PA plus PC equal 1 which is the probability of event a plus probability of event AC or event B is equal to 1 where p presents the probability of event a and Pac is the complimentary area to a or event B this rule also extends to the basic rule of probability that is the sum of probabilities of all the events in given experiment always add up to one intersection events we learned that an event is a subset of all possible outcomes of a random experiment or a subset of the sample space within events we can further classify events into simple events and compound events EV a simple event has only one outcome whereas a compound event has two or more simple events as outcomes hence in our machine learning project completion problem which is a multi-step experiment compound events like completing the data preparation stage in exactly 5 weeks can be broken down into three simple events completing the data preparation stage in five weeks and the model development stage in two completing the data preparation in 5 weeks and the model development stage in 3 weeks and lastly the third one is completing the data preparation stage in five weeks and the model development stage in four weeks note that all the three events have just one outcome in the sample in the example of rolling a dice the event rolling a one is an example of a simple event but the event rolling an odd number will be an example of a comound event as it can be broken into three simple events rolling a one rolling a three or rolling a five now let us consider the event completing the data preparation stage in five weeks and model development stage in 2 weeks we can see that it has a composite or intersection nature of two events the first event is completing the data preparation stage in five weeks which can be said as event a and the second event may be completing the model development stage in two weeks for event B for event a the sample outcomes are 52 53 and 54 for event B the sample outcomes are 52 62 and 72 the intersection or the common of these two events is 5A 2 mathematically this is represented as a intersection B intersection is represented by inverted U symbol and in this case that outcome is 5A 2 in the v diagram shown here on the screen events A and B are two events that have some overlapping occurrence hence the area shaded in green is the event representing a intersection B the probability of completing the data preparation stage in five weeks and the model development in stage two is one so there's one possible outcome 5 comma 2 and therefore the probability 1 by 9 that is the possible outcome and the divide by total outcomes let us try to understand this using a simple example where two dice are rolled simultaneously can you tell how many outcomes will be there in the sample space of course it is 6X 636 let us Define two particular events event a and event b as follows event a where die rolls and the first die has a number as 1 there are six possible outcomes 1A 1 1A 2 and so on up till 1A 6 here you must note that each outcome has two values as they represent the individual outcomes of each of the two dices so if I consider 1 comma 4 as event we mean that the first die is rolled as one and the second die rolled four similarly you can see all the possible outcome of event a as listed here now we can Define event B also where the dice rols as the number number five the possible outcomes of this event are again six that is 1A 5 2A 5 and so on up till 6A 5 thus the intersection of events A and B is the value common in both the sets which is 1A 5 in this video let us understand the union of events going back to our machine learning project completion example what if I ask you to determine the probability of the event completing the data preparation stage in five weeks or completing the model development stage in two weeks as seen earlier let us name these two events first completing the data preparation stage in five weeks as event a and completing the model development stage in two weeks as event B for event a the sample outcomes are 5 to 53 54 for event B the sample outcomes are 52 62 and 72 the or criteria represents the UN of these events and we determine this by writing all the possible outcomes that belong to these two events which are 52 53 54 62 and 72 mathematically the union of events A and B can be represented as a union b or a or b which is 52 53 54 62 and 72 if we use the V diagram between event A and B then as shown here the Shaded area in green represents the union of A and B now let us again consider the dice example where we are tossing the die two dies simultaneously event a is that the roles where the first die has number one so the sample outcomes or sample spaces 1 1 1 2 1 3 1 4 1 15 and 1 16 that is six possible outcomes event B is the roles where second I has number five and hence the sample outcomes or sample space is six possible outcomes that is 1 15 25 35 45 55 and 65 so for the outcomes of event a union b or a or b we have listed all the possible outcomes present in event a all the outcomes event B and the common event which is 1 comma 5 which is common to both event A and B and therefore the overall number of sample space events are 1A 1 1A 2 1A 3 1A 4 1A 5 1A 6 2A 5 3A 5 4A 5 5A 5 and 6A 5 these are all events that are part of a union b or a or b so now that we have understood the event compliments and finding the intersection and the Union as well for two events let's move on to another type of event which is mutually exclusive events in the next video mutually exclusive events mutually exclusive events are also known as disjoint events so we can say that two events are mutually exclusive when they do not occur at the same time for example if a student has been awarded grade C in a particular subject in an examination then they cannot be awarded Grade B in the same subject in the same examination so events number one a student being awarded Grade B in a subject in an exam and number two a student being awarded grade C in the same subject in the same examination are mutually exclusive or disjoint events we can see that in the v diagram shown on the screen here you can see that events A and B do not have an overlapping region this is a clear indication that the two events are mutually exclusive or disjoint one interesting fact for you to work out is that complement events are always mutually exclusive but not the other way around let's take the rolling two dice example and define the following events event a is roles where the first dice has the number one so the sample outcomes are event B is the roles where the first dice has the number three and the second dice has the number five so the sample outcomes are three and five looking at these two events event a and event B of di roles you can see that two events have nothing in common that is there is no intersection between these two events hence these two events are called as mutually exclusive probability for Complex events until now you learned what complement events and mutually exclusive events are and how to find the intersection or Union of events we have worked a lot on the machine learning project completion example let us now work on a different classic problem let us say that you have a bin containing four balls of different colors yellow green red and blue now let's begin with a simple problem in how many ways you can draw two balls from the bin with replacement this means that once you draw a ball you put it back in the bin for the second draw in the first draw you have four balls so four Bas and similarly you can draw four ways in the second draw hence you have 4 cross 4 equal to 16 ways of drawing the two balls with replacement now what if I ask you how many ways you can draw two balls with replacement such that both are yellow in each draw you have only one yellow ball so there is only one way in which both the draws are yellow balls this can also be understood by drawing a tree so now having known the two values those two values are number one number of ways to draw two balls with replacement is equal to 16 and number of ways to draw two balls with replacement such that both draws are yellow is equal to one thus we can say that the probability of drawing two balls with replacement such that both draws are yellow is 1x 4 into 1X 4 that is equal to 1x 16 now let us look at another way to find the probability of Complex events what is the probability of drawing a yellow ball it is 1x4 now if you remember drawing two yellow balls with replacement is a compound event and is the intersection of two simple Events first event being drawing a yellow ball in first draw that is event a and drawing yellow ball in the second draw that is event B for both both the events the probability is 1x4 and if You observe closely the final answer is nothing but 1X 4 into 1X 4 which is equal to 1x 16 in other words probability of a intersection B intersection being represented as inverted U is equal to probability of a that is PA into probability of B that is PB this is known as the multiplication rule of probabilities this multiplication rule Works only when there are two events and those are independent events are independent if and only if the occurrence of one event does not affect the other so if there are four independent events a b c and d then the probability of all these four events happening together becomes PA a * PB * PC * PD this multiplication rule can also be applied for dependent events and the formula changes slightly now you must not confuse independent events with mutually exclusive events mutually exclusive events are events that cannot happen together and we can represent this with with the help of a v diagram on the other hand independent events are such that the occurrence of one event is not dependent on the other event and we cannot represent the same using a v diagram so here's a simple question for us what would happen to the probability of A and B happening together if the events are mutually exclusive as you know mutually exclusive events cannot happen together thus the probability of A and B that is P A and B or P intersection B becomes zero in practice if you want to figure out whether two events are independent or not you can find the probability by creating a tree and cross verify by applying the multiplication rule now let's extend this example to a more complex one let's consider that the bin now has 20 balls and that out of all the 20 balls there are four yellow balls five green balls six red balls and five blue balls what is the probability of drawing a yellow ball you can draw four ways out of 20 so it is 1x which is equal to 0.2 now what is the probability of drawing two yellow balls with replacement since we know that it is this event is an intersection of two events the event first or event a being drawing yellow ball in the first draw and event b or the second event being drawing yellow ball in the second draw both of these are independent events let's apply the multiplication rule that is p a intersection B = to PA a into PB = 4x20 into 4X 20 which is equal to 1x 25 which is 0.04 then the multiplication rule made our life very easy but now let's take the same sample to see what happens if the events are now dependent what is the probability of drawing two yellow balls if we do not replace that ball after we have drawn it well of course that changes the probability so we still have 1/ fifth chance of drawing the first yellow Ball but for the second yellow ball the probability is no longer 1if this is because now we we have a bin that has 19 balls in it and three of them are yellow so the probability of drawing two yellow balls without replacing it now is 1/ 15 * 3 out of 19 and the probability here is little bit lower that is 0.032 so this was a simple example of applying the multiplication rule for the dependent events because when we drew the ball and did not replace it the probability of the second event got affected [Music] [Music] [Music]