Lecture 36 : Part 2 - Data Preprocessing and Error Analyses-II

Students, welcome back to the second part of lecture number 36. So, we have been discussing about data preprocessing, right. And I showed you this slide in the last part of the lecture. The main task of any data preprocessing are these four, data cleaning, data integration, data reductions and data transformation and discretization. So, let us talk about in detail on all these things. What is data cleaning? You know, if you have to tell me one sentence about data in real world, data in real world is really dirty. A lot of potential incorrect data, many times it is because of faulty instruments or because of computational error, human error or even transmission error, because sending some mission to moon or mars, you may have problem of transmission. So, each of these can lead to poor data or quality data. So, you need to have check on that. If you have incomplete data, that will lead to lacking of attribute values. You can have certain lack, you know, attributes or interest may not be present, right. Let us say occupation in a particular section. Occupation means, I am talking about population and set of people and we are doing occupation and if I make it occupation like this, that means, I am not providing the occupation data, correct. That is what it is. So, that is called incomplete. Incomplete data can be, because of many reasons, people could not find the data, that can be one reason or we could, did not want to provide the data, that can be also one reason. So, this is very important in statistics. Let us say, we are actually doing a census of population and we need to provide all kinds of data. This incompleteness is a big challenge, big problem there. Then we actually, many cases we do lot of analysis of the data and make common, make policies based on that. So, if your data are incomplete, then your policies will not be good. So, incomplete data is a big, big challenge in many of these machine learning, deep learning of material science. Second thing is noisy. Noisy means what? Errors or outliers. Let us say, I write salary is equal to minus 10, that is not possible. Salary cannot be negative, right. That is a noisy or erroneous data. You can also have inconsistency. Very simple example. If I say, somebody's birthday is 3rd of July, 2010 and write age of that person is 42, that is not correct. If somebody is born in 2010, the age of the person will be 14. It will not be 42. That is what is called inconsistency. Sometimes, you have to give ratings in 1, 2, 3 or A, B, C, right. That can also lead to inconsistency. Somebody gives rating in 1, 2, 3. Someone else gives rating in A, B, C. How do you know? What is the exact sequence of good or bad? 1 may be bad. 2, 3 may be good. A may be good. C may be bad. That is what the problem is. Inconsistency can happen in many ways. Internationally also, one can make mistake. It is called disguised missing data. Gen 1 as everyone's birthday. It is impossible, right. Gen 1 cannot be everyone's birthday. That is an international mistake. This can also happen. What I am trying to say? So, all these are actually possibilities of data, you know, cleaning problems. Well, what about incomplete missing data? In many cases, it is not available. What you can do? Like many tuples have no records, record of value of several attributes such as customer, income, data and many things. So, whenever you do experiment of stress-strain curves and you are doing a stress-strain experiment, in middle the experiment, the strain gauge did not work properly. You will find that your data are not available or missing, right. So, you have to redo the experiment. You may say, now why shall I do the experiment? I can generate data. Well, that is what you duplicate, right. So, missing data can be also because of that. It can be equipment malfunction. It can be due to inconsistency with other recorded data. Data not entered due to misunderstanding. Certain data may not be considered important at the time of entry. Some intelligent person thought it is not important or may not be, do not have raised to your history of changes in data and you know, this missing data will interfere your model performance, whatever machine learning models you are going to do. So, missing data need to be inferred. That is why, may need to be inferred. Sometime we can actually infer the data very easily. How to handle this? First is, you can ignore. Not effective when percentage of missing values or attributes varies constantly. This is not effective. You cannot ignore all this. Sometime, you can actually fill in the missing values manually. Well, that is very tedious and some cases, it is in physical. You may not know it, but you can do. You ask a machine to fill in. Like, you can do some, there is some global constant. It can do it. If it is attribute mean, it can do it. If it is a, you know, something else, you may not be able to do it. It is always not possible to automatically fill the data and many cases, we can think that, what is the most possible or probable value. That can be done using Bayesian formula or decision tree, by the way. So, machine can help or machine learning can help to fill in this data automatically. Manually, it is tedious and in physical many cases. Do not do it. So, therefore, we can write a machine learning code to automatically generate this data, automatically actually get mean and other things or you can actually do a reference based approach like Bayesian formula or decision tree to do that. How do you handle noisy data? That can be because of random error or variance in a measured variable. So, like, you know, data entry problems, transmission problem, I talked about it. Many cases may be due to inconsistency naming conventions, limitations, lot of things possible or it can be because of duplicate data or inconsistent data. How to handle that? That is very important, right, for you. Well, you can do binning. You have to do three things, four things actually. Binning, regression, binning, regression, clustering, combined computer human inspection. Very interesting, right? What is binning? First sort the data and partition into different bins. That is called binning. Bins mean equal frequency things. Then one can smooth the data by bins method like smooth by bin median, smooth by bin boundaries. Many bin methods are there. Bin median is very easy one. You can find out the median and then get the noisy data removed or one can actually do a regression. Very simple one is regression. Like smooth by fitting the data into regression function, right? Regression, we talked about it. You can fit the data to the regression function and then remove the noisy data. One can also do a clustering. We have all discussed about all these methods in detailed manner during machine learning. Detect and remove the outliers by clustering. One cluster of data will come here. This outlier, then it will come outside. Another cluster of data will come here. This outlier, it will come here, right? That is what we discussed clustering. So, outliers can be removed very simply by clustering. If you cannot get any answer from this binning, regression and clustering, then you can use combined computer human inspection. That means you can detect suspicious values and check by human. That is we always do, right? We have more intelligence level than this artificial intelligence. So, you can always use that and do that. But thus is the last step because if you can do it by binning, regression and clustering to remove the noisy data, it is better. I am not discussing each of these techniques separately because we have discussed a lot in our machine learning part of the course. So, data cleaning process can also be done. You can do metadata like domain, range, discrepancy, distributions, everything. It is like a metadata. You can also check overloading or uniqueness or you can use consecutive rules, null rules. All these kinds of rules are available in statistics. One can actually do lot of commercial tools available like data scrubbing, data auditing. These are available commercially. Use simple domain knowledge like if you want to do a postal code check or spell checks, you can easily do that to detect errors and make corrections. What is auditing? Auditing is nothing but analyzing data, discover rules and relationship between the different data points. That is what is called auditing. That can be done. These are all routine checks which can be done in by any software or any commercial tools. In many cases, you have to do integration of data or migration of data. Like, you know, I talked about integration or integration of different data files. Migration means what? Allow transformation to be specified. That is what is called migrations. And, you know, integration of the two processes can also be done iterative or interactive. That is what is called Porter's wheel. Interactive one. Obviously, you can do that. So, these are also possible. These are the different things also possible when you do cleaning a data. Now, I want to spend little bit on data integration. Now, let us talk about it. Data integration means, you know, understand, it is a combination of data from different sources in the coherent source which is required in material science law. Let us talk about thermodynamics data. They can come from different sources, but you need to combine to do a modeling. And, you can actually do a very simple thing called schema integrations. It integrates the metadata with different sources. Like this one. A.castId is equal to equivalent to B.castId something. That can integrate data. You can also do entity identification problem. Identify the real world entities from the multiple data sources. Let us say, obviously, this is not correct. Bill Clinton is not equal to William Clinton. It is correct. William Clinton is nothing but Bill Clinton. The W.I.L.L is converted to B.I.L.L. That is what is called. This is how we can actually identify the entity identification problem can be done. This is one example I am giving. We can also detect and resolve data values conflicts. Let us say, for some real world entity, attribute values from different sources are different. So, possible reason can be different representations, scales. Let us say, metric scale and British scale. They can also lead to this kind of conflict of data values. That can be resolved very easily. These are the different ways you can integrate the data. In material science, so much problem will not arise in terms of data integration, except you can have different sources of data and you have to integrate them. That is problem you will face. So, that can be easily done using schema integration. What about redundancy? In many cases, you will find lot of data which are not useful, redundant. You have to identify that. So, there are two ways of identifying it. One is object identification. Same attributes or objects may have different names and different derivables. That is what it can lead to redundant data or you can also have a problem with irreversible data. One attribute may be derived from the other one. Then it is a redundancy. So, if you derive some data from other one, then you are going to create lot of redundant data or if you use different names for the same attributes, then also you can create redundant data. That is easy to understand and take care of it. So, careful investigation of data from multiple sources can be done and then inconsistency or all this redundancy can be taken care of. That is what is your job. You can always check this carefully and integrate data. We can always do what is called correlation analysis. That is something which is very easy mathematically. Let us say it is called a c square test or it is called x square test. We call c square, c square means x square. Larger the x square value, more likely variables are related. This is what is statistical method. Cells that contribute to most x square value are those whose actual count is very different from the expected counts. This is nothing but a correlation. So, if you want to correlate, how do you correlate? By measuring certain parameters. There is one parameter x square test. Correlation does not simply imply that, let us talk about number of hospital and number of car thefts in a city are correlated. Is it possible? I think do not think so. Both are casually linked to a third variable called a population. So, car theft will depends upon population. Similarly, number of hospital also will depends on population. So, number of hospital as a number of car thefts, they indirectly correlated. Direct correlation is a population. This all can be found out by c square test. Let me tell you that. Let us talk about little bit more. Let us talk about one example of that. I am giving a table. Play chase, not play chase and sum. On the left side, you have like science fiction, does not like science fiction. You can see here, like science fiction is 250. All of them play chases. Do not like science fiction is 50. Total is 300. Like science fiction but do not play chase is 200. Does not like science fiction but do not play chase is 1000. That is the largest number. So, therefore, how do you correlate this? Somebody does not like science fiction but play chase. Somebody likes science fiction, does not play chase. That is why c square calculations comes in the picture. So, number in the parenthesis is actually like 90, 210, 360 and 840. Our expected count calculated based on the data distribution in the two categories. We are not doing it here. It can be shown by this, you know, when you have live sessions. We can always do that. But once you do the calculation, you will find out that. So, it shows like science fiction and play chase are more correlated in the group. Can you see that? Like science fiction and play chase has the lowest value. So, therefore, they are more correlated than the others. Least correlated is does not like science fiction, does not like chase. That is what the highest value 840. So, please go back c square calculation things in your statistic books. I am not discussing here, but that is very standard. So, therefore, we can actually call some kind of a correlation coefficient. That is very widely used. That is what it is. It is called a Pearson contact moment correlation or Pearson correlation coefficient. Many times people say PCC, Pearson correlation coefficient. It is basically R A B sum it over A I minus A bar. A bar means mean actually of that data A I multiplied by B I minus B mean. This A mean means is basically a mean in data set A I. B mean, B means mean in data set B I. And the sigmas are actually are standard deviations in A and B. Sigma A is basically standard deviation for data set A I. Sigma B is standard deviation for data set B I. So, you have to calculate mean. You have to calculate the standard deviations. Then, you can easily find out the Pearson correlation coefficient. So, you can see sum it over I equal to 1 to N A I minus A mean multiplied by B I minus B mean divided by N minus A 1 into sigma A into sigma E. That is very simple. So, now if R A B is greater than 0, that means what? A and B are positively correlated. This is very important in machine learning because many cases you need to provide whether some attributes are correlated or not. That is why this correlation coefficient is very important. Higher the values of R A B, stronger is the correlations. If R A B is 0, that means what? They are independent. And when R A B is negative, then it is negatively correlated. That means, they are not strongly correlated at all. So, when you talk about independent features in a particular machine learning model, this weak correlation is very important. If you are considering two features are strongly correlated, that means they are already dependent on each other. You remove them from your feature selection map. So, Pearson correlation coefficient gives provide that. So, this particular equation is very important. You know we are what you are measuring is basically mean and standard deviation from a data. That is what it is. We are not doing much. By that we can do that. So, compared to G square, this is most standardized method. Well, there are many methods of you know data reduction strategies because data reduction is another important thing. Why? Because you need to reduce the dimensional issue of the data. Many cases like image processing, I discussed about it a lot. It is a huge amount of data you will get. Many times it is not possible to handle them in a very nice way. So, you need to do a data reduction methods and that is can be obtained basically by doing lot of strategies. We will discuss about it. So, there is no need of talking why you need to do data reductions because there are terabytes of data. Many cases you may not be able to handle. So, first is dimensional reduction. What is that? Remove the unimportant attributes, unimportant features which can be done by wavelet transformations PCA or PCC. Just now we talked about it correlations coefficients or you can actually do by feature selections or feature creation, feature subset selections. That is also possible. So, you know wavelet transformation is what? This is basically method of statistical methods in which you can transform the data into particular waveform and then you can find out whether they are correlated or not. You can also do numeracy reduction like regression or log linear models, histograms, clustering, data cube aggregations. We talked about it. They are called data reductions. Regression or log normal, log linear models are very easily. Very easily you can do it. That way you can reduce the numbers or you can do a histogram analysis also. By that way you can do the reduction of the numbers. Some cases you can do data compression. That is important like in auto encoders you have a compression and decompression together. You can do data compression. So, data dimensional reduction is very widely used in deep learning. Many of the deep learning techniques actually use the data reduction, the dimensional reduction as a method of getting removed you know unimportant attributes. Well, the next 7, 8 minutes we will talk about some aspects of error analysis. How we can do that? I do not know whether I will get time for model score or not, but we will talk about it. You know error analysis. Why you want to do error analysis? Well, when it comes to artificial intelligence, one of the most important question is you know error. Very important. How much error we are incorporating? Developing a machine learning model has involved several steps like problem formulation, data collections, preprocessing of data, feature engineering, model building, deployment of model and monitoring. All these things. First you have to do a problem formulation. What is the problem you are handling? That is very simple. If you want to handle a problem of property correlation, structure property correlation, you have to only analyze mechanical or functional properties of particular material and correlate with what kind of microstructure you develop. That is the problem definition. Then you need data collection. You need to have lot of data. Once you do data collection, what you discuss right now? You have to do preprocessing of data. Lot of problems will be there in your data. You have to preprocess. Then once you do that, then you have to select features which are important things. After that you build a model. What kind of statistical model you will use for this purpose? That is called model building. Once you do the model building, you have to do a deployment and then finally a monitoring. But in all these things, when you take a deep dive, doing that we have to know that performance matrices of the models are taken care of. For classification models, we take care of different positions, F score, recall matrices to determine the model validity. In addition, performance matrices error analysis can also help and evaluate the models performance in terms of data rather than performance matrices. So, in a nutshell, error analysis is very vital in diagnosing any error made by the AI samples models. It enables data scientist or ML engineers to evaluate their model performance properly, identify the areas of improvements, investigating model errors. One can acquire insight into the quality and the relativeness, relative importance of the data and complexity of the problems also. So, in a short, you have to recognize the sources of errors. Then you have to assess, enhance the model performance by removing those errors and also maintain the model importance. So, it is very important as you understand vital process in diagnosing errors made by ML models during training and testing. Error is always defined. Defined what? It is defined as a difference between predicted values and actual values. It is may be because of observational error or measurement errors. We can determine the model performance, identify the areas of improvements. We can also determine insight into the quality or relevance of the data, complexity of the problem, effectiveness of the feature engineering and model selection. Nonetheless, from this slide, you understand that error analysis is very important. So, first thing is to recognize the sources of error. Then, assess and enhance the model performance and mental the model relevance. These are the reasons why you will do the error analysis. Recognize the sources of error is very important. Then, you can correct it. Then, you have to take care whether because of this error, how much is the problem in model performance and how we can maintain the model relevance by removing this error. So, there are three aspects. So, first error is what is called a mean absolute error. There are many ways you can measure the errors. First one is called mean absolute error. This is the mean of all absolute errors by the way. Correct? Very simple. It is a measure of average size of mistakes in a collection of data. Very simple. MEA is given by, you know, very simply you can write down mean absolute error. It is called MAE, mean absolute error. Machine learning people call MAE and this is nothing but simply 1 by n sum it over i 1 to n. Then, you take a, that is what it is. I am sorry, square is not there. Square is not there. So, that is something. Many people write this one like this. Let us put it that since this y cap is mean. So, you can see we are measuring difference of each data point on a mean. That is what is done. Sum it over. Correct? And then, divide it by number of data points. That is what is called mean absolute error. This is very easy. You can find out that error. So, what you need to measure is the mean. Mean is a statistical things from data. You can easily measure means and take the difference between of each data points and the mean. And then, sum it over and divide it by the number of data points. And this is widely used by the way. Why it is widely used? Because, it is a very, you know, basically it is more important in regression models and very easy to understand, interpretable, very reliable also. And it also gives more significance in terms of error. It is also resilience to outliers. Why? It is not impacted by extreme results or any other matrix. Mean square error, which we will talk about next, is kind of affected by that. It can also give a linear score. Can you see that? It is not square. It is a linear score. Right? So, all individual differences are given equal weightage. We are not giving any special weightage to the difference between a value level and its mean actually. So, that is why it is called linear score. It is very straight forward. Very straight forward. Everybody can understand. Same units are the response variables. It can be used in many disciplines and it can also offer information about the size of the error. There are many advantages of this. That is what I am trying to say. Many advantages of this. See, MEE is that is why it is a widely used technique, error technique, error methods used in this kind of aspects. That is something which one has to remember always. Why MEE is important and why it can be done so easily? Please try to remember that. So, I am just giving you list of reasons. Next is what we have already talked about. It is called MSE, mean square error and it is similar. Mean square error is similar. So, first is mean absolute error. Mean is called MSE and this equal to again 1 by n sum it over j 1 to n y j minus y cap j square. You see the difference between them. Here, you are taking absolute difference. Here, you are taking the square of the difference. That is why it is called as mean square error. Mean square error is computed by calculating the average, especially the mean and of the square mistakes. It is talking of square mistakes compared to the absolute difference between the two errors, two values, level data and the mean data. Here, you are talking about the square mistakes and it is a measure of error in prediction algorithms. So, this is statistically quantifiable and there are no errors in model. MSE will equal to 0. That is very unlikely. Every model will have error. Model worth increases in proportion of the degree of errors it contains. Average square error is called MSE. That is not important. Important one is MSE. So, mean square error is used in regression most widely. You know, it might indicate average square residuals also. You can also take error of residuals when residuals are nothing but the difference between the actual value and the error. That is what is called residual, right? So, but nonetheless, MSE is also important method of calculating the errors. Last thing is called R square error. We will discuss today. Let me stop. R square error is basically coefficient determination. It is a measure of, measure that provide information about the goodness of fit. And you know, what is it? Let me write down. So, R square error is basically, it is a coefficient of correlation R square and R squared error. Am I clear? And it is basically measure of provide that information about the goodness of fit. So, what is it? This equal to ratio of two things. One is your xi minus x square, i is equal to 1 of s. And basically square, sum square of regression minus total sum square. That is what it is called. So, basically better to write the next equation at the bottom. That is better. So, 1 minus sum square of regression. That is this. And that is what is called xi, yi minus y, yj minus yj cap whole square divided by, divided by yi, yj minus y bar. So, that is what. Let me explain to it. What is it? So, it is basically again widely used in case of, in case of your regression analysis. So, as you can see here, this is giving you the values of this one, the next equation. Focus on this. This is giving you values of yi minus yi cap square divided by yi minus y bar square. So, difference between y cap and yi is basically what? This is first is because of the, this is the residual square to the total square, sum of squares. So, this is residual square, yes, residual square and this is your, this is your total sum of square. That is what it is. So, this is how actually one can find out how it can provide information about goodness of fit. We always have. Well, in the next class, we will discuss little bit more on this. Let me stop here because, you know, we need to discuss some more aspects like, you know, the moral score, RMSE, many things we will discuss. These aspects we will discuss in the next class before we start the discussion on material informatics. Thank you.