Lecture 36 : Part 1 - Data Preprocessing and Error Analyses-I

Students, let us start the lecture number thirty six and you know we have been discussing about the advanced deep learning, but today we are going to talk about something else. Today we are going to talk about data preprocessing and error analysis and as I said we are in the last phase of this course, we have another five lectures remaining. So, for this lecture we are going to focus on data preprocessing and error analysis. From next lecture onwards we are going to discuss about material informatics and that is will consist about four lectures. So, let us see today's topic data preprocessing and the error analysis. As usual before we start this topic we always recap of what we have learned in the last lecture. In the last lecture I talked about natural language processing and what is famously known as NLP. NLP is very famous, it will be discussed more when we talk about material informatics. Deep learning is a very important technique, we have discussing or we have rather discussed many of these techniques supervised learning, unsupervised learning as well as hybrid learning. I do not want to go into details of these techniques again, only to highlight that please read this paper which is given at the bottom by given at the bottom by Iqbal H Sarkar and it is a very interesting paper in which you can learn many more much more than what I have discussed in the classes. Since deep learning is relatively new in material science engineering except the portion on artificial neural network. So, we got to learn lot and that is why you need to read various kinds of literature available in the internet, but I find this one is quite good one. We have discussed that about this advanced DL and in the advanced DL is something which we can used to combine different techniques, like we can use combined supervised and unsupervised techniques, because unsupervised techniques provides are more adaptable to the both level and unlevel data. On the other hand supervised models are only able to learn from level data, they are not good or not actually cannot be used for unlevel data. So, if you want to combine these two, if you want to make a framework of training both these supervised and unsupervised models simultaneously, we can enjoy the benefits of both the models. That is why hybrid models came to the picture and they are typically consisting of multiple deep learning models, where the basic model can be supervised unsupervised model and then this can be integrated with either other unsupervised supervised models or with some classifier. We discuss about it and I do not want to go in details about again those topics which may be repetitive, but you know NLP is kind of a machine learning based approach, but it can also be adapted in deep learning algorithm too. You know it is something which will require for material informatics, because material informatics is a combination of language tools with computer science and data science. And why do you need that, because you know human communicate each other using words and text and the way human convey information to each other is called a natural language. We speak different natural language depends on your mother tongue and what else you learn English or any other language. So, every day we share lot of information each other in using this kind of language. The question is can I make computers or machines to learn this. The natural language processing is defined as a branch of artificial intelligence that can provides computers the capability of understanding the text, whether it is spoken or written in the same way human does. What is the use of it or why we do it, because if we do that it can incorporate machine learning model statistics as well as deep learning models into computerized linguistics that is what it is. What is that? Computer linguistic is nothing but a rule based model of human language. As you know every language is a grammar, how do you use text, how do you use words, how do you connect words, this all depends on the grammar of the language. English has one kind of grammar, Swarajkita has another kind of grammar or many in your mother language, mother tongue will has another kind of grammar, but based on the grammar we are actually using or making the text sentences. So, can I actually combine machine learning, deep learning or statistical models into a computational linguistics or rather a rule based modeling of human language, which can allow computers then to learn understand the text, what are written, spoken words, what the way I am speaking to you and also understand the human language in terms of intents and sentiments. So, you understand that that is what is the NLP is supposed to do. Yes, it is a far phase thing at this moment, I know what is you are thinking about it, but that far phase things will not even far phased after few years, because NLP can provide us huge information. Those we are recording videos, uploading into social media or any other platform, can you use them to learn, can you use to use them to garner data. Similarly, we publish a lot of papers, manuscript, data or you know technical reports, it is like a billions of billions of that, can I use NLP to extract information from that, that is why NLP is very vital. It is going to provide you all the information you need to model and information means data and you know machine learning and deep learning requires data, without data you cannot model. So, this is a generator of data in a nutshell. So, in order to do that, we have to make computers to learn the rule based modeling of human language, that is the main task. And how does it work? Well, it is combines three important fields, it obviously combines human language, the language you speak with AI tools and computer science, AI tools means statistical things, that is what it is. So, it combines linguistics, that is the language, computer science as well as AI. AI is why? Because it requires machine learning and deep learning to be used. Why it is required? Because it is can then allow the computers to what comprehend, breakdown, separate significant details from the text and speech. Remember, when you speak, we speak many things, but information may be very less. It is not like a classroom, where a teacher is trying to give you as much information as possible. When you talk to each other, you speak many things which are not useful, which does not carry information. So, computers needs to learn that, they need to filter those things which are not information and collect the actual information and give it to you, that is how the NLP works. So, NLP needs to learn the linguistics, combine it with AI tools like machine learning and deep learning and as well as computer science to bring that one package, which can solve all the problems of information. I am talking about that, because finally, our goal is to use extensively NLP to get data from different sources and data is the prime thing in machine learning and deep learning. What are the different phases of NLP? Tell me, what are the different steps of NLP? Well, first thing is to do is a morphological processing and then we can do syntactic analysis, semantic, what is the actual meaning? We can also do a discourse integrations and finally, do a pragmatic analysis. Let me tell you again this, what is lexical analysis or morphological analysis? In this case, we basically break down sentences, paragraphs, into small segments called tokens. These are actually smallest unit of text, which a computer can understand. So, what you have to do? Computer needs to scan the entire text, whether it is a spoken or written and divide into meaningful tokens or meaningful lexemes, that is what is the job is. You need to scan the entire text, source text, divide it and you know when it does it, when it scans the entire text, it will find many unuseful things, useful things or meaningful things will be less. That is what it has to do, it has to separate that meaningful things from the unmeaningful things and that is why we need to divide into different tokens, this is the first step. Then after that, once you have done that, we can actually check, whether this small sentences, which has been formed or tokens is formed, whether they are really well formed or not. It might be that what word assume argument is not proper, not as per the linguistics. So, word assume argument is studied and syntactic relationship is found between them and this word assume argument is based on the grammar as well as syntax, that is what it is. So, therefore, syntactic analysis also called a parsing. In this case, the sentence is checked, sentence or paraphrase or broken small part of a sentence is checked for whether it is well formed or not, that is what you should understand. And how do you check this well formed or not, based on the words, arguments and the grammar. After that, once you have done it, you made it grammatically correct, everything is proper, then you have to check, whether sentence has any literal meaning or not. You may write a sentence, which is grammatically correct, but it does not carry any meaning. Let us say, if I say sun is setting on the middle of the sky, it is correct sentence, but it does not give any meaning. We know that sun cannot set in the middle of the sky, either it has to, it has to set in the west, when it goes down at the end of the day. So, that is what is called literal meaning. So, one can easily form a sentence grammatically correct, but it might not mean anything sensibly. Correct meaning is very important, that is what is done in semantic analysis. Semantic means, basically checking of the sentence, whether it carries any meaning or not. Then we need to also look at, whether the impact of a sentence, which you have formed before or after, has taken care of the aspects of the continuity or not. Impact of a sentence is before a particular sentence or even after a particular sentence also. And effect of this current sentence on upcoming sentences has to be determined. Basically, when you form a paragraph, you have a thought process, you write things in sequence. I went to market, then I looked at the vegetables, I checked the price, then I bought something. You never write, I bought vegetables, then I went to the market, that is what it is. So, therefore, this impact of sentences before a particular sentence or effect of the current sentence on the future sentence is very important, that is what is done in a discourse integration. You understand how it is done, it will be done very easily, because we know that human languages are sequential, they are never haphazard. Our actions are also sequential, the moment you do haphazard actions, people will call you mad, that is what is the difference between the proper discourse integrations and the other things. And the last thing we do is called a pragmatic analysis, this is the last phase. And what is this, sometime this is integrated with this, you know, discourse integration. Actually, effect of the text is discovered by this step, by applying a set of rules, that will characterize the cooperative dialogues, am I clear. So, it is effect of the text, effect means, when you read the whole text, that has to be discovered by applying a set of rules, right. It cannot be simply without any rules, it can be open ended, that can characterize the dialogues are properly done or not, that is what is done in a pragmatic analysis. And you know, that these are the different steps of NLP. Now, how NLP is implemented, NLP implemented by using both machine learning or deep learning as well as statistic inference. So, machine learning is automatically, it will focus on, you know, how, what are the rules, how it can be learned, all these aspects. And statistical inference is basically, it is a algorithm, it helps you to produce model that are robust, continue the words or structures, which are known to everyone, that is what it is. It must be robust, the models must be robust, that is what is called statistical inference. We cannot use any models to implement for NLP, because that will not work. So, therefore, containing words or structures, which are known to everyone is very important. So, what are the steps? Obviously, first you have to do segmentations, you have to do segments of sentences, then form the smallest units called tokens, then you have to remove the stop words. Stop words can be only a, you know, when I stop intermittently between two sentences, that is a stop word. In a written text, it is basically a dot, you put a dot between two sentences, stamming, lemmatization, parts of speech tagging and name entry tagging. Let us see all these things first. So, segmentation is very simple, it is a breaking the whole entire documents into constitution sentences. And we do this by segmenting the article along the punctuations, like full stops and commas, that is the best way. You can use full stop and commas to segment the article into small mock sentences. But once you have done that, each sentence can be very big, you may not be able to, computer may not be able to handle. So, what you do? Then you have to take a token, you have to create a token. Token is nothing but a smallest unit of words, which has some meaning. You need to get the words in a sentence and explain them individually to the algorithm. So, when you break down this sentence into this constitutional words and store them, they are called tokens. And each word or each phase is basically called a token. Basically, you are breaking a whole sentence. First in segmentation, you are breaking the whole paragraph into sentences, many full sentences that can be done based on full stop and commas. Then in tokenization, you take one sentence at a time and divide into different parts. Each part may contain several words or one word or several phrases, that is what is called a token. That helps computer to make pieces, bits and pieces and then take the whole thing into one go. You can only make this learning process faster by getting rid of non-essential words. We do that, right. When you speak to friends, many times you use lot of non-essential information words. Why do you use it? Even if you listen to some of the lectures in the social media or YouTube or wherever, you will see a lot of things are spoken, which are not much depending on the overall theme of the things, which the person is speaking about it. So, these are called non-essential sentences words. They have to be removed, because they are not adding any meaning to the, little meaning to the statement. Therefore, thus they have to make the statement sound more cohesive. So, words were such as was, in, is and are are called stop words and they can be removed. That is what is called removing words. So, then we can also do stemming. This is a process of obtaining word stem of a word. You know stemming for a plant or tree, there is a stem, right. So, word stem, word stem gives a new words upon adding epixes to them. You can add epix, prefix epix, you understand, right. You can have epix, after the word you can add something and make a word stem. You can also do lemmatization. What is that? The process of obtaining root stem of a word is called lemmatization. Root stem gives the new basic form of the word, that is present in the dictionary and then we can actually verify how it is derived, right. That is what is a difficult word. We go back to the root stem and then find out what is the meaning in dictionary. You can also identify the best words for different word based on the tense mood agenda. That also, a word can carry different meaning in a different sentence. How do you do that? So, we have to go back to the root words or root stem of the word and then find out the meaning of that words. That is called lemmatization. These are extra step, remember. Stemming and this one connected, but they are extra steps compared to the three steps earlier I talked about. Obviously, you have to do this parts of speech tagging. Otherwise, how will you analyze a sentence? Whether it is a determiner, noun, verb, adjective, preposition, you have to know that. Conjunction, exclamation signs, everything you should know that. So, you must explain this aspect, root words. Like here, the stars are twinkling at night. Determinant is basically, the noun is, there are two nouns stars and night. There are verbs are basically are and adjective is twinkling, preposition is at. That is what it is. Therefore, we can actually tag each of this as a one parts of speech. That is how actually we learn. When you learn sentences in school first time, you learn this parts of speech is learnt totally. Even before you form a complete sentence, parts of speech is taught. Because, you should know which part will go to where. Like articles or determinants will come at the beginning. Verbs will come in the middle. Adjectives and adverbs will come later on. Last thing is what is called a named entity tagging. What is that? Introduce the machine to a, you know, if you want to introduce a name machine to a pop culture reference. Everyday names by flagging names of movies, important personalities, locations that may occur in their documents. That is called named entity tagging. You can do this by classifying the words into subcategories. Like, you know, somebody is famous in music, somebody is famous in movies, somebody is famous in that. That is called subcategories. The words can also be done like that way. This will help you to find keywords in a sentence. Remember, if I say somebody so and so has done this or acted in this movie. That gives you keywords like the name of the person, name of the movie and acting. That is what these are the keywords. So, named entity tagging is basically done to find the keywords. Subcategories are obviously, person, location, monetary value, quantity, organization, movie, songs, whatever it is. After performing this preprocessing steps, you can give your result and data to the machine learning algorithms, different types. Name based is one of them that can create the NLP applications. So, all these things will be performed first. These are called preprocessing. Then, it will be put to a machine learning model to create the NLP applications. So, you know, in material science, there are many types of NLP coming into picture. I did not discuss all of them because it is not possible. So, first of all is basically like the three things I talk of materials, word embeddings. What kind of words? Input feature that entity for entity recognization models, right? Like, you know, these are called word 2 v's, fast text, l memo, all kinds of things are there. You can get it in this particular, you know, Olivetti's paper, very interesting paper or you can do annotate material text also. Annotate plain text synthesis of paragraphs and materials. This will allow you to turn training data for entity relation models or data mining for material science things. You can also do text mine query or nil temper like, you know, text mining or data mining is similar things. Data mining all of us understand. Text mining is similar to data mining. Training data for entity linking models that maps material mentions and properties. Simply, you can find out those words in the text and map it with the properties, right? That is possible. So, we can actually generate lot of query search from downloaded papers and then you can select even material science papers. Lot of papers are available or you can generate or you can actually take material science corpus and then you can do this math, science, but analysis and to find out, do all kind of tasks like entity recognitions, relationship classifications, abstract classifications. You can do all these things. Please read this paper by T Gupta et al and you will find out. So, that was what we discussed. Many more things we discussed. We will come back to it when you talk more about in material informatics. So, let us talk little bit about data preprocessing and we will carry forward this to the next part of the lecture also. So, you know, data is very important. All of you must have understood this, right? Without data, we cannot do machine learning or deep learning. If data is more important or data are more important and critical, I must make sure I have a quality data, right? Because if I start with the wrong data or non quality data, I will be putting garbage in and I will be getting garbage out. That I must not do. I must check my quality of data. So, data preprocessing objective is to ensure good quality of data. That is your input. Remember, your input is not good quality. Output will not be good quality. That is what is called garbage in and garbage out. So, major task in data preprocessing, then what? First is you have to clean the data. Many cases you have to integrate data, reduce data and there are cases where you have to transform data or discretize data. Am I clear? There are many more, but these are the three, four important major tasks we do. Data cleaning, data integration, data cleaning is very important, data integration, data reduction, data transformation and data discretization. These are the four important aspects of data preprocessing. Mind it, idea or objective is to improve the quality of data. How do you do it? Let us see that. Well, first of all, how do you measure quality of data? First of all, if I say I want good quality data, I must have a measurable things. Without measurable things, how can I say I have got a good quality data? So, there are multidimensional aspects of that. First is accuracy, which is very good. When you are doing experiments or reporting data for experiments, you will like to make sure you have report accurate data. You do not fuzz any wrong data. You do not generate any data, which is not part of experiments. Correct, but how many of us are very ethical about it? So, many times we find whatever data we get are not so accurate. So, that is why you have to find out correct data in comparison to wrong data. How do you do that? We will see later on. Second thing is completeness. Well, when you are reporting some data, it must be complete. If it is an incomplete data, I cannot use it for a model. Third thing, so that is very important. It should not be unavailable or it should not be recorded also. Many times what happens, you are doing an experiment, then you stop experiment at some time and you say it is not recorded. That is called incomplete data and this creates huge problem in machine learning or deep learning model, because then machine cannot learn it properly. The learning curves are very bad. Third thing, which is called consistency. Some modified, some are not like dangling bonds. There is no consistency. Dangling bonds can be any types, immaterial signs, but they are important. So, the consistency of data must be ensured. If I am giving values of heights of a student in a class and then I measure in generating, in getting the data, I will be providing inconsistent data. In many cases, timeliness is very important. Let us say, you are doing an experiment. You have got some data, publish it and then after publishing the paper, you claim that more data will be provided later on, but if you do not provide timely update on that, it is not being used. There are few more things, which are very difficult to quantify. These are very easy to quantify accuracy, completeness, consistency, timeliness. Few things, which are very difficult to quantify are believability and interpretability. How you quantify data trustable? Very difficult to say that, because trust is something, which depends on me. I am doing the experiment properly and jotting down the data properly. That depends on me. That is why the ethics comes in the picture. So, I cannot have any answer, how to quantify believability. Interpretability, how easily data can be understood? I may provide data in a such a form, it is very difficult to understand by, to be understood by the common people. Then, it is not useful. So, that is called interpretability. So, you understood these are the five, six parameters on which quality of data can be measured. Accuracy, completeness, consistency, timeliness, believability and interpretability. Out of which first four things, that is accuracy, completeness, consistency and timeliness can be measured, can be measured by numerical values. Believability and interpretability, it depends on person to person. That is about the things. Now, coming back to the task. So, that is how we can measure the data, you know, data quality. Now, how do you do the processing? First is obviously, data cleaning. So, we have to first fill in the missing values. Smooth, noisy data, identify or remove the outliers and many cases, you have to resolve the inconsistencies. You will find that data, which are giving, let us say, you plot stress versus strain and you see, if I say stress versus strain, you will understand that. Let us say, it is stress versus strain and its elastic portion look like that. Let us say, something is missing here. Let us say, I am doing like this. So, now, if I see data like this or like that, I could clearly see this is inconsistency, because I know in a plastic part of the curve, it should be straight line. So, these data are inconsistent. They cannot do any job to us. That is what is. So, we must be able to resolve that. That is what is called data cleaning. So, first is the missing values. Then, you have to check for smooth, noisy data, identify or remove the outliers. There will be outliers always in experiment. We will find that and obviously, resolve the inconsistency. Second step, that is about data cleaning. Second step, data integration. Many cases, you have to integrate multiple databases or files and you should know how to integrate that. If you use odd data files to integrate, then it is not useful. Major task, another task is called data reduction. We have talked about it about dimensional reduction, numericity reduction, data compression. Remember, we talked about auto encoders. There you can compress and decompress. We have talked about data dimensional reduction in many of these models of deep learning. We will go back to my classes. You will understand that. The data reduction is very important, because of these aspects, you need to do dimensional reduction. You need to do compression of data like image processing. You need to compress the data in compressions. The last thing is, you are called data integration transformation, data discretization. Data transformation, I mean normalization. In many cases, you need to normalize the data. That is what is required for modeling. Otherwise, it will give you wrong result. Like in case of, you know, I talked about it in the last part of last section about the length scale issues in material science. When you have 10 to the power minus 6 size grain size, 10 to the power minus 6, 10 to the power minus 6 micron size grain size versus 10 to the power minus 9 or nanometer size grain size. You are talking about some properties. You need to do normalization. Otherwise, it will not be able to handle in many of the machine learning models. In many cases, you have to also use hierarchy generation, clustering. In clustering, this hierarchy generation is very important. Where from the data started to originate and where it is going towards. These are called data transformation and discretization. You have to discretize the data separately or transform it. These are the important aspects we should know. So, we will come back to the next part of lecture, where we talk about other aspects like data cleaning in detail manner.