Lecture 35 : Part 2 - Advanced Deep Learning-IV

Students welcome back to the second part of the lecture number thirty five. So, we have been talking about natural language processing or NLP. I explained you what is it and we are talking about different phases of NLP. As I said there are five different phases of NLP, lexical analysis, syntactic analysis, semantic analysis, discourse integration and pragmatic analysis. We will talk about each of these things in detail manner. Lexical is something which is very easy to understand. Any paragraph sentence is broken into different small parts or units called tokens and that way it can help computer to divide each sentence into meaningful lexemes which are nothing but meaningful phases. In syntactic basically we check the sentence whether it is well formed or not. Here we check for arrangement of the words and grammar. You see that we have to check for arrangement of the words and grammar and word arrangement is very nicely studied here. We have to find any syntactic relationship is there between them or not. That is what is syntactic and grammatical. Therefore, you need to check arrangement of the words, which word is coming after whom, whether they are properly placed or not or they are connected by grammar or not. Then we need to also talk about semantic analysis. What is semantic analysis? This is the third phase. In this phase sentence is checked for literal meaning of each word and the arrangement together. You may say something which is allegorical which is very difficult to understand by common man. You say in a some way. Many times you read a novel, you will see the author of the novel is writing in a different paradigm. You have to get the meaning of that whole chapter to get understanding of the subject. That is what is called semantic analysis. In this phase each sentence is checked for literal meaning. For literal meaning can be only obtained when you study each word properly. That is something which is not easy for a computer. Please understand that human can learn this thing easily because we are you know we are told that this indicates that, that indicates that. We are told from our childhood, but if a computer learning this is not easy. That is about semantic. Now what is discourse integration? In this phase the impact of a sentence before a particular sentence and after a particular sentence can be understood. If I put a sentence, what is the impact before a particular another sentence? I can either play the sentence before the sentence or I can play the sentence after the sentence also. So that is something which is to be determined by this way. That is why it is called a discourse integration. How that the whole thing is connected, integrated together. That is what is called discourse integration in NLP. The last thing is called a pragmatic analysis. This is the last phase. In this phase of natural language processing, sometime discourse integration phase and this phase are combined. What is actually done in this phase? Actually part of the text is discovered by applying a set of rules and they are characterized the cooperative dialogues. So now I have put this all these things together in this board. You can see there are five different things analysis. Lexical analysis, sentence are broken into tokens or small parts. Syntactic, sentence are checked. Sentence is checked whether it is well formed or not simply like words and grammar. Semantic, actual literal meaning of the sentence. Discourse integration is whether if I have a sentence placed in a text, what is the impact of the sentence before and after this sentence? That can be determined. Finally, pragmatic analysis is actual effect of the text is discovered. These are the things which are the different parts of NLP. NLP uses each of these as one mechanism to do analysis. How is this implemented? These are all different parts of NLP. We will discuss about it. How this is implemented? They are implemented basically by two ways. One is by machine learning. Learning NLP procedures use different kinds of machine learning techniques. It can automatically focus on certain common cases, certain common text. You can allow them to learn. So, when you write rules by hand, it is often not correct at all because it will have lot of human errors. That is why it is not proper. You have to use certain kind of a text where computer can learn and then it can keep using it. Second thing is important aspect is statistical inference. NLP can make use of statistical inference algorithm. It helps you to produce model that are very robust. That means containing word structures which is known to everyone. That is what is statistical inference. Any language whether it is English or any Indian languages, we have certain kind of word sentences which is known to everybody. Everyone knows. It does not require to learn in the schools or colleges or become pundits or become very literate person. It does not require. If we can make the computers to learn this kind of things, then it can process the common language or common way of speaking of human and do the analysis, do the things much better way. That is the idea. So, these are the two things which are combined for implementation of the NLP. Now what are the different steps of NLP? Obviously, first is segmentation. You have to divide the sentence to different segments and that is very important. Then you do tokenization. Your words, text, some parts of text. You can also remove stop words. There are lot of words which will make the sentence stop. Stemming, lemmatization, parts of speech tagging and name entity tagging. These are the different things. There are actually seven different steps of NLP. What are these? So, what is segmentation? Well, you know, you first need to break the entire documents into different constituent sentences. That is what is called segmentation. If you have one page document, you can divide into different sentences. You can do this by simply segmenting the article along with the punctuation like full stops or commas. That is the easiest to do. All of us do that. If you read some books, you will see that there will be sentences which will intermittently stop by comma or a semi colon or it will be permanently stop by full stop. That is what is easiest way of segmentation. So, you have to tell computer to that. When I am speaking a sentence, I am flicking it and stopping at the end of the sentence. Then I am starting a new sentence. That is the human intelligence. But how computer machine will learn it? We have to tell the computers that this is how the segmentation is done. In a text, written text, it is easy. There is always a dot or there is always some kind of a segment in Indian languages like this to stop. So, but in this, when you are speaking, it is not there. Wherever you are, when you are speaking, wherever you stop momentarily, that is the end of the sentence. That is what is called segmentation. Now, what is tokenization? We already talked about tokenization. In a particular sentence, you can divide the sentence in small parts. So, first you have a entire document which you divide into constituent sentences. That is called segmentation. Then for each sentence, you can take a, you know, different parts of the, you know, sentence or different phases of the sentences and make tokens. So, you need to get words in a sentence and explain them individually to our, to the algorithm. So, what you do? You break down the sentence into constituent words and store them. That is what is called tokenization and each word is called as token. When you go to the bank, you always get a token number, so that your turn will come and you will be called, right. It is exactly same thing. Here, you are breaking the sentence into constituent words and then storing them and therefore, each phase is called, a phase may be couple of words or three words. That is what is called a token. Why it is done? Because that helps computer to learn how these words are put together to form a sentence. See, remember, when you learn in childhood, as kids start learning words, they speak one word, two words at a time. They do not know how to speak a sentence. Then they slowly learn how to connect these words to make a sentence. Here, we are doing opposite way. Here, what we are doing? We have a sentence. We are dividing the sentence with different parts and in such way, the whole meaning of a sentence is not lost, but the different parts of the sentence is used, stored. That is what is called a token. Well, there is another thing called removing the stop words. You can make this learning process faster by getting rid of non-essential words, which you speak many times. Whenever we are talking to each other, with friends or something, we use lot of non-essential words, which does not carry any meaning. These words are not important to computer like our machine to get information out of it. Therefore, they are just there to make our sentence sound more cohesive, like words such as was, in, is and are, are stop words. They can be removed easily. I was in the market yesterday. Important thing is that yesterday you went to the market. It is not important thing. It is not the word was. So, catch words are you, yesterday and the market. That is why these are the stop words. Was, is, in, the, are, the, itself. All the articles actually are stop words by the way. It does not matter articles are there or not. So, these are the stop words, which you can ignore or remove. Computer does not need these words. They are going to, basically, they are basically non-essential words. That is how they are removing these stop words. Then, what is stemming? This is a process of obtaining the word stem of a word. Word stem gives new words upon adding affixes to them. You can add affixes. You know affix and suffix. Affix is before the word, suffix is after the word. So, this word stem gives you new words upon adding affixes. That is what is that. This is many times is required to do this. What is, then what is lemmarization? This process of obtaining a root stem of a word. Root stem gives the new base form of the word that is present in the dictionary and from which the word is discovered. Let us talk about a word called assassination. It is a big word. What is the root? Assassine, to kill a person. That is what is the root. So, that is what the lemmarization does. It does the process of root stemming of the word. Root stem gives the new base form of the word. You can also identify the base words for different words based on the tense, mood, gender itself. That is also possible. Remember, these are all actually linguistics of the language. Now, you know we also need to do parts of speech tagging. That is very important. Parts of speech, because whenever you learn a language, you always learn parts of speech. Noun, pronoun, verb, adjective, preposition, everything. Determinant like the a and the noun like stars. See, the sentence is the stars are twinkling at the night. So, the is the determinant. Noun is the star. Star is the noun. Verb is are. Adjective is a twinkling. Preposition is at. Now, again night is noun. So, you must explain the concept of this noun, verbs, articles, other parts of speech to the machine by adding these tags to the words. They are called part of or part of speech. We call it very interesting. Is it not? Because this we learn when we are a child in the schools. So, now, we are going to make the computers learn these things as a part of NLP. Why? Because NLP will allow to extract the information by reading these sentences. That is what is important for us. Now, important aspect is this. Most important aspect is what is called the name entity tagging. It is called NET. It is not net. NET. And introduce a machine to pop culture reference. And everything names flagging by flagging themes, names of movies, important personality location etcetera. Let us first explain that. Name entity tagging, name entity recognitions or name entity tagging, what is called a natural language processing method, which is sub category of this. But what does it do? That is what let us discuss about it. I think that is very important. It is a process of extract information from text. It is referred to entity chunking, entity extractions and entity identification. What are entities? Like earlier given, the entities of you know pop culture references. Like pop is a music. So, what kind of pop culture references you want to do? That is it. So, you do this by classifying the words into sub categories. They help us find any key words in a sentence. These sub categories are persons, locations, monetary values, quantity, organizations, anything, movie. After performing these preprocessing steps, you can give the resultant data to a machine learning algorithm like name base to create a NLP application. So, like name entities, a name entity is an object of any object can be referenced by a name person. Recognitions, NER systems are trained to recognize these objects and sort them into helpful classifications. And you know there are different types of NER systems available, entity system available. One is called dictionary based and second is called rule based. Dictionary based in these systems reference terms will be listed in the dictionaries to identify a particular, they are present in the text. Dictionaries can be a collection of words. We know dictionary is a collection of words with a meaning. So, we can create a dictionary in a computer and then put this, classify these words for each of the meaning. Rule based, they are nothing but you know set of instructions or rules are given to the computer so that you can classify the words. That is what it is called. So, what you understand from this discussion is that these steps segmentation, tokenization, stemming, lemonization, parse of speech and name entities allows you to understand how the sentences are structured, how they are developed in a particular stanza or particular book. That is what is done. So, natural language processing has many uses, hundreds of uses. It can be used for writing aids in foreign languages, find and replace words, correction of spelling and mistakes, development of the writing of aids, foreign language questions and answer systems, conversations. Important ones are summarizing the text, extracting information out of the text, there is access to information and understanding the text. These are important for us, material science engineering people. So, how to use these things in an NLP, in a materials design? As you know materials have highly diverse source of physical data. Let us talk about synchrotron. You know synchrotron is basically a source which can allow you to do x-ray diffractions, you can allow to do lots of soft studies, characterization studies. So, that generates huge information, but most of the information are kind of collected wherever synchrotron source is there. Like India, your synchrotron source in Indore, that is why all the information are collected. This information are stored there and then user takes the information and may analyze them and publish something, book or paper or patent, whatever it is. This is something. Let us talk about Newton's facilities. Newton facilities are also limited, mostly the atomic energy plants have Newton facilities. So, Newton is used for diffraction information again, diffraction studies and this diffraction of materials will generate lot of data and that is something which will be stored there and then some of people which are users of these facilities will take the data, publish a paper. So, if you want to have as, you know, if you want to extract the information from these publications, because you cannot access the Newton diffraction database or information from there, you can only access the information from the published literature. How do you do that? That is why NLP can be used very widely. Many times, bench top laboratories can also generate lot of data, which are published later on. Let us talk about high performance computers. You know, when you do high performance computing, how many of you are publishing all the data? Very few, I mean, I think null, nobody publishes. They take the data, analyze it and put in a particular form and publish it. Now, you ask your NLP to get that original data out of it. That is a very big job. So, these are the things. So, in material science, you have highly diverse data and that is the biggest problem when you want to, you know, compile the data and curate the data. That is the biggest challenge. So, that is why in a general work of NLP, that is what is called the language processing. First thing is the collection of data. How do you do that? Most of the papers are published. They are available in terms of pdf or html. If you look at carefully, papers or patents, whatever, they are published in terms of pdfs or html. They are most, many times, they are either stored in a website of a journal or in the clouds. All of you know cloud, I hope so. Therefore, we have to extract the data from there, from these texts. That is the first thing for a machine language, language processing of data. That is what is the first thing. So, once you do that, then you have to do some kind of a text classification. As we have already discussed about it, text classification, you have to divide the text into different classes depending on what kind of text it is. I can go back and tell you that again, like here, different kinds of text possible, like analysis, like. Therefore, first you divide that sentence into different parts or tokens. Each part will have some meaningful meanings. Then, you decide whether they are grammatically correct or not. When you divide, it should be grammatically correct. Otherwise, what is the point? Then, you do the analysis of literal meaning of that. Also, you do this, how the sentence is coming out of the box, out of blue or their sentence are connected in the whole theme of passenger or paragraphs or so, the text. Therefore, that is what we are talking about it here. We are talking that we have to classify the text into different kinds of things. Once you classify the text, you can also do kind of a data mining. Second step is preprocessing of these aspects. After you collect the data from these PDFs and the HTML, you have to preprocess that. This preprocessing is what is called preprocessing of data, which we will discuss today or may be in the next lecture. Once you preprocess, then you can classify your text. Once you classify, then you can have information extraction. You have to classify the text, whether something is experimental, something is computational, this is one class. Then, experiments, mechanical power determination, functional property determination. Then, classification in terms of chemical synthesis or physical synthesis. This classification you can easily do. So, after you do the classification, you can extract the information from these kinds of classified text. Finally, you can do a data mining. That is the theme of language processing. That is what is done. So, material language processing has emerged, a very powerful tool that is why, in the release of material science. That will facilitate the extraction of the value of information from a large number of papers and the patents and the development of knowledge base. It uses the NLP to analyze, understand the language, using the material science text, enabling the identification of key materials and properties and their relationships. So, basically I need to create a database, curated database. From where we are getting information? Published literature like papers, reports, patents, etcetera. From there, we can get a curated database. That is what is done. So, material language processing actually, that is why it is very important. And as I told you, compared to physics and chemistry, we are not so, you know, into this kind of curated database, because of diverse nature of our research, diverse nature of our publications. So, let us give an example of that. Let us talk about battery material. So, you know, there are lot of papers, you will find, which you will talk about battery materials. Some day, we will be talking about true papers, which are true, talking true about battery materials. Some of them, we will talk about. Some papers, which talk about overview of something, which are called false papers. We can, first thing, we have to classify that. So, therefore, we have to use a double descriptors in the embedding space. Am I clear? That is something, which has done. There are many ways one can do. One can actually use, you know, levels, which are crude levels, which are design levels. We can use verbose levels, all kinds of things. And there are few techniques like batteries, but cases like SOTA or something can be done also. So, idea is that, task is to classify whether a given document is true or not, because we are talking about battery materials. So, input, the importance of the nickel based batteries in the present context is extremely valuable as their electrochemical properties are expected to offer significant insights into the application of battery output is true. Let us say another input. Introduction of contact deformities as a dermatitis to cosmetics in a common problem in general population. Although this prevalence appears to be underestimated, we review the cases of allergic contact dermatitis to cosmetic diuretics in our dermatology department. That is false. It is not battery problem. Let us talk about another thing. Electrochemical performance of NCRO2 as a positive electrode material for intermediate temperature sodium secondary batteries was evaluated. This is kind of classifications computer has to do, machine has to do. So, MLP does that. That is what it is. It is the idea of MLP. So, some more examples. We can actually even fine tune these things. Let us say these are the different papers published in APL, CMT, DSC, Material Science, PO, SMT. I do not even remember what are these journals are. Now, fine tuning will allow us whether each of these papers published in these journals, how much they are talking about battery material. What is the level of information available? If it is completely talking about battery material, it should be 100 percent. Some of the things, but if it is not, it will be less than that. Let us say something like 80 percent is still better than many other things. So, first is we can always use prompt like here we are talking about reduced graphene oxides, nano hybrids, metallic zinc powder produced in step one, synthesis of enhanced photo current and catalytic response, reduced graphene oxide hybridized. These are the texts and from these texts, we should be able to extract information about the battery or electrodes. So, what are the NLP tools? How they are connected? First is the waves scrapping, the content acquisition and what is called markup parsing. There are different kinds of this kind of tools available and then we do a tokenization and lemmatization. We have discussed about it earlier. So, thus we will generate, you can see that came data extractor, came spot. These are part of the preprocessing, text preprocessing things. They are tokenization. Once you do that, then you can do the document segmentations. That is nothing but text summarization or text classification. There you do a lot of semantic analysis. We say what is the actual meaning of the text and you can also find latent deletion allocation also. What kind of latent information is available? Third thing, fourth thing is what is called entity recommendations. Here, you have to use rule based approach or rather you can use dictionary approach or machine learning based approach. Dictionary approach is better that provides you specific rules of the sentences, how they are coming in the picture, how the grammar is done and finally, you can do entity adulation and linking also. So, this is in a nutshell the flow of NLP tools in extracting information. Some examples are given like lithium, strontium, manganese as electrode material. So, you see there are different kind of extractors like came extractor, chemical tragger, came spot 201, banner, came in under lead mine. So, each of these does information like that like came extractor will direct the chemical tables. Came spot will do chemicals and what are the approaches they use? This kind of approaches whether it is a CRF, Oscar. There are two important as approaches they used which will again which will use all the features of NLP by the way and you know. So, now, if you want to go into details of that of NLP, you can have different kinds of material like polymers, magnetic materials, superconducting material, photovoltaic materials, material processing, material characterization, material synthesis, all kinds of things. That is your text. Text mining will do that. Once you do that, then you can actually have a data visualized in front of you. Text is analyzed, data curated. You can value the data and you can use it for analysis also. So, what you do that? After that you can actually use this data for optimization, this classification via machine learning program. You can do encoding of empirical issues or you can actually do transfer learning. All the things which you learn, you can use here. Region tree or you can use hierarchical classifications, you know, trending networks. All the things which you learn, I taught you or can be used there and from there you can actually get lot of material property predictions, material characterizations, informations. You can even discover new materials like you can do magnetic phase diagrams, you can do thickness of the barium, BIFE2O3. You can do all kinds of things. So, that is what is actually, what is the dream. That is what we want to do in future from the text to the model to the information. So, this is something which is also recently come into picture, whether you can use images. From the images you can calculate informations only. The deep learning methods do this, does this very well and analyze the images or even the figure captions. You can see that, analyze the figure captions, not images, sorry. Can analyze the figure captions to get informations out of it or segment the information out of it. That is something which is possible. So, figure captions will have materials words embedded, annotate material text embedded or even text, you know, mind things will be there also like, you know, Curie and Neal temperatures. We can get it from figure captions and then we can curate the data from there. Well, so there are many more things we can talk about it. Let me stop here and in the next class, we will discuss mostly about the data preprocessing as well as the error analysis in machine learning. So, that will be probably the last lecture on deep learning techniques and all these things. Then from lecture number 37 to 40, we are going to talk about material informatics. There again, we are going to come back to the NLP for some part of the lectures. So, let me stop here. Thank you.