Lecture 27 : Part 02

Students, welcome back to the part two of lecture number twenty seven. So, we are discussing about recurrent neural network, right. And this is very important for NLP, natural angle processing. And what is that I told you? Recurrent neural network is basically a type of artificial network that you sequence your data or time series data. Somebody is speaking or somebody is typing something. How it can predict what the next word will be? That is what is recurrent neural network does. Or somebody can predict what is the next sentence I am going to speak. That also is part of neural network. So, this is basically time series data. Remember, sometime back we talked about it, right. Sequential data, that is the time series data. Time series data can be in terms of speech, in terms of images, in terms of videos also. So, that is why you can combine CNN and RNN very easily, right. So, RNNs are commonly used for language translation. What is language translation? All of you know, right, language translation. Whenever you want to translate something like Chinese to English, German to English, language translation is very important, right. Google translator does it. That is actually uses RNN network. Or you can also use NLP, I told you, natural language processing or you can also use speech recognition. Speech recognition is very important to for crime. Very remember that. There are lot of times videos are changed. So, there the how to detect the speech was given by the same person or it was altered. That is what is speech recognition is very important or you can use for image capture and many more. It can be used for many more things and we told you also that fundamental unit. Fundamental unit for recurrent network is what? Fundamental unit is the recurrent unit, which is nothing but a physically memory state or hidden state. This is nothing but a hidden state that will memorize the previous steps. So, that can be used for the feed to the next step. It can be, you know, can be either external or internal recurring. Like an external recurring network information can flow in both forward and backward directions, both from future and both from past. It can feed from both. That is what is the unique thing about external network. Internal network is always feed forward. That is, it will be feed from input step by step. So, it is basically a network which relies on this and they are incorporated in many popular things like Siri, voice search, Google translator, many things it can be used. But you know, first thing you should know, what is the difference between, what is recurrent state here? First, let us talk about. So, this is what I already explained, but let me tell you. So, one of the distinguished properties of recurrent network is this memory. It takes information from the prior inputs to influence the current input as well as the output. This is a neural network, right? One by one networks are present. So, it will take information. It will memorize the information from previous steps and then feed with the next step and also change the input and output for the next step, right? So, while traditional deep networks, deep neural networks actually assume input and outputs independent of each other. They do not consider them. Input is giving feeding to the next layer, then next to next layer and finally, gives the output. So, they are independent. They are not dependent on each other, but here, output of the current network depends on the prior elements or prior memorized states within the sequence. That is what happens in an English statement. It is a sentence. If it has to predict the next word, it has to remember what has happened in the previous step. So, future events can also be helpful sometimes in determining the output of a given sequence and that is why both kinds of networks can exist, external and internal. It is used to predict, you know, flood levels based on the past daily floods like during this high tide and low tides. Every day or in a sometime, high tides, low tides will happen in the oceans. So, it can actually take the data, time sequence data and predict this flood level during high tide and low tides. It can also be used to solve temporal problems such as, I told you, language translations, NLP, speech recognition, image capturing. It is already been incorporated in various things. So, now, first thing, let us talk about it. What is feed forward? I have already explained to you. Let me tell you again. In a multi-layer perception, it is a network formed by teaming up this perception. It is just like one input layer. So, you have input, input layer. You give feedback to the next layer. So, this is input. Then, this is first hidden layer, second hidden layer and then, you have output, but information is feed forward, right? Information is feed forward always. That is what that gives the output. So, this is the way information is passed and then, processed by these layers. That is what is called a feed forward neural network. This much probably you have understood, right? So, each layer can have large number of perception. That is what you see and there can be multiple layers, but fact is that input layer sends output to the next layer. Then, next layer sends output to the next to the next layer and it goes on. That happens and therefore, output does not depend on input. They are independent of each other. Sorry, output depends on input, but the values are not, they are independent of each other. So, in RNN, this is not done. This basically unlike DNN, this assumes input output independent each other here. Output of RNN depends on the pair elements within the sequence. That is the major difference. Remember this. Output of this network depends on the pair element within the sequence. So, that is why you have to memorize. There has to be memory step. That is what I told you. Main distinguished characteristic of recurrent network is this. They share parameters across each layer of network. In feed forward networks, there are different weights across the node. There are different weights. Here, you can have different weights for each of these inputs w1, w2, w3. That is what happens, but in recurrent networks, say the same weights, same weights parameters within the each layer of the network. That said, the weights are still adjusted by this back propagation things or you know gradient descent methods to facilitate processing, but within each layer weights are not changed. But in case of ANN or even CNN, within each layer, you can even change the weights. So, recurrent networks, neural networks leverage this back propagation through time algorithms to determine the gradients, which are used later on to determine or to basically optimize your data. So, that is what is the major difference. So, memory is the first step. This is the fundamental aspects of this. The fundamental repository unit is this. The recurrent unit or which is nothing but a recurrent neuron. This unit has an ability to detect, maintain the hidden states, allowing the network to capture the sequence and dependencies by remembering the previous input while processing. So, you can see that this is exactly, this is the sinks actually. It is using the previous test data here to predict the input. This is the weight and then it goes on. So, h t minus 1, h t, h t plus 1, there are three sequences and h t minus 1 gives the input to the h t. Same thing is for h t plus 1. So, you can clearly see that these are the time series inputs x t minus 1, x t and x t 1. These are your memory units l t minus 0, l t minus 1 l t, l t plus 1. They are remembered and these inputs are provided to the step and then it is recalculated to provide the next one. That is what we are talking about it. So, let me explain you in this detail manner. So, how this is done and in a next couple of slides, but before that I want to tell you one more important aspects. I think before I go ahead what is known as a vanishing gradient problem in DNN. What is that? You know this is a very challenge problem, very big challenge in deep neural networks. In fact, this problem occurs very frequently in all kinds of DNN. What is the problem? Derivatives of slopes of the activation functions can become progressively vanishing or small. Remember the slide I showed you. I can even go back to that slide and to show you that derivative functions I showed you. So, these are the different transfer activation function levels. Remember that in every step of neural network we are using this activation function to calculate the output. That is what is the key thing. So, this you know if the slopes or derivatives may be shown on the right side, the first two derivatives. If derivatives of slopes of these activation functions become progressively smaller as you move from one step to other steps, you can go from either forward direction or you can come from backward directions. What will happen? This will lead to problem in predicting the next output. You must be thinking why? Let me explain to you this. This phenomena in a 200 is called a vanishing gradient problems. That is what is called because the slopes are and this is very peculiar for mostly tan h functions and sigma y function. It is very typically widely used. The gradient propagates through the layers and to calculate the next output. Therefore, if they decrease significantly, the output and input tell you not much difference because the gradients are what is used to calculate the output. Remember, we sum it up weights multiplied by the input values and add a bias, then you have to activation functions. So, when you activate a function, you are using a slope to calculate the next step, the derivative. So, if the gradients of slopes are very small, change will be happening very small. So, input and output will not have much difference. That is the key thing happened. This means that at this, they leave the output layer, return to the input layer, gradient become very small. As a result, weight associated with the initial levels which are used to accommodate the small gradients have little or no effect on the next output. This problem is more predominant in sigmoid functions. Here, sigmoid functions or tan h functions, this is more predominantly widely seen because they have gradients values which varies between 0 and 1 or it can varies between minus 1 to 1. So, they can saturate between 0 and 1 or minus 1 to 1. In case of tan h, it is basically minus 1 to 1. In case of sigmoid, it is 0 to 1. These derivatives at various points can become 0. If you plot these functions, it function may look like this, but this derivative may become 0. So, once this derivative may become 0, if the function is derivative, they become 0. Then, what will happen? The gradients are very close to 0. This will not lead to any major change in the output. Input and output will remain same. Therefore, we are not going to get any extra advantage. That is why we just now talked about it. Variance in gradient problem is very widely seen. This is very subtle in case of RNN. This problem is very certain and important in case of RNN. One of the key indicator of how this can happen is that if you observe the model weights converging to 0 or stagnant, it does not change much. As the cycle goes on or the things goes on for training epochs, then during training epochs, this will also lead to this thing. During training event, loss functions will fail to decrease. Loss function is what is the difference between output and the actual values. If that fail to decrease during training process, then also you know that there is a problem of vanishing gradient. You can also have problem of this. You can detect by visualization of this. How do you take care of that? We can take care of using a thing called ReLU units. You remember that. That has a very important function, where x is 0, when say value of the function is 0, when x is less than 0. Value of the function is x, when x is more than 0 or equal to 0. That is what is called ReLU. This is equal to 0, when x is less than 0. This is equal to x, when x is more than equal to 0. That is what happens. This can be taken care of by using these functions or it can be taken care of by using what is known as a short term long memory networks or gated recurrent networks in case of recurrent neural networks. These two can be used for that. I hope I could make you understand what the problem is and how it can be solved. This is mostly used or mostly found in case of sigmoid function or tan h functions, where the slopes of these functions vary between 0 and 1 for sigmoid or minus 1 to 1. What will happen is that slope may not change drastically or may be very small during the subsequent steps in the neural network. This will lead to drastic effect on the output. Output will not change significantly. So, how the network will move forward? That is why one can solve using different kinds of functions, activation function like ReLU or you can use this kind of other kinds of RNN networks or variants like LSTM or GRU. Remember that I talked about it. You see this function or this function. They have problems in terms of this. So, you know RNN networks can be also four different types. One is 1 to 1, 1 to many, many to 1 and many to many. What is 1 to 1? This type of RNN network behaves same as a simple neural networks as it is known as a vanilla network. In this neural network, there is only one input and one output. You can see that you are giving only one input, single input and get a single output. This is a very simple network which is called vanilla network many times, but you have 1 to many. This type of RNN, there is one input, but many output possible. That is what happens in when you type something. You type one word, there will be many output possible. One of the most widely used example of the network is capturing, image capturing. When a given image, can you predict a sentence having multiple words? That is what it is. When you have a given image given to you, can you predict a sentence having multiple words? That means you have one input, but it can give different outputs. That is called one to many. You can also have many to one. In this type of networks, many inputs are fed to the network at several states of the network process. It will generate only one output. This type of network is widely used in a problem like sentimental analysis. If you have a stanza written by some of your friends, what kind of sentiment it reflects? Does it reflects passion? Does it reflects anger? Does it reflects some kind of a, again some community or something? That can be detected. You have many inputs, but it gives you only one output. Wherever we give multiple words as inputs and predict only sentiments of the sentence as output. So, one sentence or couple of sentences as multiple words, what kind of sentiment it provides? That can be detected by RNN. Then last one is what is known as a many to many. In this type of neural networks, there are multiple inputs and multiple output possible. That is a very difficult to understand initially. For example, this will be used in language translations. You have many inputs. You are giving a one or couple of sentences from Chinese or Russian or Spanish. You want it to convert it to English or your mother tongue. So, what you will do? You are giving many inputs as many words and convert into many words in your mother tongue or other languages. That is what is used. You can see this architecture. You have multiple inputs and they are connected. It provides multiple outputs. So, these are the different kinds of networks which are widely used in this. What do I say? In this kind of neural networks or RNN networks, I think one more thing I need to talk about it. So, let us talk about these variations of neural networks. I told you. So, first one is called LSTM. What is LSTM and how does it work? Let us see that. LSTM is basically what did I tell you. This is nothing but a long short term memory. Short term memory all of you know, but what is the long short term memory? So, this is a variation of RNN networks. Variation of RNN network which is called long short term memory LSTM. It is a network which was introduced by some gentleman like Sepp and Juergen. You know this network was to address the problem of long term dependencies. I think I have already told you our dependencies. When you want to predict different words in a sentence or sequence, the next word depends on the previous one. Suppose, if I write today I am, then you can have many words possible. Fine, good, bad, worse. How will it detect? So, that means it has to remember previous sentence and previous words. To do that, you should remember you should have a short term memory, but for long sequences. That is what is called LSTM, long short term memories. So, that means if I only say this word, these three words, it may not be able to detect my mood. How am I? Whether am I good, bad, fine or worse? My health is okay or not? It cannot, but if I speak many sentences, it should be able to understand from that and write. That is what is basically in a layman point of view, what is known as a long short term memory. This leads to long term dependencies that is previous state. Previous state, what is influencing? Previous state, what is influencing? The current prediction, we want to predict the current things. So, previous text which is going to influence the current predictions is not in the recent past. R and N may not be able to predict. So, that is why you have to use LSTM. So, one of the example is Alice is allergic to nuts. I write this sentence. So, next sentence is automatically write, she cannot eat peanut butter. Now, if it has a short term memory, it can only remember the last two words, nut or may be two. Then, she cannot write this. It has to remember also allergic word. Then, only it can write, she cannot eat peanut butter. So, that is the essence of long short term memory. That is why I want to give you these examples. So, context of nut allergy can help us to predict the next sentence. That Alice cannot eat peanut butter. So, therefore, LSTM has to do that. This is what LSTM has to do. To do that, LSTM has to have a specific architecture or structure. What is the specific architecture has? It has cells. Remember this. LSTM has cells. What these cells will do? These cells are hidden layers in neural network, which can have three important gates. They are nothing but hidden layers. These hidden layers have three parts. Remember one is called input gate, three gates, input gate and then, output gate and a forget gate. These three things will be there in a hidden layer. Hidden layer is built in LSTM cells. So, these gates control the flow of information, which is needed to predict the output of the network. For example, the pronoun such as she was repeated multiple times in the sentence for Alice. So, you might include them and exclude them from the cell state. That is what is done in case of LSTM. Alice is allergic to nuts. She cannot eat peanut butter. So, you have a word called she. So, the gender pronoun is she and this is used in the last sentence when I talk about Alice. So, that LSTM cell can actually remember. So, input gate will take the input, forget gate will not remember whatever is not required. Only store those those are important to generate the next state or next statements. This is how actually the LSTM can solve the problem of vanishing slopes or vanishing derivatives. This problem can be solved. This is one of the major ways of do that LSTM. Another network is known as GRU. GRU is another network. So, in a variation of RNN, remember this all are widely used in case of natural angle processing. It is similar to LSTM GRU. Let me tell you what is GRU. This is obviously for input and this is gated recurrent unit gated recurrent unit. So, what is that? It is basically similar to LSTM as it works to address the short term memory problem. Instead of using cell state, they do not use cell state which can regular the interaction uses hidden states. This uses hidden states. All of them use hidden memory state. It is not only GRU or LSTM. Hidden state instead of gates, LSTM using gates like input gate, output gate, forget gate instead of gates, this use hidden state. So, but they do the functions like this gates only and hidden states can store they have the long term memory, long short term memory possible. That is what actually it is widely used. These two variations, one is LSTM and GRU. They are widely used techniques in case of RNN. So, with this let me stop and you know next lecture may be we will discuss something new or something which we will discuss either from this from the deep neural networks or we will move to the next set. As I told you as a part of deep neural networks, we have two aspects to be taught in this course. One is a normal discussion on the deep neural network, other one is advanced neural networks. So, some of the things which I showed you in the last part of the lecture. So, some of the things which we talked about. So, we principally finished the supervised learning thing of deep neural network ML, NLP, CNN, RNN. Here, we talk about LSTM and GRU. Bidirectional is basically the text from future and the past both together. That is what it is not much different from LSTM. So, we need to talk about unsupervised learnings. So, depending on the time frame I have, I have to check it either I will discuss in the deep advanced deep neural network sections or we will discuss something here in the next lecture also. So, let me stop here. We will come back to the lecture number 28 in next one. Thank you.