Lecture 27 : Part 01

Students, this is lecture number 27 and we are going to discuss about deep learning methods. Deep learning is something we have been discussing for the last 2-3 lectures and we are discussing about different algorithms. Yesterday, at the last lecture, we discussed about the CNN or Convolution Neural Network. Today, we are going to discuss about RNN, which is nothing but Recurrent Neural Network. Subsequently, we will talk about some more algorithms as we move along. So, let us talk about whatever we learned and we are stressing about deep learning because deep learning is something new to many of you. Machine learning, you may be aware of because you are probably have learnt to use machine learning in different types of courses, but deep learning is not so widely used in material science engineering. Remember, the basic thing about deep learning is this Intricate Neural Network and this relies on the neuron, how it works in our human brain. So, therefore, since it is based on neural network, it enables computers to automatically discover patterns, which is distinctly different from machine learning. In machine learning, sometimes you have to make the computer learn how to detect the underlying patterns and then only they can predict. Here, in case of deep learning because of utilization of neural networks, it can automatically discover patterns and that is why it allows us to make decisions from vast amount of unstructured data means, it need not be structured at all, it can be text, it can be images, it can be even videos. So, from the unstructured data, these methods can allow you to make decisions and that is why it is used in different kinds of domains. Most importantly, computer visions, NLP that is Natural Language Processing or even health care. So, since it is based on neural network, the first thing it will come to your mind is Artificial Neural Network or ANN and ANN is basically Parallel Processing System in which the interconnected nodes which are called neurons, they work together to process and learn from the input data. Now, the question is as the data can be of different types, how to handle data? In today's lecture, we are going to discuss about RNN which can handle sequential data, time dependent sequential data and you know sequential data is data will be in order like you have a text stream coming or text stream means, when you type something on your social media or on your different kinds of platforms, you could be seen that if I type I, automatically next word comes in the picture, even whole sentence will be built on, it is you who can decide on whether it is correct or not. So, that is called text streams or audio fragments also, sometimes you can actually use audio fragments to create a new audio, video clips which are very famous nowadays social media like reels or I talked about it, time series data, that is what we are going to discuss today in RNN and later part of the lecture. But you know most of the images as we discussed in the last class, they are actually digital images nowadays and if you go back to my lecture, in the lecture number 26, digital image can be represented in terms of matrix in a scale of 0 to 55, which is nothing but a rectangular array of numbers and they arrange in a sequence like rows and columns. So, therefore, there are four important aspects of these kinds of digital images, what is called matrix which is this one n by n, you can see that or it can have also element like pixels, picture elements, voxel like video elements or it can also have bit depths. These are the essential or fundamental characteristics of the 2D images data. Now, finally, you can also have tabular data which all of us use, like if you want to put data mechanical property data of any material, you can always put in terms of a tabular data like stress versus strain, stress versus strain rate, strain versus strain rate whatever that is a tabular data. So, this is, these are all different kinds of data deep handling can handle and that is why there are different kinds of techniques available. The techniques which are based on supervised learning are basically MLP, which is nothing but a multi-layer perceptron, CNN convolution neural network, which basically processes images, RNN which processes time dependent sequential data and RNN will today see it has three different parts LSTM, bidirectional LSTM and GRU. And then we will also talk about, it may not be today's lecture, we will talk about some of these things unsupervised learning like AE, which is nothing but you know auto encoders or variants or SOM, which is nothing but self-organized maps, DBN and then we can also talk about with time permits, I will talk about hybrid things like CNN plus LSTM, GNN, GN versus CNN auto encoder versus SVM or many other things. So, this is something actually we are doing it here, this part of this lecture of this portion. So, as I said in our course, there is also a portion called advanced deep learning, some of the techniques we will discuss there also, especially SOM, REM, DBN, these techniques we will discuss there. And we will automatically understand how we can combine these two and do a fantastic machine learning or deep learning work in material science engineering. So, convolution neural network, which is called a ConvNet, which is widely used for digital imaging processing, we have already discussed about it, today we will talk a little bit about that also and we will start discussing about RNN, Recurrent Neural Network and also later on we will talk about AE, SOM, DBN or may be DEEP, DTL also. So, as you have learnt the first technique in deep learning is a multi-layer perceptions, they are nothing but you know artificial neural network, which are connected to each other in a multiple layers, so that it can imitate human learning capabilities. Because human learning capability depends on the complex architecture of the neurons in our brains. So, you have to imitate that, we create multi-layer perceptrons, perceptrons is nothing but neuron, it is nothing but a binary type of algorithms as I told you and it is basically a you know based on human brain construction or neuron. So therefore, it can have basically either minus 0 or 1, the two binary classifications however, when you combine with many other perceptrons it forms an ANN. So, whenever you combine that it forms basically ANN, so therefore, neural network in principle effectively can answer any question, how complex it is by enough training data and computational power. That is what is the dream, it should be able to solve everything and that is why deep learning methods are coming into big way in recent literature and it is thought that these techniques can learn hidden patterns and allow us to predict things in a very easily. So, what is the multi-layer things? Multi-layer is nothing but a different layers, it is a network formed by teaming up perceptrons with additional ones stacked in several layers, you can see that actually, these are the layers we have put it, input layer, output layer and some hidden layers. Each perceptron in the first layer is going to takes input from the whatever you are feeding and then send output to the next layer and it goes on, second layer, third layer, fourth layer. So, each layer as you see here, input also has large number of perceptrons and it depends on you, how many perceptrons you can put it and they can be multiple perceptrons, so that they can quickly become a complex systems. So, this is a three layer system which is nothing but a shallow network, if it is a more than a large number of layers, more than three layers, four layers, then it is called a deep neural network systems, four or more layers actually, if it is so, then it is called a deep neural network, three, two, they are actually called a shallow neural networks. So, and you know that how the neural network works, MLP works, basically nothing but your inputs which are weighted with some weights and add some bias, sum it over and put some activation functions. We discussed about the activation functions in the last lecture also and then you do this process again and again, because they are parallel networks, one neuron is attached to another neuron and it goes on. And many cases actually, if this forward feed forward, we call a feed forward thing, this is nothing but a feed forward, please remember this word, this is a feed forward network that means, you are feeding information from one layer to the next layer and it goes on in the right direction, many cases we use a back propagation network to optimize many of these parameters. And that works the principle that error between the output and the actual is what is used as a reason for back propagation, sometime people call it a loss, loss is nothing but a difference between output of the network minus actual values, this is the loss or this is the error. Once this error is large, you need to do a back propagation that means, you have to feed back same thing into the layers again, so that you can back calculate weights or it can basically optimize the weights, you can also optimize your activation functions if it is required, but that is normally not done, what is done normally is to optimize the weights and the bias. So, that the loss can be minimized that is called a back propagation network and back propagation is widely used in neural network, it is widely used, it is nothing but a back pass in football, so this is how it works. Now, I told you there are many activation functions, here I am showing you some of the functions, most important ones are what, tan h, you can see this is tan h and you can also see there is a function called r e l u, this is a very good function f is equal to 0, when x is less than 0, that negative and is equal to x, when x is more than 0, that is a positive. You can also have other kind of things like E l b, sort plus, many other things, the first one is a straight line, which is a identity you can see there, but look at the derivatives, first were derivatives, for the f x is equal to x, derivative is constant one, but for this kind of functions like a logistic functions, which has called sigmoid many cases, this is also called sigmoid, the f x is very different. Similarly, for tan x, tan hyperbolic x, this is the function, so this you need to remember, because depends on this first derivative, your back propagation network efficacy will change, if this derivatives are very small, infinitesimally small, we can land up a problem called vanishing slopes problem or vanishing derivative problems, that we will discuss even today also in the part of our lecture. So we have been this, we talked about back propagation, so let me tell you the few points, back propagation algorithm allows to perform backward pass, and this is used to mainly tune the weights of the inputs, and it is widely used to train neural networks, so if you want to give a back pass from there to here, this will allow you to tune the weights of the network, and it is a iterative process, remember that, it is not a single step process, it goes on iteratively, and it is used to minimize the loss or the difference between the known correct predictions, and actual prediction of the model, or rather predicted value for the model and the actual values, actual values can come from experiments, come from some theoretical calculation. With each backward pass, weights will move forward towards an optimum value, that minimize the loss function results in more accurate predictions, when the neural network is initialized, weights are set to individual elements, there you can set any weights actually, but basically we set the weight based on our perception, which factor is going to play more important role in predictions, and that may not be case for actual prediction model. So, once these are loaded, they pass through a network of neurons, and network provides an output for each one given the initial weights, and it is the back propagation backward pass that helps you to adjust the weights of the neurons, so the results become closer to the actual values, that is what is the reason we use back propagation, back propagation is used for all kinds of DNN or ANN networks. Now, let me little bit talk about spend some time about CNN, because this is the important methods for material science, this is a network which uses architecture for computer vision, and computer is a field which enables you to understand, interpret images or visual data, that is what it is, and this is an extended version of the CNN, which can be used to extract even the features from this matrix of the images, matrix data. So, this is something which can be explained simply by this way, that this is an image of myself, you can see that, now computer does not read the image in the same way, computer reads the image in this way, in terms of pixel, pixel elements, and depending on the intensity of the picture elements, you can see that I have divided this whole thing into lot of pixels, and this pixels can be made smaller or bigger depending on how a resolution you are talking about it. So, that is why you have different cameras of 12 megapixel, 16 megapixel, 8 megapixel, depending on that you can actually define as pixel size, now once you divide the pixel size depending on the picture, and the picture elements, intensity of this thing can be defined by number. So, therefore, it starts with 0, 0 correspond to a completely black, like if you look at my hair, they are black, so 0 will be here, you can see it is minus 6, a plus 6, it is not completely black, but it is somewhere here, it is showing a black, I do not know why, may be there is some issue, but anyway it does not matter, so 0 is basically correspond black points, but may be there are 0s here, I see 6 here, I see 6 here, but I do not see 0 here, but I saw the computer reads, but you can see 0 here, you can see 6 here also, 1 here, 0, 0 here also, so then the white areas are very high numbers like 237, 239, 240, 233, where they are close to high density. The whole thing can be put in a matrix, can you see that, this is nothing but matrix like this, n cross n matrix, so the question is how can I do just classification on a very large image, that is very difficult, when you have very large image to run a machine learning program on this is very difficult, instead we can exploit image spatial structure we can learn patches and then build on, that is a very unique feature of CNN, we can learn patches and then build on the whole concepts. So, I will not give talk about this, but let me talk about convolution, let us talk about this one, let us talk a very simple cuboid, you can see that, which has a definite length and width, the height is defined by this RGBs, red, green, blue, this gives you numbers, RGB scale starts from 0 to 255. So, it is consisting of what, instead of operating on the whole cuboid, we can say small patch or what is known as a learnable filters or kernels, it has small width and heights and same depths as the input volume, same depths, this is a depth the input volume, this is a depth I am talking about it, during forward pass what we do, we slide each filter or each of this in a patch, this is the patch actually, each pass across the whole initial input volume step by step and that is called stride. We also work, when you work we also take stride, this is exactly same thing and compute the dot product between the kernel widths and the patch from the input volume, we will talk about it little later. So, we simply slide over this, this small patch is slide over across the whole volume step by step and when you, so this patch will contain information about this image, the same matrix and that we took a dot product between the kernel widths whatever kernel or filters you have put it and the input on the patch. So, kernel means a filter, we will talk about that, as you slide over slide over filters we get 2 D outputs of each filter and then you can stack them together and put you know whole big things together and network will learn from the filters. Let us do that, this is exactly shown here, this is a small patch in a 2 D data. So, you can see that this patch has 4 by 4 matrix and this is get slided over. So, one can actually slide over from 4 4 cross 4 to next 4 cross 4 or actually one can slide over to like that also, like this you can overlap also, that also possible, this is another way of sliding over overlap. Overlap is much better because that gives you you know an understanding or idea how this you know overlap can help you to build the images. So, we can actually put weights on even these patches depending on our need, why do you put weights? Because we may have a image which is more complex and we can think about it, different puts on a different colors contrast. So, depending on our need we can assign weights to that and do this ANN network. So, here you see this is a flash like you know this is what I shown you filter size of 4 cross 4 and apply the same filter to 4 cross 4 patches in the input, then shift by 2 pixel elements to the next patch that is what is done here, this is what I was discussing in last class. So, in a nutshell if I have to put things together we apply a set of weights or filters or kernels to extract this local features and you can use even multiple filters to extract different features. Features means different things in a picture you can have different we will talk about that and then you can basically share the parameters of each other, what how you know these are called high level feature directions. How do you what are the features you see here? You see a nose, you see hair, mouth, spectacles right you can see even many other things you can see here also. So, automatically the CNN will detect features easily without doing without help from you or some kind of a learner it will automatically detect that. For a car you can have wheels, headlights, windows and also the rear view mirrors right. So, this all it can detect easily, house you have doors and windows it can easily detects. So, identification of the key features in image categories even can be done to the very high level of feature detections that is what I showed you right here. So, multiple features filters will extract multiple features. So, now what is convolution? That is what we are talking about that melting multiplication right. Remember that we talked during the forward pass we slide each filter across the whole input volume step by step and compute this dot product right. What is that dot product? Dot product is this, this is a filter, this is basically a kernel and this is what your patch that is the intensity pattern of the patch right. And what is the kernel? You simply multiply 1 into 1, 1 0 into 1, 1 into 1 then 0 into 0, 1 into 1, 0 into 1, 1 into 0, 0 into 0, 1 into 1 together is 4. So, then you bring the next patch, you can also have the next patch overlapping and then do this operation that is what is called multiplication. And this whole thing is a convolution, this is a mathematical operation allows to merging two sets of information. One is a filter, other one is a patch that is what is done. You might be thinking how does it help? It helps very drastically because you are using a patch instead of using the whole image if you say patch you are basically reducing the computational time a lot. And depending on the filter you can change the images extensively. These are the different filters, identity filters has only one here you see everything else is 0 in the matrix. You can also sharpen filters, you can also have box blur, Gaussian blur, many other filters available. If you go to any you know laboratory like matrix laboratories, MATLAB there are so many kernels of filters available. Depending on your need you can image process the images this way. So, therefore, you know in a real architecture of CNN you can input layers, convolution layers, pooling layers, fully connected layers or dense layer. So, I discussed about you convolution layer. What is pooling layer? Because in convolution layer you will generate lot of data. So, there extract features from the input by sliding this small filter. So, kernel over the whole image or video and then compute the dot product which I just now shown you. But pooling layers will down sample the output because this convolution can create very large amount of data and many times it will be not easy to handle and you know deep learning requires lot of data. So, what pooling layers will do? It will down sample the output of the convolution layer to reduce the dimensionality of the data and that makes the computational very efficient very very efficient computational it can do. So, that is what is there here. So, this will be more computational efficient that is the idea. This is a very unique idea and it works very nicely. Then you have a fully connected layer or dense layer you can put again things information together. That is what is shown in input. Then you do this convolution one where feature maps is possible. Then you do the pooling. Pooling means down sampling. So, feature maps is reduced you can see that. Then again you can do convolution two feature maps. These steps can repeat actually and finally, you do a network with a dense map. So, that is basically what is done in case of convolution neural network. So, let me speak little bit about recurrent methods in this part of lecture and then we will carry forward. So, recurrent neural network is little bit different and you know what is difference in traditional neural network all inputs and outputs are independent of each other. You could have seen that. That is what is this in traditional ANN or even CNN all inputs and outputs are not dependent on each other. RNN is a type of neural network where outputs from various steps is fed as a input to the current step or even future steps. Like you know very simple example if you want to you know require to predict next word of a sentence. Let me also give you a very good thing which I recently got. Let us talk about if you say I know a very simple idiom and I can go and talk to the board. If you say simple idiom what is that filling under the weather. This is the idiom and you know which is commonly used when someone is ill. You know we always say that because of change of weather somebody has fallen sick. That is what is called filling under the weather. Now, this in order to order for the idiom to make a sense it needs to be expressed in a specific order filling under the weather. That is what it should be done. As a result this recurrent network needs to account for the position of each word in the idiom and there this use this information to put you know select the next word in the sequence. What I mean to say is that the RNN must know there is a word called weather later part of sentence. So, it will it has to put under. So, if I what I mean to say is that if I type to my friend that I am not feeling well instead of that if I write filling under the weather I simply write filling. You will see you know social media or you know this thing they will bring many words here correct, but the moment you put the weather automatically another word will come into picture. So, that means what RNN remembers recurrent network remembers which word has to be placed in which sequences that is what is done. So, it is very widely used to predict next word in a sentence or the previous words you know for that if you want to predict next word you need to have remember the previous words right. That is why this is a sequential event time sequential event actually and how does RNN solve this problem this problem is solved simply by using a hidden layer or rather let me tell you also much more detail and you know this is a type of artificial network that use sequential data or time series data sequential or time series data. And these are characterized by this hidden layer or memory because as you see here in order to predict the next word the network must remember the previous word or they may have to remember even a future word that we will discuss how this network can change. And so fundamental building blocks in RNN is this recurrent layer or recurrent unit which is nothing but a hidden layer or it is a nothing but a hidden state or memory that is always updated in every step based on the current input and the previous hidden memory. So, this is widely used in NLP natural language processing widely used. So, main and most important feature is this hidden state which remembers some information about a sequence I told you right filling under the weather. So, the moment weather is there it will automatically put the word under that is because it has to remember the information in a sequence and this state is always called as a memory state since it has to remember the previous input in the network. So, it uses the same parameter for each input as it performs the same task and all the inputs in the hidden layers to put the output that means what weights remain same you do not change the weights in the neural network at all and this reduces the complexity or the problems. So, fundamental block is basically the recurrent unit that is why it is called a recurrent neural network and it will basically it allows RNN to retain the information from the previous step enables them to capture the temporal dependence and they produce the predictive results in a sequence. So, it remove it remembers this previous step. So, that it can capture the temporal evolution of the next step and also predict results in a sequential manner this three things important first is it will has to retain the information it has to retain the information what is the information previous step information. So, recurrent is this first is has to remember or retain information from previous steps from previous steps then what it will do enable this information to capture the temporal dependencies time dependent information temporal dependencies of the temporal dependencies of the previous dependencies only a time dependent things and third thing it has to put in a sequence it has to put in a sequence what is that produce predictive results in a sequential manner this three things recurrent action will do. So, let me stop here we will come back to the next part of the lecture and discuss more detail.