Lecture 28 : Part 01 - Deep Learning Methods-VII

Students, welcome to lecture number twenty eight. So, we have been discussing about the deep neural network and we will continue doing that today. However, we will talk about unsupervised learning of deep neural network. So, first let us see what so far we have done. So, DNN, a deep neural network has three different categories. First one is supervised or discriminative, second one is unsupervised or generative, you can see here and then hybrid type. So, first we discuss supervised one like MLP, CNN, RNN. MLP stands for multi-layer perceptron, CNN is convolution neural network, RNN is regenerative neural network, recurrent neural network, sorry not regenerative, recurrent neural networks. And today we are going to talk about unsupervised one that is mostly we are going to talk about JAN, AE and SOM. So, then we will wind up this particular portion and come back a little later part about discussing this various advanced deep neural network models after few lectures. So, as you know DNN has various types of models available and all of them may not be required for material science modeling and that is why you are not discussing all of them. But you know convolution neural network was very important for image processing that we have seen that. Recurrent neural network, RNN and variants are good for sequential data or time series data like you have a video or audio time series, how to use this. Then you also have generative neural network, this is V, generative adversarial neural network or GNN and this we are going to discuss today autoencoder as well as self-organized maps which is SOM. Some of these techniques which will be widely used in the material science modeling. So, first thing we discussed about this multi-layer perception, it brings back the discussion again and again because traditional neural network model started with artificial neural network which is nothing but forward directional neural network. And MLP is one of the famous ones and it is formed by basically teaming many perceptions with additional ones stacked in different layers. And these allow us to solve complex problems, very, very complex problem because this mimics the human brain structure. So, perceptron is basically a simple thing like a neuron, but it can take both values of 0 and 1 basically binary digits that is what is the difference between perceptron and neuron. And each perceptron in the first layer will take the inputs and as you know neural network or multi-layer perceptron is a parallel processing unit. So, therefore, this input layer will send output to the next layer which is second layer or hidden layer and finally, it will go on and then output will be received in the final layer which is at the right end of the network. And this is a three layer network is shown here it has it is basically known as a non-deep shallow network because it is only three layers. The moment there are four and more layers we call it as a deep neural network and that is why actually deep I think came into picture deep means quite large number of networks. So, MLP is basically very simple it can have input layers with some bias then you have some weights and then we sum it over the weighted inputs as well as bias. Then at the next step we apply the activation functions and finally, you get an output. Remember this particular thing is done in every step this is just like a parallel processing again you will also have same thing done that is what is basically the typical feature of any neural network model. So, you are going to do it again and again finally, we arrive at a best possible answer which will be matching with your expected values or you know experimental values whatever you are thinking. So, that error can be minimized and I talked about also some activation functions. So, I am not going to discuss about it there are many activation function use most important was sigmoid, tanh or relu or even some other variation of relu networks. So, an every neural network whether it is a deep neural network or normal ANN type here we have to use back propagation algorithms because this allows to perform the backward passes to tune the weights of the network. As you have seen here we are using different weights here although I have written everything w, but it is not w, w1, w2, w3, w4, w5 something like that wn as well as your bias and this bias is basically to take care of things which are not tunable or we add this bias to have a certain kind of variation on the values of the inputs. So, finally, backward passes are done to optimize or fine tune this weights a w that is what is the idea of this. So, this is an very important algorithm commonly used to train all kinds of neural networks and it is a iterative method as you know it is a iterative backward pass and it will always attempt to minimize the loss function or the difference between the known correct predictions and the actual model predictions. So, that is the idea now how it is done exactly that will be taught by the TA's when you will do the live sessions you know how we can do in python programming to take care of this backward propagation network problems that will be shown to you, but I would not want to you know tell you that please do attend this live sessions in every live session. So, that you can get an understanding how this modeling will be performed if you want to learn to model this model you know using a I M L for material science problems then you need to know how to do it and that is why actually real challenges are there I am discussing only the algorithms and the problem associated with it how to take care of those. So, now, then we talked about little bit about recurrent neural networks that is what the major discussion happened in the last lecture if you look at any traditional neural network and the inputs and outputs are independent of each other they are not connected each other and also neural networks are actually kind of forward type from it goes from front end to the back end it is never comes back right although there is a back propagation algorithm which takes care of the weights of the input, but that is a different thing and normally the way neural network function is that it in a forward direction. So, an inputs and outputs are independent of each other, but RNN is a typical network where output from the previous step is fed as input to the current step or even future steps also. So, in cases where it is required to predict the next word of a sentence how do you do it we need to know the previous words hence there is a need to remember the previous words otherwise you cannot predict it right you must have done when you type email or you type words in a social media networks what you do you type something then the next word automatically suggested by the program itself. So, this is a typical you know behavior of recurrent neural network and it is if you carefully see it means that we take the information from the prior inputs to influence the output and the output of the current output as well as the final output. So, we take the information from the input as usual, but it is used to influence the second output as well as the final output and that is why this is used for time series data or sequential data that we discussed in the earlier lecture. So, fundamental building block that is why in RNN is was as a hidden layer or a memory layer right that is called a recurrent unit and that is why it is called a recurrent neural network. So, this hidden layer is what is called a recurrent unit or memory layer it maintains a hidden state of memory that is then updated every time or every state based on the current input. So, it is basically an process which will update the recurrent unit every time a new input is given. So, it allows a RNN to retain the information from the previous time steps and enable them to capture the temporal evolution or temporal dependencies and also produce the predict results in a sequential data that is what it is. So, most important feature is this hidden state which you have to understand it uses the same parameters for every input as it perform the same task on all inputs or the hidden layers. So, this reduce a lot of complexity you know you can you have seen in case of neural network you can use different weights, but here we do not do that we use the same parameters weight parameters because it perform the same task on the all inputs or the hidden layers and that is reduces the complexity of the parameters and which is distinctly different from the early other neural networks. So, the why it is required because as I told you most of the neural networks are feed forward neural networks whether it is a talking about MLP or you are talking about CNN the feed forward neural network that means what is basically goes in the forward directions. Although it is a stack up of different layers and this different layers are having different purposes, but the information goes from forward direction. So, that is why it is kind of a feed forward network RNN is not that it implies a sequential data and that is what is the important thing and then output depends on the previous steps. Maybe I can show you that is what it is you see here output depends on the previous steps it is as a memory which it has to remember every time it does the operation that is why output depends on the previous steps. A recurrent learns from the training input however distinguished by the memory and I told you right this recurrent unit or a memory unit or hidden whatever is it allows them to impact the current input and the output using this information available in the memories. So therefore, it is very popular architecture lot of times it is used for long short time memory LSTM or bidirectional LSTM or GRU which you have discussed where the problem is vanishing gradient is very significant and I told you what is our vanishing gradient right I do not want to go back to it anyway. So, there are three important RNNs are widely used one is called long short term memory LSTM, bidirectional LSTM, bi LSTM and graded recurrent unit that is what is the three different types of things which you use normally. Because fundamental limit is this recurrent unit and that is why this unit is as a unit ability to memorize things or capture the sequential dependence of the previous inputs while processing that makes it different. So, now the network can be of different types based on the input and output like one to one, one to one is very simple one it is called manual neural network this is very simple in which you have only one input and one output that is rarely happens mostly you have multiple inputs or multiple outputs. The second type is called one to many in this type there is one input and many output associated with it the most important example is the image captioning where given the image we have to predict a sentence having multiple words. So, I give one input as image and I ask this the software or whatever to provide the caption of the image and that means a sentence and that is means it is a single input to many output. You can also have many inputs to one output in this type of network there are many inputs free to the network a several states of the network and that will generate one output this is typically used in sentimental analysis what is that you ask the software to read or this network to read one chapter of a book and ask them ask it what kind of sentiment reflects some kind of a happiness or whatever then it is can tell is the informations are many inputs are many and output is one right that is what is another neural network. But remember when they reads through that this is widely used in NLP or natural language processing that is why it is has to do this kind of jobs then you can also have a last kind of network is called a many to many in this type of network there are multiple inputs and multiple outputs corresponding problem one example is language translations right very simple language translations in language translation we provide multiple words from some language as a input and allow the it to predict multiple words in the second languages output right that is what is done. So, you could have done that you could have translated from some foreign language to English. So, that is why you do many to many right so you have understood right one to one, one to many, many to one and many to many there are different examples also I have given it. So, now question is there are different types which will not discuss but now we will talk about little bit about today's lecture unsupervised learning and what are these first of all let us talk about what is unsupervised learning well it is a basically a very important area of machine learning or deep learning and that involves hidden structures or patterns without any level samples. I hope you remember that there are two types of learning unsupervised and supervised there is another type of learning also we discussed about it which is reinforcement learning but thus depends on agent let us not bring it here. So, supervised learning means you have a input as well as corresponding output or data levels available right and in the unsupervised learning you have only input you do not have any output or data level available correspond to each input. So, under such a conditions how do you find out hidden patterns or hidden structures hidden features okay that is what is the job of unsupervised learning. Now we talked about in machine learning also like clustering is one such unsupervised learning technique in machine learning but DNN is different. So, in this kind of DL techniques or deep learning techniques it is typically used to characterize what higher order correlation properties higher order means the correlations which are not very simple in machine learning we find correlations are very simple like clustering in clustering what we are doing we are trying to find out how many points you can bring it together right you remember we are how we have done we have started using a mean value and started calculating distance of this mean value from each of the nearby points and if all of them fall same distances then you call it is a cluster but it depends on that we can have different types of clusters we have discussed all these things please go back to my lectures. But here we are not finding out any simple correlation properties we want to find out or characterize higher order correlation functions or features as well as joint statistical distributions of a visible data this is important here in DL you can also find out statistical distribution also how this data are statistically distributed and how they associated classes are there it can be easily done using unsupervised learning techniques these are the two important aspects one is a characterize the higher order correlation functions or properties our second one is a joint distribution statistical distribution these are actually widely used in the DNN. So as is investigates underlying relationship or underlying patterns a correlation in the data that is why it is an effective tool for many kinds of problems like anomaly identifications dimensional reductions or clustering again so we know a dimensional reduction right that is called a PCA and we will talk about a Pearson correlation analysis and that is something which relies on the DNN technique unsupervised learning of DNN. So when you have a large number of data that is why the DNN works actually you could remember the small data DNN is not good so large number of sub data sufficient data the interpretation of this kind of things it provides very insightful information which can enhance the decision making as well as facilitate the comprehensive integrate data patterns that is very important what kind of data patterns are providing like you are talking about let us say sell up any item by a company let us talk about all these biggies like Amazon or Alibaba when they are selling some products they will like to find out integrate patterns in the selling pattern that is very important because that is how it can which can make you understand it can make you understand that what kind of intercorrelation exist among the data and this is can be only done using unsupervised learning and deep learning techniques here remember this network is not under any guidance of features we have not placed them to work under any guidance of features rather it is going to find out higher correlations from the or the statistical distribution from the visible data. So in a nutshell this is what actually the unsupervised learning does in DNN but you know what is this thing basically he has to use algorithms to find patterns in unlabeled data that is what is the key aspects in unsupervised learning in DNN you have a lot of data you want to find some pattern or features and so therefore that is the key thing to develop this kind of algorithms so what is the key idea so key ideas during learning you have to have precise supervising supervisory information or target level is not required at all because that is what is required because of supervised machine learning techniques or even some of the DNN also but here you do not need any precise supervised information as a result of this we can find correlation of input data or features even we can have data representations also will tell you today also how data represented is so important in today's way because this has connection the informatics remember material informatics data science relies on how a present data so that is the key idea since it has no supervisory information supervisory things at all required we can actually find correlations of the input data or features or even you know representations very nicely using unsupervised DNN learning what are the various components of the unsupervised learning well there are four components let us discuss about that first one is coder and encoder okay what is coder encoder it is used to encode and decode the data it is normally used to that because that is what is it is encoder basically responsible for transforming the input data into the lower dimensional representations or on which neural networks can then used so what is the encoder does it basically takes the input data and transform into a lower dimensional representations that is very important because then it reduces your computational time or the power and then you apply the machine learning on this or deep learning on this by decoder what will do it takes the encoded representation and reconstruct the data from it and this is the architecture which is used in all unsupervised learning techniques once encoder one decoder in fact there is one answer by lightning techniques it is auto encoder which has this is automatically or rather we will discuss about this built-in second thing is called a latent space what a latent space is basically immediate represent of the data which are created by encoder has to be placed in some where that is what the space it is placed usually it contains feature which can capture the important aspects of the structure of the data so it is this one the abstract representation of the features that can kept informations are placed in a latent space which is hidden so encoder will transform this data into a lower dimensional data right and then you can store or keep it in some space that data is called latent space and this is important because of the architecture because it is it is what on which the deep learning technique will be applied so the once it is kept in latent space what you do is basically we do a apply training algorithm or a neural network model unsupervised neural network models use this specific training algorithm to get the parameters some of this common optimization algorithms are stochastic gradient descent or Adam stochastic gradient descent you can understand basically it is nothing but gradient problem in a time sequence manner that is a very common method which is used in training algorithms but you can also use Adam also they are used depending on the type of model and the loss functions so what are the loss functions this quantifies how well the model understand the data that is very funny right if I tell the sensors is quantifies how well the model understand data because it does not tell you how they are connected it is a common component for all machine learning or deep learning techniques is basically calculates the models output and then difference between model output in the actual measure data that is how it is called loss functions so the back propagation actually works on the loss functions you try to minimize the loss function that is what is the back propagation technique so these are the different components of the unsupervised learning methods which always widely used in the DNN. So now we will talk about few of these techniques first let us talk about what is auto decode encoder so as you understand it is an encoder we will come back to it auto encoder in a little moment time what are different types but first let us discuss what is auto encoder and what does it do auto encoder is basically a one of the most popular network very popular DNN and PNN auto encoder let us discuss the first is that and this is has a it is to it will always work with high dimensional data high dimensional data and then reduce dimensionality of this data dimensionality will be reduced why because it will allow us to it will allow us to handle the data or apply the machine learning model much easily that is what auto encoder does or sometimes people say it basically compresses the data compresses the input and generate a code compress the input generate a code and then this which is decoded substantially this is in subsequent steps to reconstruct so basically what it does it compresses the input data on which you can apply a model deep learning model and then again you generate a code that is what is the deep learning model this will allow this will then which is basically decoded once first you compress the data by encoder then apply a model then this data is again decoded to reconstruct the data that is what is the that was is done most generally in case of auto encoder so in a nutshell if I can say it is compresses and decompresses data that is what it is so this network is knows how to compress and decompress data so in a nutshell if I write it it is a neural network DNN which compresses and decompresses both the process first compresses and apply the model and then decompresses data this is widely used in image processing if you have a large image which is not easy to handle data it can widely use that that is what this this one do so they are all feed power networks as you understand these are all feed power network not like RNN. This is a feed forward network these are the important features of this auto encoders but what are the different kinds of feature is has it has three features remember any auto encoders or any variants it has three features first one is a encoder and second one is the encoder decoder and also you have a latent space right the three things it has encoder decoder and latent space so encoder will compress the data latent space that is what is done so when input data is compressed it will be compressed to a lower dimensional that is what is a job of the encoder one dimensional called latent space and then it will be decoded later on to reconstruct data the first part of neural network is to do with what is called the encoder and this encoder facilitates effective data in constructions the encoder actually compresses data and to facilitate the effective represent reconstruction of data so decoder is used the construction and reducing dimensionality and after the compressed data data will be stored here so reducing dimensionality learning features denoising data so therefore, important characteristics are what why do you do that why do you are doing this compressing data the characteristics are as follows one is reducing the dimensionality second one learning the hidden features this is also typically important and third thing is denoising the data we will talk about it in a moment or so so these are the important aspects of that but three important parts are encoder latent space and decoder so let us stop here this part of this lecture we will take up the next part of lecture and discuss in details.