Lecture 26 : Part 1

Students, welcome to lecture number twenty six and we have been discussing about deep learning. Today, we are going to learn few important techniques of deep learning, but let me tell you one important aspects about deep learning. Deep learning actually utilizes the concept of neural network. So, you must have a good idea about the genesis development and application of neural network and that is the key aspects, which you will see in all the different kinds of techniques we are going to discuss on deep learning. Another important aspect of deep learning is that we have divided deep learning lectures into two parts. First one is introduction deep learning and some of the simple techniques of deep learning. In the latter, I will discuss about advanced deep learning techniques. There again some of the basic concepts will be utilized to develop advanced techniques required for deep learning methods. So, with this introduction, let us move ahead and you know deep learning as such mimics the intricate neural networks of human brain. I already talked about it that this is the basic aspects or the basis of deep learning techniques because it uses the concept of neural networks of human brain. It enables you or enables the computers to autonomously discover the patterns. Unlike in machine learning, we have to do the supervision to make the computers, the machine learn the patterns. Here, you do not do that. It can autonomously discover the patterns and it can also make decisions from the vast amounts of data, unstructured data. You need not be having structured data. Remember, we have discussed about the data in the last lecture. So, you can go back and see. So, it can make decision based on the unstructured data, but it requires vast amount of data. Compared to machine learning, deep learning requires more amount of data. That is why it is used in various domains like computer vision, natural language processing or NLP, health care diagnostics or even autonomous driving. So, basically it is based on neural network framework architecture. As you have learned artificial neural network, it uses the layers of interconnected nodes, which are nothing but neurons. They can work in a parallel processing units to process and also learn from the input data and thereby it can make predictions. So, first important thing is that it is based on the neural network. It can autonomously learn or discover the patterns and it requires vast amount of data. Data can be unstructured data or structured data. So, these concepts are the main focus. Why deep learning is so important today? It is because of the industry 4.0 and that is what all of us are dreaming to achieve. We will have a completely technology driven automation in the industry, which will be smart intelligent systems. That means, all the machines will be run by data driven or technology driven autonomous aspects. So, that it can make a drastic change like industrial revolution happen in 1700, in the middle of 1700 or 18th century rather. Therefore, it can be used in business intelligence, smart city applications, cyber security and including material science engineering things. It has started in 2006. I told you and it started with one scientist. Let me tell you the name of the scientist. It basically started with Hinton in 2006 and then it became a very popular technique. It has grown dramatically in terms of performance, in terms of applications for various fields and because it provides excellent solutions for uncovering complex problems, complex architecture in higher dimensional data. That is why it is so useful, but it requires large amount of data. That is the key thing about deep learning. Small amount of data, you cannot use deep learning methods. You can basically use lot of machine learning methods, but deep learning is not useful. It will play a very important role in future for building this intelligent data driven systems, whatever you need today and even using data, which are available in our literature. Therefore, it can change the world as we see. Automation is going to happen more and more and that can do that. It can learn from data. It can learn. There is all kinds of data structure and unstructure. That is why deep learning techniques are always related to it, informatics and the data science. Let us talk about the data, which we have discussed in last class. You can have three types of data. First one is called sequential data. This is a kind of data, which with order is important. It is orderly data. It is basically in a set of sequence like text to extreme, some audio fragments, video clips, reels, time, series data. All of you watch reels in the various social medias. That is also very, that is also like an order set of sequence of data actually. This is what we normally use in everyday life. Then, you can also have a 2D data. It is called images, which we will discuss today about convoluted neural network or CNN. Digital image is made up of matrix. I will show you that how it is made of matrix. This is nothing but a rectangular array of numbers. It can be also symbols. They are arranged in a row and column in a 2D array of numbers. So, that matrix can be pixels like picture elements, video elements like voxels or even bit depths bit by bit. There are four essential characteristics of fundamental parameters for the 2D data. Matrix is like this. Then, you have pixels or picture elements or you have video elements or you can have bit depths, which are actually the four essential characteristics of any fundamental aspects of 2D data. You can also have tabular data in terms of a table like an excel file. You generate tabular data, which are widely used for calculations, day to day calculations also. So, 2D data is for digital images, which are more widely used for deep learning methods nowadays. As I discussed, deep learning is good for when data amount of data is large. It always perform much better, but when you have amount of data that is small, machine learning will do much better. Most of the machine learning algorithms, that is the key aspect. As you see here, the predictive capability of product performance of deep learning methods will increase almost exponentially as the amount of data increases. As I told you, deep learning there are different kinds of models. Most widely used one is ANN or artificial neural network. Then, you have also something called CNN or convolution neural network, which we will discuss today. At the end of the lecture, I will talk about something about recurrent neural network. May be in the next lecture, we will talk about autoencoders or some amount of deep belief networks, if possible. If not possible, we will talk about these aspects in advanced deep learning methods part of the lecture. So, what are the key dependence and properties? Obviously, it depends on large amount of data. You cannot do it without that. TL requires relatively large computational power, but that does not mean that you require super computer. You require reasonably good computational power, high performance computers, high performance processors or laptops can do quite a lot of deep learning methods. It does lot of feature engineering. That is why it decreases the time and effort required to construct the feature extraction. It can automatically extract the features. That is why it uses the TL, uses the ANN and CNN methods. Obviously, training in TL takes lot of time, much larger than time required for machine learning. Remember one thing. This is a black box actually and difficult to interpret how deep learning actually works. It works. It gives you results, but scientifically how does it do? It is not very, not much clear concepts so far. So, it does not matter for you. How does it work? It works and it gives you good predictions. That is what is important for us. You can use different kinds of network architecture like that. You can use shallow networks where you have only one hidden layer. You know that input and output must be there for any deep learning or rather artificial neural network methods. Other than that, you can have one hidden network or you can have multiple hidden networks. Depending on that, it can be a shallow network or a deep neural network. So, N can be anything. N can be even 100s also. It depends on your computational power or the ability to handle data. So, as you see here, these networks are all interconnected. That means, every input layer is connected with the hidden layers, with each other. That is what cross connections are made. Cross connections are made. That may not be always required, but normally it is done. So, what are the different deep learning techniques we used? Obviously, we use you know supervised techniques, unsupervised techniques and hybrid learning methods. Supervised techniques are basically NLP. MLP is what? You know that. This is basically multilayer perception method, which is based on ANN. Then, we can also have CNN convolution neural network and we can also have RNN recurrent neural network. RNN has different parts. LSTM, if I get time, I will discuss about it. Bifunctional LSTM, GRU. Then, in case of unsupervised, you can have A E, which is auto encoder. Then, you can have SOM. You can have GAN, RBM, DBM, DBM, deep neural network. Then, you can also have hybrid networks, mixture of two things like CNN plus RNN, GAN plus CNN, A E plus SBM or any others. These are all hybrid, mixture of between these two, unsupervised and supervised. So, they are actually very advanced techniques available for DL networks, much advanced than what one can actually think about it. For your purpose, we have discussed about MLP, but today again we will talk about little bit more about MLP. We will talk about CNN and we will start probably RNN and see how much you can understand this concept, because they are not very easy concept to learn at the beginning. It takes time to learn these concepts. There are various kinds of features of each of these concepts used. Then, later on, we will discuss about A E or DBM or SOM, whatever possible amount of time available with me. I can discuss each of these techniques. First, let us talk about ANN and carry forward what we have learnt, because the basic concepts of DL or deep learning is the neural network. You know neural network is based on how human brain works. Human makes this parallel processing system. You have to understand that and it contains large number of processing elements. That is another aspect. That is why you have to use many hidden layers in case of DL. That is the basic reason why you use many hidden layers. If you ask your friends, they will say I use 3 hidden layers, 5 hidden layers or may be 7 hidden layers, but one can use, go on use even 100s of hidden layers, because of this complexity of the problem requires this uses of large number of hidden layers. We have about 10 billion neurons and they are interconnected and they do parallel processing. What is the meaning of that? Even neural network, I will go to the next slide. In a neuron, which is typically the processing unit, the dendrites actually take the input. Input can be any data, variables, whatever it is and then it is processed by nucleus. After that, the data is sent by action to the synaptic knaps and to the next element. Next element will sit here, correct or it can also have a previous element like this. So, previous element or next element, previous element will give the input output to this element, which is shown on the screen and next one will receive input output from this. So, that is why it is called a parallel processing and this concept is used in all the neural network problems. Now, as we are talking about MLP, which is nothing but multi layer perception. This is a supervised deep learning technique. What is the MLP actually? As you know, human has this unique capability to recognize, identify patterns based on certain accessible information and we can do it with a very high precision or degree of accuracy. So, we have to mimic that. That is what the idea is of the MLP. Artificial neural networks are basically systems that we, basically system is, these are all computational systems that intend to imitate human learning capabilities. How do you do that? We use the same concept that neural network has or neurons has in our brain. The complex architecture that can resembles our nervous systems and you know the basic unit here, MLP is the perceptrons. Remember this word perceptron. This is not as a neuron. It is a new thing, but similar to neuron. It is simply binary classification algorithm. It can take minus 1 or 0 or 1 and this perceptron was model using the concept known for human brain or neuron. So, if you it is kind of takes 0 or 1, it can be combined with many other perceptron. That is why it forms the neural network, artificial neural networks. In principle, does any neural network can in principle answer any questions using enough training data and computational power because it depends on you, how complex you can make the neural network or this MLP network. So, neuron as you know receives input from other neurons, same thing here also. Inputs are then combined and once inputs exceed a critical level, then neuron discharge a pulse. That pulse actually travels from the body down the action to the next neuron. Same thing happens when the data process by the one perceptron. It will be sent to the next perceptron and this is how this whole thing actually works. It is shown here that you have a input and this inputs are multiplied by weights. Depending on the importance of a input, weight value will change and that is what you get here. You see here and then what you have is a summit over and then apply some activation functions. Then, it provides the output. That is same as like a biological neuron because you have dendrites which are receiving the information. Then this processing is done here. That is what the processing window is. So, they are first of all before processing, they are multiplied with certain weights. I have been telling you again and again based weights. It depends upon which parameter is more important than others. After that, you have to submit over from I0 to N and then we have to use activation functions. Now, you are thinking what are the activation functions to use. So, that is what is shown here. Sometimes we add bias. Bias is basically something which has no weights. You can see no weights. Bias is added in order to take care of the change in values of the input. Sometimes, inputs may not be having fixed values. Then, you need to change that values input values. That is why bias is used. Now, these weights are multiplied with inputs and bias is added. Then, it is summit over. Then, you do an activation function. We are talking about this activation function a lot. Let us talk about how this activation function are and what are these. I just wanted to jot it down and show you that. What are the different activation functions and what are these? Let me go back to the board and explain to you what are the different activation functions because these are what is called black box. The first one is linear. Remember, these are called activation or transfer functions, which you have to apply on the sum. So, sum is nothing but I0 to N wi xi plus the bias. On the sum, these transfer functions will be applied. So, first transfer function is what? First transfer function is a linear function. It is very simple. Remember, regression. We use a linear function for regression. So, linear. The busiest linear function is what? f x is equal to x and x can vary from minus infinity to plus infinity. So, how it will look like? If I have to plot, let us do a small one so that I can do every bit of this in one slide. f x is equal to x will vary like this. That is a linear function, which is simpler one, but even in material science, very rarely linear functions are used because all our problems are very complex. So, you cannot use linear functions, but nonetheless in some cases, you may like to use linear function. Next one is sigmoid. Sigmoid function is also used. What is this sigmoid function? Well, this I have already given you. f x is equal to 1 by 1 plus exponential minus E x. That is what it is. So, it is very simple function. Now, how does it look like? I draw it on the extreme end so that it remains same. So, it is very function looks different and you should know it. So, how it will look like? It looks like this. We can use different color. So, this is basically a sigmoid function. How does it look like? Sometimes, you will use pieces of black box. You may not know it, but for the sake of this understanding, this is required. Then, another one which you must have is called tan h. This function looks like this. 2 divided by 1 plus e to the power minus 2 x, not 2, x minus 1. That is how it looks like. It is much more complex than the sigmoid function. It will look like, I can draw this again on the same chart. That is better actually. It looks like little similarity is there with this. That is how the tan. So, this is tan h. The other one is sigmoid function. These two are actually widely used in material science. Let me tell you because they have very complex nature. They can provide, they can mimic all different kinds of situations which you see in the real experimentations. Then, another one which is used is Re Lu. Name is very funny. Meaning is rectified linear unit. So, this is rectified linear unit. So, Re is basically Re is rectified L unit. That is what it is. This is a very funny function. This function is very funny. It says f x is equal to 0 when x less than 0. f x is equal to x when x greater than equal to 0. That is what it is. This function is correct. So, you see that f x is a linear function when x is more than 0. But, f x is 0 when x is negative. So, how do you plot it? It is very easy to plot. It will look like something like this. There is a mistake done. It should be here because middle is 0. To redo it, this is 0. Therefore, this will look like this. This part is f x is 0. This part f x is equal to x. So, compared to this, difference is this part. f x is equal to 0 between minus infinity to 0. But, between 0 and plus infinity, this is a linear function of x. So, these are the four important functions, which are called activation functions or this function. There is another function, which is used. It is called, I just simply write. It is not working. I will just write there. So, it is called Leaky Rayleigh. Very funny name. It is a combination of these two functions. Let me draw it and show you in a small way. I hope you will be able to make out. But, this is rarely used. No need to worry about it. So, first one is exactly like that. This second one is little bit different. So, this is f x 0. This is f x is equal to x. This is f x x is equal to a x and this is f x is equal to x. That is what it is. So, let me write this one little bit differently. So, that is what the Leaky loop function will look like. These are the different functions, which are widely used in the literature for ANN. Please remember that. These functions you will be using for your modeling purpose, but you may not know. That is what I thought I should give you in a simple chart. Next slide will also have it. You can see that. So, you have first one is called identity and it has derivatives is 1. Second one is this binary step. Very important. Then, you have sigmoid 1 by x minus x or tan h x. I made some mistake again. Sorry about it. So, tan h x is like different. This will be 2. This is correct tan h x. So, that is correct. So, you can see that differentiation here and do not worry about arc tan. Arc tan is basically tan inverse. Then, you have Re loop and then, you have other things also possible. Some people use this one. Experimental linear unit ELU. This is alpha exponential to the power e to the power x minus 1 or it is linear when x is more than 0. So, why this different type of functions and their curves are given on the left side for your purpose? You can see that these plots are given here. Function is here. Derivative is here. So, this is derivative. This is the function. This is the plot. So, you can see that how complex these functions can be. So, that is why ANN can actually simulate or they do a deep learning methods like ANN and can actually take care of any complex problems. We can solve this complex problem with these activation functions, which can be widely used. So, let me just do little bit of that and we stop here. So, it is, remember perceptron is the unit. You can have multi layers. You can have many layers and this is simply by teaming up with different perceptrons and each of these perceptron will do the same calculation, similar functions, same calculations and that is what is done. So, one is input layer, output layer. Then, you have lot of hidden layers. This has been talking about for so many times and we can have, you know, deep or shallow networks or deep networks. So, that is also we talked about it, but important aspect is this. Although we are adding a bias, how do you know that weights, which are putting on each of these things are proper or not? That is why you need to do a backward pass just like a foot wall with a backward pass. This algorithm commonly used to train the neural networks. That is what is called back propagation. It performs very nicely using iterative backward passes, which attempts to minimize the loss, loss quote and unquote. There is a difference between the correct predictions and the actual model predictions and each backward pass weights are moved forward, backward forward to get an optimum, you know, values, so that loss can be minimized. That is what it is. So, not much on these aspects. When neural network initialize, weights are set by you and it does not matter what weights you are putting on. Although you have some idea, how, which one is more important, which feature is more important than the other feature, like whether composition is more important than temperature or pressure is more important or stress is more important, which one, but still you can, you can use some weights, which may not be the perfect ones. That is why once the input is loaded, they pass through this network of neurons and they will give you some outputs and given the values of the initial weights. So, back propagation actually helps to adjust this weights of the neuron, so that results will become closer and closer to the, what is known. Well, we talk about these aspects, you know, in the next lecture. In the next lecture actually what we will talk about is this CNN or conmonate and how this is going to work in very nicely in describing many of the problems. Let us wait for the next lecture.