Lecture 34 : Part 1 - Advanced Deep Learning-I

Welcome to lecture number 34. In the last 3 to 4 lectures, we have focused on the length scale issues in materials engineering. So, now we are going to change the topic. We will be talking about some of the advanced deep learning methods. Here, I will focus on basically few techniques which are relevant for materials engineering, but before that let me tell you that for the section on the length scales, different length scale issues in materials engineering. I hope you have got a good glimpse about the whole thing in the sense that it is a vast topics. So, we wanted to provide you some basic knowledge on this topic. Idea of providing that section was mainly related to make you believe that we can generate lot of data using simulations or using theoretical calculations or computational techniques like whether it is a computational technique like Calford or phase field modeling or DFT or MD simulations. So, that each of these techniques can provide you lot of data which are reliable and can be used for application in machine learning and deep learning. That is why we wanted to teach you that, but if you want to know details of each of these techniques, you need to follow different courses in NPTEL as well as books and the papers. These are actually part of the whole subject called computational material science and which is taught in various universities and institutes. So, let us move on. We will talk about advanced deep learning methods in this lecture, but as usual first we will look into some of the recaps what we have learnt so far in materials design at different length scale. These recaps actually are going to provide you one important aspect that is the snippets of the discussion which are important for you to learn and move ahead. So, you know we can use advanced computational tools, computational material science tools to do various kinds of calculations. We can actually use them to reduce the cost and the time consuming experiments. It is widely been used for alloy design or material design. So, first we can do first principle calculations like DFT, dielectric functional theory or MD or molecular dynamics to find out crystal structures, stability of them. Not even that we can even find out electronic contributions or electrons, role of electrons, distributions everything using first principle calculations. This is also sometime known as ab initio. So, this is what is been taught by the guest lecturer professor Tarun Kumar Kundu for the last two lectures, lectures number thirty three, twenty two and thirty three. Then from this information we can actually calculate lot of thermo dynamical parameters like enthalpy of mixing, enthalpy of mixings. We can actually determine experimentally also extrapolate also we have I shown you in my lecture and find out phase equilibria, phase diagram informations. It can allow us to find information of phase equilibria of the equilibrium phases as well as non equilibrium metastable phases. So, that is provides us the phases present, but it does not provide you microstructure. And as you have been told that microstructure is the connecting link between the processing and the properties. So, how to get it that? So, for doing microstructure we have discussed about phase field. Phase field is a technique which is widely used to generate microstructures or understand microstructures. And phase field requires you to provide requires the important aspect like interfacial energies, austenite energies as well as mobilities the diffusion coefficients of this. So, here we are showing a microstructure in which precipitates are present in a matrix like aluminum copper alloys, where precipitates are basically present in the aluminum matrix. Now this precipitates can be formed by aging treatments. So, we can actually understand that also. Finally our basic goal is to connect the microstructure with the properties such as mechanical functional properties. So, therefore, we also need to understand micro mechanical models for that, but remember I am we are not going to discuss about this in this course. There are many micro mechanical models which can be used to predict even mechanical properties. So, hence our discussion is limited to them. So, and many times if you want to get a full scale microstructure the slides I have shown you already. We need to connect thermodynamics and kinetic databases. Thermodynamics provides you phase stabilities, kinetics provides you information about this phase transformations how the phases are going to form that can be obtained from that. So, basically experimental diffusivity data or diffusivity data from atomistic calculations like MD simulations can be used to find out mobility databases. And in fact one can also use machine learning tool to optimize that data and that is what is used in dicta or diffusion interface control phase transformations. Diffusion and interface control transformation that is called dicta. Diffusion is D, interface is I, control is C and transformation is TRA. So, we can use thermocol using FANDAT or even thermocol database also and find out phase diagram information is shown here. We can also find using diffusivities we can find out the rate of reactions or phase transformations. And once you have the microstructure using for phase field simulations we can find out elastic properties. Interfacial properties has to be used to generate the microstructures that is interfacial energy, strain energy all these aspects that also can be calculated from ab initio simulations. So, why thermodynamics? Because properties are what is important for us. We want to design materials with target set of properties mechanical, chemical, electrical, magnetic or thermal or any other properties and properties are dependent on structure as you understand. Structure means atomic, crystal, grain structure, phase distribution, defect structure. So, in ourselves from atomic scale to the micron scale. And then you make any component by processing and during processing you are using different kinds of variables like temperature, pressure, stress, strain and chemical composition of the alloy. So, that is how is the whole processing structure property relations. And thermodynamics is the basis for understanding this, because the material can respond in terms of the change in temperature, change in pressure or even compositions and we can understand that. And finally, we are interested to know the performance and which is dictated by connecting these three links together, processing structure and properties. So, that is how thermodynamics plays the most important role. And in thermodynamics we have to use you know data mostly to calculate the phase diagrams and phase equilibrium information. Data can obtained from experiments as well as first principle calculation and then we can actually predict different kind of phases. Nonetheless, we can use statistical thermodynamics also to obtain some of the thermodynamics data. So, what are the different properties we are looking at? We are looking at enthalpy, entropy, use of energy or thermal expansion or the heat capacity as a thermodynamic properties. We can also look at kinetic properties like diffusion coefficients or shear viscosities. On the other hand, mechanical property like elastic and plastic properties as well as the interfacial energy properties like all these things stacking polar energy, twin dislocations can also be calculated using first principle simulations. And this can be all used to gather information about the phase transformation, phase equilibrium and the microstructure evolutions also. So, therefore, you need to connect data science with this whole picture what is known as ICME, Integrated Computational Materials Engineering, which is the big umbrella that requires lot of data. That is the basic goal. Data will be connected with the ICME approach to calculate phase diagrams and understand the phase information. Now in case of CALFED, we have to use five different things. First, we have to capture data either from experiments or the calculations, analyze and monitor or modeling the data which is called assessment part as shown here and then followed by optimizations because we need to fit certain parameters to optimize the thermo dynamical functions. Finally, we store the data some file data files which can further be validated using experimental data. So, finally, it will be applied for various kinds of uses. There are five steps data capture, assessment, optimizations, storage and validations. And this requires that is for this finally, general thermo dynamical database which will be widely used for various kind of applications. One can look at this thermo CAL website for details of these. So, that is what is shown here in this sense and Gibbs free energy is calculated, sorry Gibbs free energy is basically found out for all phases using this approach which is called assessment and optimizations. So, one can do thermo dynamical calculations, one can do diffusion calculations which is called diffusion interface, control phase transformations as well as one can do also both together which is done in PRISMA. So, I do not want to go in details of this. This is the thermo dynamic approach I used. I talked about it. So, thermo dynamics means free energy. We talked about Gibbs free energy in material systems that has three components which is called the pure components, the entropic or mixing component and third is the excess entropic contributions that has to be assessed properly because that is the difference between the actual free energy and the free energy obtained using both the pure component and the entropic contributions. So, you can see G excess is nothing but G minus these two terms. So, therefore, G excess has to be measured obtained from binary data set. Once you have this is possible to obtain because we know the binary phase diagrams of all the systems are available in the literature. So, you can easily get that, but ternary and quaternary may not be available, higher order systems may not be also available. So, what you do is extrapolate the free energy from binary to ternary and quaternary. Finally, this free energy functions are obtained and they are globally minimized to get which phase is stabilized more or less. So, we also discussed about something about phase field simulations and in particular I talked about how to obtain microstructures in a precipitation formation system like aluminum copper alloy and in a main idea of this phase field simulation is to solve a single set of partial differential equation of the entire domain. So, remember there are two aspects in phase field simulations because we are talking about a phase transformation from a single phase microstructure to a precipitate plus matrix. So, we need to have information of thermodynamic information like biothermodynamics as well as we need to have information about the surface properties, interfacial properties and diffusivities also. So, these two together must form a single set of partial differential equation and this that is why phase field model incorporates both the bulk thermodynamics of the multiphase systems as well as surface thermodynamics like Gibbs free Gibbs surface excess quantity which is nothing but surface energies. And free energy is always approximated like that is a total free energy is a volume free energy plus surface energy. And we know that important aspect in phase field simulation is to assume the evolution of the one variable phi as a function of time. So, therefore, for the whole system when you are talking about phase transformations phi is allowed to vary continuously from one phase to other phase via the interface. This is what is shown here for this phase at the top the phi value is 1 for this phase phi value goes to 0 in the interface phi value varies from 1 to 0. So, that is why actually we can track the interface very easily by simply looking at the values of the phi as a function of time how it evolves. So, for solid liquid phase transformations we can use this kind of phi, phi is 1 for solid liquid is 0. And so that is why this is the in a free energy expression you can see here this term is basically coming from surfaces or interfaces and that term next term is coming from bulk free energy the bulk. So, you obtain the free energy then we can solve this dynamics a variation of phi as a function of t using this in this equation where m is called mobility and this mobility is nothing but relative diffusivity. So, that is one can easily solve and we can actually have different kind of free energy equation basically we need to use double well potential to the like this kind of functions or this kind of functions to be used here you see that for phi is equal to 0 g is 0 and phi is equal to plus minus 1 g is equal to 0 here for phi is equal to 0 it is 0 and phi is equal to 3 by 2 it is also 0. So, therefore, we have a double well functions as you can see here also you have a double well function it will come down and become 0 sum. So, that is why this kind of functions actually help us to make the simulation better and for liquid solid we always use free energy differences between the solid and liquid and that is nothing but latent heat multiplied by under cooling divided by the melting temperature and do that. So, I do not want to go in details of each of these functions and all this stuff which you can read from my earlier lecture as well as from the stokes. So, we can make various kinds of simulations here we can have dendritic growths as you can see here nice trend rides are going these are taken from nest national institute of science and technology USA. We say acknowledge their you know availability of this kind of microstructures on the websites. So, therefore, you know the microstructure evolutions of solid to liquid phase transformation can easily obtain we can also show how this microstructures dendrites can form in a liquid to solid taken from G Mac Mullis and Rosen of university of lives. So, that was something I discussed. Now, next of the lectures were discussed by professor Tarun Kumar Kundu that is on lecture number 32 and 33 and here briefly I will talk about it what probably has been discussed. You know with important aspects in for electronic and atomic structure is what is known as a the Schrodinger equation that is nothing but H psi equal to E psi H is Hamiltonian and is the energy and this is something which is the one which shall be widely used. So, electrons are present in the in molecules as well as materials or atoms we have to expand this psi which is the wave functions that provides you atomic orbital or plane waves solve this equation for self consistency and we can obtain property like energies molecular orbitals densities. And we can also obtain for spectroscopic properties like NMR IR usual things also because they are also electronic interactions. So, once you can understand electronic structures very easily by solve this it is possible, but you know this equation is not a standard equations it requires lot of understanding solution of this equation for multi atom systems or even say multi electron systems is very difficult. It is easy to solve for hydrogen and helium probably, but for multi electron system it is not easy to solve. So, consider a single hydrogen atom it is possible to calculate the potential energy very easily and we can also calculate the kinetic energy. So, total energy is equal to this. So, now H psi equal to E psi can easily be solved and we can find out from the psi the electron distribution. So, probability of electron will be depend on an absolute value of psi square you know. So, we can easily find out that. So, one can also find out elastic properties by looking at these kind of binding energies. This is the bell curve which this is summation of total potential energy plus kinetic energy simply we can find out this and then we can also find out the minimum distance between the two atoms well. So, hopefully you know this bell curve I am not discussing much of that. So, this is something which is very easy to do, but you know this allows us to you know prevent the localization of electron near the photon and that is the uncertainty principles and it also returns the collapse of the atoms provides correct density of the matter and also yield the properties proper binding energies of the atoms also. This particular theory is very important, but you know when you talk about electrons. Electrons are not only particles they are also can behave like waves we all know de Broglie's equation right. So, you can convert the momentum in terms of well lengths very easily well lengths of particle depends on its momentum just like a photon the main difference is that the matter particles have mass and photons do not have mass that is the main difference. Well that is about the electronic calculations using has using this ordinary equations one can do actually consider atoms also atomistic calculations also consider atoms as a particles moving in the matter then you can solve the Newton's equation of motion that is what is shown here you can see here acceleration is shown as a force divided by the mass. So, we know that force is nothing but mass into acceleration. So, we can write down mass and acceleration is like this. So, therefore, we can calculate we can write down acceleration like this is nothing but force by mass easily we can do the calculations and this can be integrated and we can find out the location of each atom in the whole system. So, we can do movements of atoms solve numerically and propagate that is how we can find out reaction rates diffusion coefficients stress strain using MD simulations or MD stands for molecular dynamics this is what is called MD it solves the Newton's equation of motion furthermore we can also do electronic structure calculation using a magnesium that is what is called a density functional theory and this is something which is widely used DFT based tight binding models allows us to find out you know nearest neighbors minimum basis we can also use semi empirical models like MOPAC we can fit the experimental data find the nearest neighbors minimal basis also finally, what you are doing to try to do is that by that you are going to find out the electronic structures in the material. So, how it is done simply solving set of equations which I will not go details of that I hope you have learnt from professor Kundu's lectures, but you know it is it is allows to find a density functional of a density electron densities in terms of energies that is what is shown here usually is expanded in orbitals and then one can apply various kinds of boundary conditions to solve that, but please take more literature to understand this is something which you have to do yourselves. So, this is in a nutshell what we discussed about the last four lectures of different kinds of computational techniques which will be used to calculate a generate data let me also tell you that the cost and the number of atoms. So, classical force field you can use almost like a 100,000 atoms cost is very low it scales very easily with number of atoms, but semi empirical methods are reactive force field methods semi empirical method especially are cannot use large number of atoms right, but you know m d can use very large number of atoms 100,000 atoms or so that is what is classical force field methods. Now, d f t can use very small number of atoms 500 maximum and it scales with n to the power 3 minus 1 it is very costly relative cost is almost you know 10 to the power 4 or 5 times larger typical accuracy is very large can you see that compared to this and this and finally, one can actually go to better methods like c c s d's and there you can use only 30 atoms to do the d f t calculation that is what now a days done and you know relative cost is massive and scaling 10 to the power 7 9 a. So, n to the power 7 and we can get down to typical accuracy of there are 1 kilo calorie per mole. So, it tells you how accurately you want your data if you want more accuracy you have to pay more cost or you have to use better systems time also will be larger. So, that is why this depends on the system and properties what you want I just wanted to provide. So, with this let us move into the advanced stiff learning techniques this is something which requires lot of data and that is why actually this you know scaling materials systems all kinds of scaling things we talked about it starting from you know basic things like to our d f t can generate lot of data and these data then can be used for deep learning advanced deep learning techniques that that is something. But here we are not going to discuss everything because that will be quite a lot and that time also is not there to discuss all the advanced deep learning techniques possible we will talk about focus about three techniques there as a d t l d r l and n we use d t l deep transfer learning this d t l comes from d t l d r l that is a deep reinforced learning reinforcement learning error and third one is n l p natural language processing these three things will focus on that, but for the today this lecture lecture number thirty four we are going to talk about d t l and d r l well first to give you some you know aspect which already discussed deep learning is very important and it mimics our the neural network systems in our brain that is why it can allow the computers to understand recovery patterns it can allow us to make decision from vast amount of unstructured data remember this first amount of unstructured data that is why it is used in various domains like for the computer visions n l p health care dynamics autonomous diving and n l p and computer visions are very important for material science because we are going to deal with lot of images where we are going to deal with lot of literature how to use deep learning in this kind of situation. So, basically typically based on neural network architecture and you know we already talked about lot of these things in a n n artificial neural network with part of a m l methods and also a part of deep learning methods in some lectures early lectures. So, and we know that why use deep learning because it is provides you know much better accuracy performance is better especially it provides excellent solutions of for uncovering complex architecture in high dimensional data it provides a key role in building intelligent data driven systems whatever you need today it can also change the world as well as humans everyday life through autonomous power and learning from experience. So, D L is capable of learning from data and generate the patterns autonomously that there is a strong linkage between D B L and the data science which we will discuss even at the last part of this course also and deep learning is good when your data are amount of data are large m l is very good or rather good when your amount data are less. So, what I can conclude is that deep learning performance much better than the m l when for big data and we talked about a n n we talked about c n n we talked about r n n we talked about auto encoders and we talked about also deep philip network or d v n also these are part of a you know normal deep learning models. This chart shows you what all deep learning methods possible remember supervised and unsupervised learning. In supervised we talked about m l p that is the multi layer perceptron model we talked about c n n we talked about r n n and then as a part of r n n we talked about l s t m also g r u also right I hope you remember and you know unsupervised learning we discussed mostly auto encoders a e general advisory network j n some we discussed about it right. Therefore, the advanced deep learning techniques will combine two different techniques together like model one will provides you combination of c n n versus you know l s t m l s t m is a part of r n n recurrent neural network. Therefore, and the model two is talked about g n n plus c n n and that is called net versus plus the generalized adversarial network we discussed about it. Model three connects one of these generative supervised and unsupervised learning with classifiers like s v m here correct. We can also use that is what we talked about we can use d t l and d r l that is what we are going to talk about today what is d t l and what is d r l. So, you know first let us talk about why you need advanced t l methods as you know supervised models are unable to learn from un level data, but it out performs the unsupervised generative counterparts that is what is very important to understand. These models are cannot learn from un level data because they are supervised models, but they can actually give better performance some cases in compared to so unsupervised learning. On the other hand unsupervised learnings are more adaptable. They have capacity to learn from both level as well as un level data. Basically, they are more you know useful with un level data. I hope you know now what is level un level data. So, hence if we can make a frame work for training both these supervised and unsupervised model simultaneously. We can enjoy the benefits of the both the models that is what is the motivation of using advanced d l and this means we have to use hybrid network. So, hybrid d l models that is what I talked about it hybrid you can see hybrid learning methods here that is what we need to discuss in the this lecture. And what is this hybrid d l? They typically composed of multiple deep basic learning models. We have we have just discussed about basic models right already lot of basic models you have learned. So, this hybrid models should basically composed of multiple deep learning models where basic model is either a supervised model or unsupervised model. That is why we need to integrate different models and that is very important and this is something which we will see how more. So, how can you hybridized or how can you make hybrid d l's like you can have a model type one in which you can integrate different generative or discriminative like supervised unsupervised models. Remember generative means unsupervised and discriminative means supervised learning methods. So, therefore, we can combine this different supervised generative models to extract more meaningful and robust features like c n n common n plus l s t m or a e plus g n n. We can also use hybrid model two which is nothing but integration of supervised model followed by unsupervised model example is d b n plus m l p a e plus c n n. We can also integrate a generative or discriminative model by a followed by a non deep learning classifier like s b m a e plus s b m or c n n plus b m. So, we can do this kind of hybrid modeling to obtain a better of two models. So, let me stop here for this part of the lecture we will come back to the next part of the lecture where we will discuss more details about these d t l and d r l methods.