Lec 04 | Applied Optimization | Inner Product Space and its Properties-I | IIT Kanpur

[Music] [Music] hello welcome to another module in this massive open online course we are looking at the mathematical preliminaries for optimization let us continue our discussion with another concept namely an inner product space ok so you want to start looking at the concept of what is known as an inner inner product inner product piece now what is an inner product space now an inner product space of a real vector now the inner product of a real vector space the inner product of a realtor space is an assignment of a real number is an assignment of a real number and for any two vectors u bar V bar that is denoted by this notation u bar the inner product of u bar comma V bar and this is defined for any two vectors u bar b bar in this case real vectors for any two vectors u bar comma v bar and this is the inner product which is a real number in the case of a real vector space and which satisfies the inner product satisfies the following properties the inner product of the vector satisfies the following properties first property is linearity which is a u bar Plus that is the inner product of a linear combination a u bar plus BV par is the linear combination of the inner products that is this is a times u bar inner product of u bar W bar plus P times the inner product of V bar comma W bar okay so this basically is the linearity property that is a linear code during the product of the linear combination au bar plus BV bar with the vector W bar is the linear combination of the inner products a times the inner product u bar W bar plus B times the inner product of B bar comma W bar okay this is the first property then we have the symmetric property or the symmetry it is very simple that is the inner product of u bar V bar equals the inner product of V bar comma u bar then we have so if you call this property number one linearity to symmetry then we have the positive semi definite property the positive semi-definite property it must be the case that for any you bar that is for any you bar element of the vector space V then the inner product of U bar with itself must be greater than or equal to zero and more importantly and also you are the inner product with itself equals zero if and only if u bar is itself zero so inner product is zero if and only if the vector u bar is zero so these are so this is the definition it's an assignment of a real number for a real vector space it satisfies the linearity symmetric a symmetry and the positive semi-definite properties ok let us look at a simple example to understand this for instance the standard product let us consider the standard dot product between two vectors all right consider two vectors the dot product is defined for between two vectors consider u bar equals u 1 u 2 up to u n + V bar equals V 1 V 2 up to V n then now these are real vectors and dimensional vectors that is you can say that belong to the space of n-dimensional real vectors denoted by this bold r2n and is also termed as the euclidean n space this is also termed as the euclidean in space and the inner product in this euclidean in space between vectors u bar and V bar is defined as u bar transpose V bar which is basically if you look at this that is u 1 u 2 UN row vector times the column vector V 1 V 2 VN which is basically U 1 V 1 UN VL okay so this is the definition of the dot product okay so this is the dot product between two n-dimensional real vectors something that you might have will see in your in your vector space or vector calculus class in high school knowledge is a standard dot product which is also denoted by the dot operator that is U bar dot V bar okay this is the dot product between two vectors in fact more specifically to real and dimensional vectors now we will show that dot product is an inner product now that is very simple to see first let us look at the linearity property if you look at a u bar plus BV bar dot product with W bar that is simply a u bar plus BV bar transpose times W bar as we have seen that is the definition of the dot product which is a times U bar transpose W bar plus B times V bar transpose W bar which is a time's the inner product of u bar with W bar plus B times the inner product of V bar with W bar okay so it satisfies the linearity property now too now coming to the symmetry property we have you bar dot product V bar which is equal to u bar transpose V bar which is basically V bar same as V bar transpose u bar which is V 1 u 1 plus V 2 u 2 plus VN u n which is V bar u bar and 3 so symmetry satisfies a symmetric property now we have to look at the positive semi definiteness no positive semi-definite property now you can see that the inner product of U bar with itself is U bar transpose u bar which is U 1 square plus u 2 square plus u n square which is greater than or equal to zero further in fact we have seen this is nothing but this is used to define the two normal square that is this is the L 2 norm square norm u bar square which is greater than or equal to zero in fact equal to zero if and only if some of the squares is zero which means each of the components is 0 u 1 equals u 2 equals u of N equals 0 which means you are equals zero so it's positive semi-definite its positive semi-definite that is U bar inner product of u bar with itself is always greater than equal to 0 it is 0 only when u bar the vector is identically 0 all right and therefore the standard the dot product is is an inner product and in fact this is also termed as the standard inner product this is also termed as standard inner product on RN that is the Euclidean in space or the n-dimensional set of in dimensional space of real vectors Euclidean in space or the in dimensional space of real vectors the dot product is an inner product let us now consider another example for two dimensional vectors instance we have X bar equals x 1 x 2 and y bar equals y 1 y 2 so both of these are basically 2d vectors that is there belong to the two dimensional Euclidean space and let us define this assignment X bar Y bar for this to 2d vectors as twice X 1 Y 1 minus X 1 Y 2 minus X 2 y 1 plus 5 X 2 y 2 now what we want to do is we want to show that this is an inner product this is a valid this assignment is a valid inner product and this can be shown as follows as usual we start with the linearity property now let us consider a vector that is a X bar plus BX tilde that is a times X 1 X to the 2 dimensional vector plus B times X 1 tilde X 2 tilde all right so we are taking a linear combination of the two vectors X bar and X tilde now for this to be an inner product let us consider now a X bar plus B X tilde the inner product with any vector Y bar now this can be shown as now we can see this is equal to twice X 1 plus B X 1 tilde into Y 1 minus X 1 plus BX 1 tilde into y 2 minus X 2 plus BX 2 tilde into y 1 plus 5 times X 2 plus B X 2 tilde into y 2 and now you can clearly see this is linear that is this can be written as a if you separate the components and write it you can easily see that this is a times X bar comma Y bar plus B times X tilde cama y bar and therefore it is linear okay so satisfies implies that this is linear okay alright so we have shown that the assignment is linear now let us know symmetry the symmetric property and this can be shown as follows so coming now to the aspect of symmetry you can see that X bar Y bar this is equal to twice x1 y1 minus x1 y2 minus x2 y1 or minus y2 x1 I'm sorry minus x2 y1 minus x2 y1 + 5 x2 y2 okay and now this can also be written as without any effort you can see this is twice x y1 x1 interchanging the terms the second inter term this will be minus y1 x2 minus x1 y2 plus 5 y 2 x2 and you can readily see that x1 y2 I will write as minus y2 x1 and you can readily see that this is nothing but the assignment of Y bar comma X bar so we have X bar Y bar equals y bar comma X bar and hence in this implies that that it satisfies the symmetry property and finally now coming to the positive semi-definite e is positive semi-definite property coming now to the positive semi-definite property now if you look at X bar X bar that will be twice X 1 square minus 2 X 1 X 2 plus 5 X 2 square which is equal to now I can write this as the sum of two terms X 1 square plus X 2 square plus 2 X 1 X 2 plus X 1 square plus 4 X 2 square minus 4 X 1 X 2 and this is equal to well X 1 plus X 2 whole square plus X 1 minus 2 X 2 square which is greater than or equal to zero so sum of two perfect squares this is greater than equal to zero so implies this satisfies the positive semi-definite property this satisfies the phd property and in fact you can see this is equal to zero only when it is equal to zero only if x1 plus x2 both must be zero because it's sum of perfect squares so both must be 0 X 1 minus 2 X 2 must also be 0 and this implies that X 1 equals x 2 equals 0 so it's only 0 when X 1 equal to X 2 equal to 0 that implies X bar equals so we have the PSD property which is X bar X bar the sign when it is greater than or equal to 0 and equal to 0 only if X bar equal to 0 that is what we have shown therefore it also satisfies the PSD property hence because it satisfies the linearity symmetry and PSD properties this is an valid inner product all right so finally what we have is that takes bar for my Y bar equals to X 1 Y 1 minus X 1 Y 2 minus X 2 y 1 plus 5 X 2 y 2 we can now claim that this is a and this is not a coincidence in fact such an inner product can be constructed by observing the following thing I can write this thing as this is equal to X 1 X 2 twice minus 2 minus 1 minus 1 comma 5 y 1 y 2 which is nothing but X bar transpose a Y bar where a is this matrix you have a is the matrix 2 cross 2 matrix which is given as 2 minus 1 minus 1 5 you can see that this is nothing but identical to the this basically another way of writing the inner product that we have just defined and now you can show in a very interesting property of this mater if in fact this matrix a is a positive semi definite positive definite matrix all right so you can see that this matrix a is a positive definite minute definite matrix and you can see this as follows remember we said this matrix is first see that this matrix is symmetric we have a equals a transpose further if you look at the eigenvalues a minus that is if you set the determinant of a minus lambda equal to zero then what we have is you have the determinant of 2 minus lambda minus 1 minus 1 pi minus lambda equal to 0 this implies 2 minus lambda into 5 minus lambda plus 1 equal to 0 this implies now if you simplify this lambda square minus 7 lambda I'm sorry this determinant is minus 1 minus lambda square minus 7 lambda plus 10 minus 1 equal to 0 implies lambda square minus 7 lambda plus 9 equal to 0 implies lambda equals well 7 plus or minus square root of 7 square 49 minus 14 9 36 7 plus or minus 13 divided by square root of 13 divided by 2 and you can see both the eigenvalues are greater than 0 all it has two eigen values and the eigen values are strictly greater than again values strictly greater than 0 so symmetric plus the eigenvalues greater than 0 implies a is positive definite okay so a is a positive definite matrix so a is a positive definite matrix and therefore hence X bar transpose so now in fact an interesting property and you can easily show this that is if you look at two vector if we define a neural inner product that X bar between X bar and y bar as X bar transpose a Y bar where a is positive definite this is a inner product that is this is a easy this is a inner product where a is a symmetric positive let me also just write this Acer 8e is in that product this is an inner product X bar transpose a y bar where a is a symmetric positive definite matrix is an inner product alright and one of the other interesting aspects of the inner product is that it can be also be used to define a norm and that is one of the most interesting and important aspects of the inner product it induces a norm and that norm is given as follows so the norm is so inner product can be used to define a norm so the norm can be defined using this concept of inner product and in fact it can be defined as follows we have norm of u bar Square this is equal to the inner product of u bar with itself which means basically the norm of vector u bar is equal to square root of the inner product of u bar with itself and in fact the unit norm vector can now be defined as u hat equals u bar divided by norm of u bar that is u bar divided by square root of u bar u bar and in fact what we have seen is and this also this process also termed as normalization that is when you divide a vector by its norm you can also say that the vector is normalized and you can also see this is true for the the standard inner product that is the standard inner product on RN that is if you look at X bar for my X bar where X bar belongs to RN and this is the standard inner product we already see in that inner product of X bar with itself is basically this is X 1 square plus X 2 square plus X n square which is nothing but the l2 norm square okay and therefore this is the square of the norm and we have already seen this for the standard inner product what this result says is that not just for the standard inner product which is given with the dot product of two vectors but any inner product on these two vectors X bar Y bar can be used to define a norm that is the norm is given as the square root of the inner product of X bar with itself ok for instance in the previous example we have seen X bar transpose a Y bar all right which means the norm of X bar under that inner product is given as square root of x bar transpose ax bar okay so the norm of X bar is equal to square root of x bar transpose a X bar and this is for the previous example okay let us look at other examples of inner products so we have some other examples we are already seen you bar transpose V bar that is the standard normal RN this is an inner product this we already seen now another interesting application of inner product is let us consider the space of continuous functions on an interval a comma B denoted by C of a comma B that is the set of continuous functions on a comma B on interval a comma B and let's say we have two functions F comma G which belong to C of a comma B that is their continuous functions on the interval a comma B then the assignment defined as f of G equals integral over the interval a comma B f of X G of X or f T DT that is this integral f of X G of X DX this is an inner product in fact this is a very interesting application in fact this can now be used to define a norm so this is an inner product for functions F comma G and in fact the norm that arises is basically nothing but norm of F is integral a comma B or norm of F square a is basically inner product of F with itself that is integral of a comma B integral on the interval a comma B F square xdx or this is nothing but the energy of the signal that is if you replace if you look at this as signal in time if I think of this as a signal in time this is basically the energy of the signal in interval this is the energy of the signal in the interval a comma B alright so that is another so it can also be used so basically one can also define an inner product correct one can also define an inner product on the space of continuous functions all right and therefore define the norm or the norms norm and those as well as the norm square in fact the norm square of the function is based or the signal is the energy of the signal in that particular interval another interesting example of this inner product space consider the space of M cross n matrices so we have the space of M cross n matrices example m equal to 3 n equal to 2 implies we have three closed two matrices and for instance the 3 cross 2 matrix is R equals a 1 1 a 1 2 a 2 1 a 2 2 a31 a32 and we have B the matrix B equals b11 b12 b21 b22 b3 1 B 3 2 and the inner product a be defined as trace of B transpose a that is the now this is an interesting concept that is trace trace of a square matrix this is the trace of a square matrix the trace operator this is defined as equal to is equal to sum of the diagonal elements of a square matrix some of the diagonal elements of a square matrix that is the trace of the square matrix and this can be shown to be an inner product this can also be shown to be an inner product trace of this can also be shown to be an inner product alright so what we have done in this module is we have looked at the inner product its definition the various properties or when is an assignment and inner product and several examples we will continue this discussion in the next module thank you very much [Music] [Music]