Lec 27 | Applied Optimization | Jensen's Inequality and its Application-I | IIT Kanpur

Hello, welcome to another module in this massive open online course. So, we are looking at convex functions and in particular, convex functions of a vector variable, vector correct and how the test for convexity that is a function which convex function, the test of convexity for a function of a vector. So, let us change tracks a little bit and look at something known as Jensen's inequality which has significant applications in various areas. So, what we want to look at in this module is Jensen's inequality for you can say convex as well as which is a handy tool that arises in several scenarios. In fact, we will try to justify this by looking at a practical application. So, this is a Jensen's inequality and the Jensen's inequality is very interesting and that is as follows, that is remember if you go back and look at the definition of the convex function, it is this bowl shape function we said and if you look at two points x 1 and x 2 and you take any linear combination that is let us make this as vectors theta 1 x 1 bar plus theta 2 x 2 bar and this is the value of the function at the linear combination and this is the value of the function that is this is remember f of theta 1 x 1 bar plus theta 2 x 2 bar and this is theta 1 times f of x 1 bar plus theta 2 times f of x 2 bar, where we can say theta 1 you can set it as theta remember theta 2 equals 1 minus theta or you can simply say theta 1 comma theta 2 greater than equal to 0 and theta 1 and theta 2 satisfy this condition that theta 1 plus theta 2 equals 1. So, this is a convex combination of two points theta 1 times x 1 bar plus theta 2 times x 2 bar, where theta 1 theta 2 are two scalar quantity two numbers which are greater than or equal to 0, which are non negative and sum theta 1 plus theta 2 equals 1, which automatically implies since they are non negative and sum to 1 automatically implies that both theta 1 and theta 2 lie in the interval 0 to 1. Further, now you can see the and therefore, the inequality for the basic inequality is that f of theta 1 x 1 theta 1 x 1 bar plus theta 2 x 2 bar is less than or equal to theta 1 f of less than or equal to theta 1 f of x 1 bar theta 2 f of x 2 bar this is the inequality for a convex function for convex function f f of x bar. Now, you can see that now look at look go back and look at this quantities theta 1 plus theta 2 greater than or equal to 0 and theta 1 plus theta 2 equals 1 remember this should remind you of something this remind you of probabilities reason being you have two quantities theta 1 plus theta 2 which are non negative greater than or equal to 0 and the sum to 1. So, this remind you of the this should remind you of a probability distribution or a probability mass function in this case, which you can say that you can consider a distribution probability x equals x 1 bar equals theta 1 and the probability x equals x 2 bar equals theta 1 that is the probability random variable x takes x 1 bar is x 1 bar with probability theta 1 and it is x 2 bar with probability theta 2 sum of the probabilities probabilities are greater than or equal to 0 and sum of the probabilities is 1. And now therefore, if you look at this quantity now it is interesting if you look at this quantity theta 1 times x 1 bar plus theta 2 times x 2 bar this is equal to what is this equal to? Well this is if you look at this is x 1 bar or this is probability x equals x 1 bar times theta I am sorry times theta one is probability x equal to x 1 bar times x 1 bar plus probability x equals x 2 bar times x 2 bar this probability x equal to x 1 bar is theta 1 probability x equal to x 2 bar is theta 2 that is, what we said above. And therefore, this is nothing but, if you can look at this, this is the expected value of the random variable X, because remember, what is the expected value? Expected value is nothing but, you take each possible value, multiply it by the corresponding probability and sum, that is expected value. This is nothing but, summation over i probability X equal to X bar i times X bar i, this is your definition for the expected value of random variable of X. So, this is the definition for the expected value of X. Further, if you look at this quantity and therefore, if you look at this quantity, that is f of theta 1 X 1 bar plus theta 2 X 2 bar, you can now write that as f of, well that is f of simply expected value of this random variable X, because remember, we have shown that theta 1 X 1 bar plus theta 2 X 2 bar, that is basically equal to this f of, that is the expected value of X. Now, on the other hand, if you look at this quantity, that is theta 1 f of X 1 bar plus theta 2 f of X 2 bar, well this is equal to, again I can write this similarly, f of X 1 bar times the probability that X equals X 1 bar, which is theta 1 plus f of X 2 bar times the probability that X equals X 2 bar. And, this is equal to the expected value of f of X bar, this is equal to the expected value of f of X bar. And therefore, what we have now, if you look at for a convex function, what we have just shown is that theta 1 f of X 1 bar plus theta 2 of f of X 2 bar, that is the convex combinations of the two points, that is the point on the chord is nothing but, the expected value of f of X bar. And, for a convex function we know, for a convex function we have f of theta 1 X 1 bar plus theta 2 X 2 bar, this is less than or equal to theta 1 f of X 1 bar plus theta 2 f of X 2 bar, well what is this, this we have seen is expected value of X and this is expected value of f of X. Therefore, we can represent the same thing as f of expected value of X is less than or equal to the expected value of f of X. And, this is basically Jensen's inequality for a concave function, convex function I am sorry, this is basically your Jensen's inequality. So, very important and a very handy tool, as we will just, we will see in a practical application this is Jensen's inequality. And, it is frequently used in signal processing and communication, especially if you look at information theory, there are several instances where this Jensen's inequality is fairly handy to prove various results, it states that for a convex function f of the function of expected value of a random variable is less than expected value of the function of that random variable. So, this is again f of expected value of X less than or equal to expected value of f of X for concave, for convex. And, for concave it is the other way round. So, this is for a convex function, for concave we will naturally have the reverse of this, there is f of expected value of, because the chord lies below the curve. So, f of expected value of X is greater than the expected value of f of X. And, this is basically for your concave function and this is for your concave function. So, these are basically you can think of this as basically Jensen's inequality for a convex function and Jensen's inequality for a concave function. And, so let us now, and you can generalize this to, well we have simply considered random variable that takes two values, you can generalize it to, you can generalize this as follows. So, you can consider theta 1, theta 2, theta n as the probabilities and X 1 bar, X 2 bar so on, X 1 bar are the various valuables of the random variable. Then, you have again f of expected value of X equals f of summation i theta i X i bar, i equals 1 to n is less than or equal to f of, or is less than equal to expected value of, again for a convex function, expected value of f of X, which is basically summation i equals 1 to n theta i X bar. So, this is the Jensen's inequality. In fact, it holds even for a continuous random variable X. In fact, that is what makes this very interesting and a very powerful inequality. And in fact, that is what we are going to see shortly in a practical application in the context of a communication system. So, what we want to see now is, we want to see a practical application of this to demonstrate the applicability. So, we want to see a practical application. And therefore, consider the channel y equals, well consider a communication and this practical application is in the context of the BER, what is termed as the bit error rate. I think all of you are familiar with this or most of you are familiar, those who work or those who are familiar with the properties of a communication system or the performance analysis of communication systems. This is basically bit error rate, which also is the probability, that is, it denotes the probability with which a bit is received in error over a communication channel. So, consider a very simple channel, which where y is the received symbol, x is the transmitted symbol and n is the noise, correct. So, y, this is your received symbol, that is the channel output, this is your transmitted symbol and this is your noise, which is typically assumed to be white Gaussian, correct. This is white Gaussian with mean 0 and variance sigma square, x can be a symbol, typically take this as plus or minus square root of p, that is, BPSK symbol power, BPSK stands for binary phase shift keying, power is p. The noise is white Gaussian and it is additive in nature, correct. Therefore, this is termed as an additive white Gaussian noise channel. Therefore, this is termed as an additive white Gaussian noise channel. And since this is an additive white Gaussian noise channel, if you are transmitting BPSK symbols, power square root of p, then we have the SNR is nothing but, the symbol power, signal power divided by noise power, that is, sigma square. And we can denote this by gamma. And the interesting thing about this is, if you look at the bit error rate of an additive white BPSK of communication, of digital communication, BPSK or the transmission of BPSK modulated digital symbols over an additive white Gaussian channel, that is given by the well known expression, that is, the bit error rate is given as Q of square root of SNR, that is equal to Q of, Q of square root of gamma, this is the bit error rate of BPSK over AWGN. And now, however, so now, this model, additive white Gaussian noise model is generally associated with a conventional digital communication system, in which there is a wire medium between the transmitter and the receiver. This is also known as a wired channel, a wire based channel, such as the twisted copper pair or a coaxial cable and so on, or your conventional telephone, where there is a wire that connects the telephone to the local exchange and so on. So, this is a model for a conventional communication system or a wire line, what we call conventional or basically your, this is your conventional or basically your wire line system. Now, on the other hand, what happens in, if you look at a wireless system, something interesting happens in a wireless system. In a wireless system, you have a base station, which is transmitting, this is your base station, which is transmitting to, let us say, a mobile in the cell. Then, in addition to this signal, which is, you can call this as a direct path, there are also going to be several reflections or what are also known as scattered paths. So, these are also known as the scatterers, such as large buildings. So, these give rise to these non-line of sight, the NLOS, non-line of sight, scatter. So, there is a line of sight. Now, what happens, when these multiple paths, the signals from multiple paths, they superimpose, correct. So, when these signals superimpose, so multiple signals from the multiple paths, this is also known as a multipath environment, multiple paths superimpose, implies this leads to interference, this leads to interference, correct and that is the problem. And once you have interference, the signal can, now interference need not be only destructive, interference can also be constructive, but the moment you have interference, there is uncertainty in the received signal level. The signal level can dip, if the interference is destructive or the signal level can rise, if the interference is constructive. So, in general, the signal, the level of the signal or the power, the power of the received signal is varying with time, unlike a conventional wire line communication system, where there is no phenomenon of multipath reflection. In a wireless system, because of this multipath reflection phenomena, the resulting interference leads to a time varying power for the received signal. This process is termed as a, this process is termed as fading and this channel, the wireless channel is known as a fading channel. So, this leads to variation in the received signal, which is termed as fading, implies that the wireless channel is a fading channel, as we have seen in some examples before. Now, therefore, I cannot simply use the AWGN channel model for the wireless channel, I have to use a different model. So, for instance, your wire line or a conventional channel model is additive white Gaussian noise, that is your y equals, received signal y equals transmitted signal x plus noise. In addition, in a wireless channel, I will have the presence of a multiplicative factor h, which is a coefficient, this is termed as the fading channel coefficient, this is termed as the fading channel coefficient. So, this is termed as a fading channel coefficient and this is a random variable. The important thing to note is, since the received power is random, this fading channel coefficient is a, this is a random variable, this implies that received power is random in nature or received signal level, the received signal level is random in nature. And therefore, now you can see the SNR is influenced by this fading channel coefficient. So, one can write the SNR, which was previously P over sigma square, will now be multiplied by magnitude h square. So, it is P over sigma square, remember this is gamma. So, this is gamma times magnitude h square. So, you can think of this as the SNR of the wireless channel, SNR of wireless channel, this is the SNR of the wireless channel. And therefore, now what we can do is, we can look at the resulting bit error rate of, the bit error rate depending on this SNR of the fading wireless channel and apply Jensen's inequality and derive a suitable conclusion. So, this is the practical scenario in which we want to apply the Jensen's inequality and we will continue this discussion in the subsequent module. Thank you very much.