Lecture 41

In this lesson, we'll understand the implementation of principal component analysis and clustering with the airline data case study. We'll visualize the data and extract the PCAs first. Then, using the eigenvalues and scree plot, we'll decide on the suitable number of PCAs, that is, principal components. We'll further analyze data along the variable and individual dimensions. We'll examine the quality of representation of variables and individuals on these principal components. We'll also visualize the variables and individuals on 2D factor map, the dimensions being PCAs extracted, PC1, PC2. We'll also visualize the contributions from variables and individuals on these PCAs. Finally, we'll compute 3 and 4 clusters using k-mean clustering technique. Also, we'll employ survey data containing brand image, price perception, value for money, and features to identify the latent PCA dimensions and correlate with these survey variables. In this video, we'll introduce the airline's case study on principal component analysis and clustering. To begin with the case, you're a consultant to an airline company. It wants you to do a competitiveness and positioning survey. This requires you to examine the company on various parameters including brand image, price perception, value for money, and features. Based on your experience, you conduct two surveys. In the first survey, you randomly select 27 customers and ask about their airline preferences amongst 10 commercial airlines and give them overall points in airline data file. In another survey, you separately choose 27 random customers and request them to give scores to the same airlines on their brand, value for money, and various other features. You also collect historical average prices on common routes, which is available in survey data. Now, to complete this consulting assignment, you need to carry out the following analysis. First, you need to load the airline and survey data, summarize and visualize the data. You need to conduct the PCA analysis and extract the factors from the airline data. Next, you examine the eigenvalues and compute the share of variance explained by each of the PCAs. You need to draw the plot of variances, scree plot, and select the relevant set of factors from the data. Next, examine PCA results with respect to airline's variables. Examine and map each airline with respect to each PCA and plot different airlines on these factors in combination of two. Examine the relative contribution of each airline to different dimensions, i.e. PCAs. Examine PCA results with respect to individuals. Examine the mapping of each individual with respect to each PCA or dimension. Visualize the coordinates with a 2D factor map and plot different individuals on these factors in a combination of two. To conclude the case study, you also need to examine the relative contributions of each individual to different dimensions. cluster the individuals as per their attributes on the graph using the survey data identify the dimensions for example price perception, brand image, VFM features extracted from the principal component analysis and with this we conclude the case. From this video we'll start with our airlines data case study on PCA and clustering. As a first step we'll set the working directory so you get all the files and everything on relevant place. This is the location where I have all my data and files so I'll for record I'll just copy paste this location that I need so I've set the working directory so the first step we'll set the working directory and load the packages. Now just as a test if I have done everything correctly on my files tab the data files airline and survey will appear they are there that means I have set the working directory correctly. Now we'll load the relevant libraries. Please remember I have already installed these packages so I'm just adding it to the current working environment. If you have not installed them you need to first install them before this library command and adding them to the current working environment. So open ggplot then gg40, gg45, grid-extra, so factory-extra, so these are some of the libraries we plan to use in the session in this exercise. So now we have loaded the relevant libraries at the working directory so let's start with first part. In this part we'll load the variables load the data and visualize, load the airline data and visualize. So for that let's use a simple route we'll click on this airline.rds and load it. The data is already read but for reference for future reference you can just copy keep this file. I need not have this full command because I have already set the working directory it has a lot of benefits for example I only need to give the file name I don't have to specify the full address and if I have done it correctly the data is properly read I can run this glimpse command airline. So you can see what this data is it has 27 rows each row indicates the data about individual the survey points to airlines 1 to 10 so each column is column 1 is airline which has now it the representation may seem to be like row is this but no this is sort of column as you'll when you'll print the data so let me show you so if I print the data the columns are airlines and rows are individuals so let us go back to our glimpse command so you can see there are 10 airlines score of individual 27 individuals are collected the best way to visualize this now this this needs a little bit small trick what I'll do is in order to visualize first I need to split my because there are 10 airlines I need to split my plot window now right now I have a one plot window here I need to create sort of maybe 2 cross 5 or 5 cross way if you've seen this command if you've worked before or not this command will split the are interested in 2 rows and 5 column kind of setup and give me a nice plotting area now I can apply a for loop and you'll see the interesting user for loop what will happen is this I variable will run 1 to 10 times and what I want to do is I want to each time this code runs I want to plot a histogram which will contain airline data each time ith column will run so with this command the indexing that I am doing the subscript I am using every time the ith column starting from 1 to 10 that means airline 1 and so on will be the histogram will be printed now very interesting color scheme I'm using I'm fixing color with the variable I so R has each color index to a number also so for example 1 would be index to some color 2 would be index to some color so every time this loop will run the histogram will be plotted and color would be index to I I want to make a nice central heading name so I'm using this space command that will combine two strings so one string is airline number and now I'm using a variable I again same dynamic indexing where I'm using I this I would change a every time the loop will run airline number is a string which will remain constant but will be merged or sort of combined with the variable I which is more dynamic now the last part is I want to keep the xlab empty so I don't want to do much so this is done now so I just want to be sure so let me run it so yeah so you can see all the 10 airlines airline 1 2 3 and then so on their plots are given to me and the colors are different because I have put I have indexed the color with this I so it seems there's some reputation after 6 7 8 9 10 so after 8 there is some reputation seems the 9 and 1 and 10 and 2 are similar so but I still we get the point we know what airline number they are so we can visualize the data like this so this is our data visualization so to summarize this video we set the working directory we added the relevant library packages read the data we summarize the data and visualized also look at the histo looked at the histogram plot from next video onwards we'll formally start with the principal component analysis exercise from this part we'll formally start with the computation of PCA so we'll start PCS we'll give the variable name as PC and the command is simple this PR comma com command will compute the PCS for us one thing we should remember that you have to scale the data if you do not scale you will get very odd results so scale the data so this PCA variable carries our PCA let's have a look at this PCA data what we call it so these are PCA is that are there the loadings are provided but on its own it's very difficult to make sense of this data so we need to sort of make this data more meaningful for us so the first thing will compute I've been volumes so let's call it yeah I can values for this the command is simple get in the score again and you provide your PC objects there so this will carry your eigenvalues you can print them and now it will make more sense look at the eigenvalues along the each PCA dimension the first PCA has this eigenvalue of 3.74 and it carries 37 percent of your variance the second carry 17 15 and so on so if you want to extract factors at least that can explain 10 percent of a data you would have 4 if you want at least 15 percent then you'll have 3 factors so probably 3 to 4 factors would be a good choice here the also one way would be to visualize the data through a scree plot so you can visualize the data with three plots to examine how variances are stacked up so what we can do is we can already we have sort of splitted the plot so we'll first sort of increase this and clean this plot so we need to clean the previous plot and then only we can draw the fresh so what we'll do is we'll plot PCA or other that would be slightly odd so we'll you want to keep scree plot so we'll so this would be probably a nicer plot so you can see nicely the first factor almost 37 then you can see here 17 15 so it seems 3 to 4 factor choice would be good probably so we have seen the scree plots now now let's start the analysis in a more detail starting with PCA results relating to variables now there are two dimensions either you can relate them to variables or individuals let's start by relating them to variables we can extract the data var get underscore PCA so here we'll put the var similarly if you want you can do it for individual as well all you need to do is just change the command here a little bit and provide it for in variables and so here it is for individuals now the question is what is inside these objects so if you want to know what you can do is you can just print the object you can see it carries correlations coordinates cost to cost to is sort of effective how effectively the dimensions are plotted what is the quality of representation of each dimension in the PCA and same goes for individuals so let's do it one by one so one way is to plot the loadings visualize sort of loadings let's do it so this core plot is very useful not the so these are quality of projection of the so we can use this core plot core plot and here there are two ways to one on the dimension along the lines of variable and is dot core this is useful because this is not correlation if it is otherwise it will create from minus 1 to 1 so we tell it that it's not correlation you can see if you look at individual dimensions your dimension 1 has like seven airlines associated dimension 2 has airline 8 and 4 dimension 3 is sort of giving more this thing to airline 9 and dimension 4 gives to airline 10 so this way we know which airline is getting captured in which dimension if you want you can do the same thing for individuals also so the quality of projection for individuals slightly more clumsy but so you can probably if I zoom it you can see what individual do a better dimension wise quality analysis so let's say you want to see individual dimensions how they explain so dimension wise analysis what you can do is you can use this F vis cos to plot it will give you a nice dimension wise plot so let's say first let it let's go by variable wise so suppose if I pick only 1x because first one the first principle component which explains the maximum if I keep on increasing it let's say 1 2 3 I would be explaining a lot of it you can see and if I keep on increasing that let's say I add 5 dimensions a lot would be explained you can see the amount of explanation is increasing only a small bit left if I do it for 10 everything will be explained so with 10 everything will I can change individual if I do it you can see certain level that explain and I can keep on increasing it so it will be increasing a lot and finally with 10 you get full explanation so with 10 dimensions you get the full explained variables as well as individuals we also saw if for each 1 to 10 PCS how quality of explain each of these dimensions particularly the variables here lines or individuals are explained with adding more and more PCS in the data so with this close this video in this video we will do some advanced data visualization exercises related to PCA so let's start with plotting factor maps factor maps for variables and individuals let's for variables makes more sense to visualize in factor maps let's see how so we can make more sense of these variables with factor map so we need to provide the PCA variable we need to also specify the axis generally it is best to choose the first two axis because these are the principal components that explain the data best so I'm choosing the one first two you can change it later we'll see how to change them you need to specify the color of the very just cost with the variable because it shows the quality of projection of data how good data is represented then you have to specify the gradients generally in this argument you can I randomly choose some colors maybe green at one end of the spectrum in between a little bit yellow and by end of it right so this would give me the color spectrum if you want a little bit this thing to be dispersed in use a repel argument so if I find that these are slightly so it's not that cluttered if it was cluttered I would have used the repel argument but I need not do that so you can see how airlines on these dimensions are very well captured so you can see airline one is here to his hair for years color gives you the quality of representation so the more color means more better projection for example let's say you want to change the dimension what you can do is you can change the dimension let's say I want two and three you can do that you can see how they like but the quality of projection would be lower because in this case what is happening is we are using the lower PCA so you can skip that now we can also do it for individuals for example I'll not do for or maybe for these two I can I'll do for individuals but you can do is you can just change this to individual and in addition to that you can here you can change to individual lastly this so this is good so let's run this and you'll see how individuals are grouped and picked here so you can see individual dimensions here dimension one dimension two and different individuals plotted you can see which individuals are together this exercise is very useful in clustering your target segment but this exercise will be done more in a formal manner with the clustering analysis so we can stop here now for this exercise next we would like to visualize the contributions along the lines of variables and individuals how do we do that so we'd like to visualize contributions for variables and individuals so let's start with the variables so what we can do is we can use our contribution PCA now choice is for variables for now and axis we start with different different say let's say start with first axis and you can see how they are the contribution is so for dimension 1 it is like somewhere around the contributed to 17% for dimension 2 it is around 15% and so on if you keep on increasing the contribution keep on increasing the contribution maybe 2 it increases you can see when you have 2 dimension so quite a lot of so for example are explained in after inclusion of 2 and so on so you can see now if you put it 10 here then everything will be explained so this was our variable so you can identify which is plotted where similarly if you want to know the contributions on individual dimension all you need to do is change this to individual and you can see which individual so you can see with dimension 1 which individuals are captured more similarly if you are 2 dimensions a lot of it would be captured and if I attend dimension then quite a lot of dynamics along the individuals is captured with 10 dimension so this way you can see the contributions finally as a last result I would like to do the clustering exercise as we have discussed already how clustering is done will be doing k-means clustering and in this I either I can go with the 4 clusters so let's do that k equal to 4 and you'll get clusters like these I can zoom it so you can see how clustering is done so these individuals in slightly violet color purple purple color they are there then these individuals here and this so you have 4 clusters these 2 red and green clusters slightly touching each other so if you want to make it more perfect segregation so you can probably give 3 cluster and then clustering would be more sort of segregated as you can see here red green blue so very segregated clusters and so this kind this kind of analysis would help sort of doing target segmentation a lot so with this we did some advanced visualization of PCS in this video we also did the clustering exercise in the next video we'll try to identify what dimensions these PCS are in this exercise we would like to identify the dimensions with some physical meaning time to them for this we have also survey data with us let's load the survey data the survey data we load the data first and then let's have a glimpse of the survey data if you look at the console the survey data can contain some data collected related to historical prices brand perception scores value for money and some feature latest score from the consumers you'd like to associate the dimensions with the hidden piece late in PCA well the reason is we do not know what these animals PCR we understand that these less number of dimension reduce PCS can explain data well but we need to know in order to make some policy related implication we need to know what these dimensions are at least have some shape some thing about this animal so what we'll do we'll see with the survey data let me combine this survey data and in order to extract the PCA factors there's this rotation rotated factors available with us so these are your PCS and along with their loadings what we need to do is we need to combine these PCS along with the survey data so this I'll combine them let's me call the new data frame combine and I need to combine them column wise so see bind data frame this will combine my data survey data along with the factors now let's compute the correlation of the factors and let's interpret them so as you can see here let's start with the first PC 1 and this PC 1 is highly related to the price so we can say PC 1 a lot of PC 1 comes from price VFM is generally correlated with price so we can say that same thing is over spilling here we have a PC 2 PC 2 is predominantly we can see coming from sort of brand side so PC 2 is coming on load loading on brand PC 3 is loading on value for money negative suggest that probably it is on the lower the better on the lower side of it PC 4 is loading on features so we can see the initial four dimensions they are loading heavily and in fact it has a somewhat VFM overlaid correlated with VFM with features and so on so we can now relate these PC dimension PC 1 with price PC 2 with brand PC 3 with value for money and PC 4 with features so we have some idea there so with this we have identified the factors in this video we have identified some more meaningful dimensions such as brand price features and so on so with this we conclude the exercise to summarize this lesson we started the case study implementation by setting the working directory loading the relevant packages and reading the data next we did the exploratory data analysis as part of this exercise we explored and visualized the airlines data next we extracted the principal components and computed the eigenvalues of these pieces this little component analysis suggests that the first four pieces capture thirty seven point five percent seventeen point five percent fifteen point two percent ten point three percent variation the data respectively escape plot and variation as we suggest selection of three or four factors to represent the data correlation plot suggests that PC one captures airlines one to seven airline eight nine ten are captured by PC two three and four respectively PCs we mean principle components we also analyze the data along the individual dimension and with four pieces most of the variation data is explained we also visualize the variables on factor plots with PC 1 and PC 2