Let's imagine that you work for a major streaming media provider right? So you have I know some 100 million drivers So you've got I don't know ten thousand videos on your site or many more audio files, right so for each user you're gonna have collected information on what they've watched when they've watched it how long they've watched it for whether they Went from this one to this one. Did that work? Was that good for them? And So maybe you've got 30,000 data points per user We're now talking about trillions of data points and your job is to try and predict what someone wants to watch or listen to next best of luck So we've cleaned the data we've transformed our data everything's on the same scale we've joined data sets together The problem is because we've joined data sets together perhaps our data set has got quite large right now or maybe we just work for a company that has a lot a lot of data certainly the General consensus these days is to collect as much data as you can like this isn't always a good idea We what we want remember It's the smallest most compact and useful data set we can otherwise you're just going to be wasting CPU hours or GPU hours training on this wasting time We want to get to the knowledge as quickly as possible And if you can do that with a small amount of data that's going to be great So we've got quite an interesting data set to look at today based on music It's quite common these days when you're building something like a streaming service for example Spotify You might want to have a recommender system This is an idea where you've maybe clustered people who are similar in their tastes, you know what kind of music they're listening to and you know, the attributes of that music and if you know that you can say well this person likes high tempo music So maybe he'd like this track as well. And this is how playlists are generated One of the problems is that you're gonna have to produce Descriptions of the audio on things like tempo and how upbeat they are in order to machine learn on this kind of system Right, and that's what this data sets about. So we've collected a dataset here today. That is Lots and lots of metadata on music tracks right now. These are freely available Tracks and freely available data and put a link in the description if you want to have a look at it yourself I've cleaned it up a bit already because obviously I've been through the process of cleaning and transforming my data So we're gonna load this now this takes quite a long time to do Because there's quite a lot of attributes and quite a lot of instances It's loaded right? How much is this data? Well, we've got 13,500 Observations that's instances, and we've got seven hundred and sixty-two attributes, right? so that means another way of putting this if in sort of machine learning parlance is we've got thirteen thousand instances and 760 features now these features are a combination of things. So let's have a quick look at the columns we're looking at so we can see what this data sets about so names of Music all right, so we've got some 760 features or attributes and you can see there's a lot of slightly meaningless text here But if we look at the top you'll see some actual things that may be familiar to us So we've got the track ID album ID the genre, right? So Jean was an interesting one because maybe we can start to use Some of these audio descriptions to predict what Jean with its music is or something like that things like the track number and the track duration and Then we get on to the actual audio description features. Now. These have been generated by two different libraries the first is called Lib rosa, which is a publicly available library for taking an mp3 and Calculating musical sort of attributes of it What we're trying to do here is represent our data in terms of attributes an mp3 file is not an attribute It's a lot of data. So can we summarize it in some way? Can we calculate by looking at the mp3? What the tempo is what the amplitude is how loud the track is these kind of things this is a kind of thing We're measuring and a lot of these are going to go into a lot of detail down at kind of a waveform level so we have the Lib Roza features first and then if we scroll down After a while we'd get to some echo nest features. Echinus is a company that Produces very interesting features on music and actually these are the features that power Spotify is recommender system and numerous others We've got things like acoustic nurse. How a coup stick does it sound we've got instrumental nurse I'm not convinced that the word speech enos their hat hat to what extent is it speech or not? Speech And then things like tempo how fast is it and valence? How happy does it sound right a track of zero would be quite sad? I guess and a track of one will be really high happy and upbeat and then of course We've got a load of features. I've labeled temporal here and these are going to be based on the actual music data themselves Often when we talk about data reduction We're actually using its dimensionality reduction right well way of thinking about it is we as we started we've been looking at things like attributes and we've been saying what is the Mean or a standard deviation of some attribute on our data but actually when we start to talk about clustering and machine learning We're going to talk a little bit more about dimensions. Now. This is in many ways The number of attributes is the number of dimensions It's just another term for the same thing, but certainly from a machine learning background We refer to a lot of these things as dimensions so you can imagine if you've got some data here So you've got your instances down here and you've got your attributes across here So in this case our music data, we've got each song. So this is puts on one This is on two song three and then all the attributes of a temple echo nest attributes its tempo and things like this These are all dimensions in which this data can vary so they can be different in the first dimension, which is the track ID But they can also down here be different in this dimension Which is for tempo when we say? Some data is seven hundred dimensional What that actually means is it has seven hundred different ways or different attributes in which it can vary and you can imagine that first Of all this is going to get quite big quite quickly My seven hundred a tribute seems like a lot to me Right and depending on what the algorithm you're running is it can get quite slow when you're running Oh this kind of size of data and you can maybe this is a relatively small data set compared to what Spotify might deal with on a daily basis But another way to think about this data is actually points in this space so we have some 700 different attributes that you can vary and when we take a Specific track it sits somewhere in this space So if we were looking at it in just two dimensions You know a track one might be over here and track two over here and track three over here and in three Dimensions track four might be back at the back here. You can imagine the more dimensions We add the further spread out these things are going to get But we can still do all the same things. We can in three dimensions in 700 dimensions. It just takes a little bit longer So one of the problems is that some things like machine learning don't like to have too many dimensions So things like linear regression can get quite slow if you have tens of thousands of attributes or dimensions So remember that perhaps the the default response to anyone collecting data is just deflect it all and worry about it. Later This is a time reporting when you have to worry about it. What we're trying to do is Move any redundant variables if you've got two? Attributes of your music like tempo and valence that turn out to be exactly the same Why are we using Bo for making our problem a little bit harder right now in actual fact echo nests features are pretty good They don't tend to correlate that strongly but you might find where we've collected some data on a big scale actually A lot of it variables are very very similar all the time and you can just remove some of them or combine some of them Together and just make your problem a little bit easier So let's look at this on the music data set and see what we can do So the first thing we can do is we could remove duplicates Ryba sounds like an obvious one and perhaps one that we could also Do during cleaning, but exactly when you do it doesn't really matter as long as you're paying attention what we're going to say is music all equals unique of music all and what that's going to do is look for find any duplicate rows and Remove them the number of rows. We've got will drop by some amount. Let's see thinking It's where you live timer Actually, this is quite a slow process You've got to consider that we're going to look through every single row and try and find any other rows that match Okay, so this is removed a bit about 40 rows So this meant we had some duplicate tracks You can imagine that things might get accidentally added to the database twice or maybe two tracks are actually identical because they were released multiple Times or something like this now what this is doing? The unique function actually finds rows that are exactly the same for every single attribute or every single dimension, of course in practice You might find that you have two versions of the same track, which differ by one second they might have slightly different attributes Hopefully they'll be very very similar. So what we could also do is have a threshold where we said these are too similar They're the same thing. The name is the same. The artist is the same and the audio descriptors are very very similar Maybe we should just remove one of them Well, this is the other thing you could do just for demonstration what we're going to do is focus on just a few of The genres in this data set right just to make things a little bit clearer for visualizations we're going to select just the classical jazz pop and Spoken-word genres, right because these have a good distribution of different amounts in the data set So we're going to run that we're creating a list of genres. We're going to say music is musical Where any time where the genre is in that list of genres we just produced? and that's going to produce a much smaller dataset of 1,600 observations the same number of attributes or dimensions now Normally you would obviously keep most of your data in this is just for a demonstration But removing genres that aren't useful to you for your experiment is a perfectly reasonable way of reducing your data size if that's a problem Assuming they've been labeled right in the first place, right that's on someone else. That's someone else's job Let's imagine but 1,600 is still too long. Now actually computers are getting pretty quick. Maybe 1,600 observations is fine, but Perhaps we want to remove some more The first thing we could do is just chop off the day to half way and keep about half. So let's try that first of all, so we're going to say the first music that's the first few rows of our music is Rows 1 to 835 and all the columns. So we're going to run that and That's even smaller. Right so we can start to whittle down our data. This is not necessarily a good idea We're assuming here that our genre is equally, you know, randomly sampled around our data set. That might not be true You might have all the lock first and then all the pop or something like that If you take the first few, you're just going to get all the rock right depending on what you like That might not be for you So let's plot these on was in the normal data set and you can see that we've got very little spoken word but it is there we have some classical international jazz and pop in sort of roughly the same amount if We plot after we've selected the first 50 you can see we've lost two of the genres like we only have classical International and jazz and there's hardly any jazz. That's not a good idea. So don't do that unless you know that your data is randomized So this is not this is not giving us a good representation of genres if we wanted to predict Jonatha, for example based on the musical features cutting out half the genres seems like an unwise decision So a better thing to do will be to sample randomly from the data set So what we're going to do is we're going to use the sample function to give us 835 random indices into this data and then we're going to use that the index our music data frame instead Alright, that's this line here And hopefully this will give us a better distribution if we plot the original again It looks like this and you can see we've got a broad distribution and then if we plot the randomized version You can see we've still got some spoken. It's actually going up slightly, but the distributions are broadly the same So this is worked exactly how we want So how you select your data? If you're trying to make it a little bit smaller It's very very important and consider but obviously we only had 1,600 here and even the human is whole data set is only 1,300 rows you could imagine that you might have Tens of millions of rows and you've got to think about this before you start just getting rid of them completely Randomized sampling is is a perfectly good way of selecting your data. Obviously, it has a risk that maybe if the distributions of your Genres are a little bit off and maybe you haven't got very much of a certain genre You can't guarantee that the distributions are going to be the same on the way out And if you're trying to predict Jama that's going to be a problem. So perhaps the best approach is stratified sampling This is where we try and maintain the distribution of our classes So for example in this case genre so we could say we all we had 50% Rock 30% pop and 20% spoken and we want to maintain that kind of distribution on the way out Even if we only saw about 50% right? This is a little bit more complicated in our but it can be done And this is a good approach if you want to make absolutely sure with Distributions of your sample data are the same as your original data. We just looked at some ways We can reduce the size of our data set in terms of a number of instances or the number of rows Can we make the number of dimensions or the number of attributes smaller? Because that's often one of the problems and the answer is yes And there's lots of different ways we can do this some more powerful and useful than others One of the ways we can do this is something called correlation analysis so a correlation between two attributes basically tells us that when one of them increases the other one either increases or decreases in General in relation to it. So you might have some data like this. We've actually won And we might have attribute two and they sort of look like this These are the data points for all of our different data obviously We've got a lot of data points and you can see that roughly speaking they kind of increase in this Sort of direction here like this now it might be but if this correlation is very very strong. So basically Attribute to is a copy of attribute one more or less Maybe it doesn't make sense to have attribute two in our data set. Maybe we can remove it without too much of a problem What we can do is something called correlation analysis where we pitch all of the attributes versus all of the other attributes We look for high correlations and we decide Ourselves whether to remove them now, sometimes it's useful just to keep everything in and try not to remove them too early But on the other hand, if you've got a huge amount of data and your correlations are very high This could be one way of doing it. Another option is something called forward or backward attribute selection Now this is the idea that maybe we have a machine learning model or clustering algorithm in mind we can measure the performance of that and then we can remove features and See if the performance remains the same because if it does maybe we didn't need those features so what we could do is we could train our model on let's say a 720 dimensional data set and then we could get a certain level of accuracy and record that then we could try it again by removing One of the dimensions and try on 719 and maybe the accuracy is exactly the same in which case we can say Well, we didn't really need that dimension at all and we can start to whittle down. Are they set this way? Another option is forwards attribute selection this is where we literally train our machine learning on just one of the attributes and then we see what our accuracy is and we keep adding attributes in and Retraining until our Performance plateaus and we can say you know what? We're not gaining anything now by adding more attributes Obviously, there's the question of which order do you tribus in usually bandim? Lee, so what you would do is you would train on all the data for example of a backwards attribute selection You take one out at random If your performance stays the same you can leave it out if your performance gets much worse You put it back in and you don't try that one again And you try a different one and you stole slowly start to take dimensions away and hopefully Whittle Daniel data Let's have a quick look at correlation analysis on this data set you might imagine that if we're calculating features based on the mp3 from Lib rosa or echo nest Maybe they're quite similar a lot of the time and maybe we can remove them Let's have a quick look. So we're just going to focus on one of a set of Lib rosa features just for simplicity So we're going to select only the attributes that contain this chroma kurtosis Field which is one of the attributes that you can calculate using Lib rosa so I'm going to run that we're going to rename them just for a home simplicity to Kurt one Kurt - Kurt 3 and Then we're going to calculate a correlation matrix of each of these different features versus each other like this Ok, finally, we're going to plot this and see what it looks like hopefully we can find some good correlations and we could have candidates for just removing a few of these dimensions if it's redundant and it's not too bad so you can see that we've got for Example Kurt 7 here. So index 7 is fairly similar to 8. That's a correlation of 0.65 Maybe that means that we could remove one over two of those. This one here is 0.5 nine We've got a point four eight over here These are fairly high correlations if you're really stretched for CPU time, or you're worried about a size of your data set This is the kind of thing you could do to remove them Of course, wherever point six five is a strong enough correlation that you want to delete and completely remove one of these dimensions It's really up to you and it's going to depend on your situation one of the reasons that the Correlations aren't quite as hard as you might think is that these libraries have been designed with this in mind if you just if echo Nests just produce 200 feet all exactly the same. It wouldn't be very useful for picking playlists So they've produced 200 features that are widely different. So we're not necessarily going to correlate all the time, right? That's the whole point and that's a really useful feature of this data We've looked at some ways we can try and make our data set a little bit smaller Remember our ultimate goal is a smallest most sort of useful data We can get our hands on right then we can put that into machine learning or clustering and really extract some knowledge The problem is that what we might do may based on correlation analysis or forward backwards attribute selection We might just be deleting data and maybe the correlation wasn't one. It wasn't completely redundant Do we actually want to completely remove this data? Is there another way we can transform our data to make more informed decisions as to what we remove and more effective ones? That's PCA or principal component analysis At the moment. We're just fitting one line through our two-dimensional data There's going to be more principal components later, right? But what we want to do is we want to pick the direction through this data However, many attributes it has that has the most spread. So how do we measure this? Well quite simply
