Matplotlib Tutorial (Part 9): Plotting Live Data in Real-Time

hey there how's it going everybody in this video we're going to be plotting data that's continuously being generated in real time so these real-time plots would be great for plotting data that is changing frequently that you want to monitor so for example I see a lot of examples on certain sites that will you know plot things like real-time Youtube subscriber counts or or maybe you're reading in data from a sensor and want that immediate feedback no matter what you're doing you're likely going to find a use for something like this so first we're going to look at a basic example where I plot some data that's directly in my Python script so that we can wrap our heads around how this is working and then we'll look at a real world example where we're actually monitoring a CSV file now if you're pulling data from a real time API or a sensor of some kind then it's pretty common to write those results to a CSV file so we'll monitor that for changes and make updates to our plot when there's new data available now I would like to mention that we do have a sponsor for this series of videos and that is brilliant org so I really want to thank brilliant for sponsoring the series and it would be great if you all could check them out using the link in the description section below and support our sponsors and I'll talk more about their services in just a bit so with that said let's go ahead and get started okay so I've got some sample code pulled up here in my script so first we'll look at these real-time plots using this list of data directly in my script and then we'll look at a real world example with data that I'll load in from a CSV file now if you've been following along with the series then you'll likely recognize the other matplotlib code that I have here at the moment but if not let me go over this real quick just in case okay so I have a few imports here at the top I'm importing the random module the count function from itertools we'll go over how we're using those here in just a second I'm importing pandas here we're importing pipe lot for matplotlib we are setting our plot styles to use the 5:38 style just so these look a little nicer now this entire section here is just the data that we're going to be plotting just to get our heads wrapped around how this is working right now we'll look at that here in a second and finally here at the bottom we are doing PLT dot tight layout to add some automatic padding to our plots and PLT dot show we'll just show us our plots and as usual I'm gonna have a link in the description section below to all of this code here if you would like to download that or a copy and paste it into your editor so that you can follow along with this video okay so first off we're used to using static plots that we've already seen in this series before so for example I have an X and a wide list here with values and I'm plotting this with the PLT dot plot method which is just going to make a basic line chart so if I run that then we can see that works so that is pretty simple but now let me delete the current data that we have in these lists and I'll uncomment out this function that I have right here and explain what our desired outcome is so let me remove PLT dot plot and for our x and y values I'm just going to set these to be empty lists and now let me uncomment out our index and our function here so for anyone who has never used the count function from the itertools library basically by default it just counts up one number at a time and each time we get the next value so I have a function here called animate and with this within this function we are appending to our X values list and also our Y values list now X values is just going to append a value that is just going to count up by one so it's just going to be sequential and Y values is appending a random number here between 1 and 5 or between 0 and 5 so let's say that we wanted to run this function every second and plot these values that are getting appended to our list well how would we do that so to do that we can use the func animation class from the matplotlib animation module so let me import that and we'll take a look at how this is done so below our other imports here I'm going to say from mat plot Lib dot animation import func animation and now down here below our function let me make sure I spelled all that correctly yep so down here below our animate function let's tell matplotlib that we want to run that function on a specific interval so we can plot that new data so to do this I can simply say create a variable called any and I'll say an e is equal to func animation and now we need to pass in a few arguments so first we need to pass in the figure that we want to animate now we'll learn more about figures in the next tutorial when we learn about subplots but basically we just need to pass in the figure from our current plot and to do that we can just say PLT dot G C F which is get current figure okay and next we need to pass in the function that we want to run for our animation so that was the animate function and also the interval for how often we want to run this function the interval is in milliseconds so if you want one second then you can pass in 1000 and I think one second would be good for us in this video but you can change that depending on your data so I'm going to pass in an interval equal to 1000 which is one second okay so now matplotlib is going to run this animate function every second but currently we aren't plotting the data so within the animate function we're going to plot our X and our Y lists so I'm going to say PLT dot plot and we will plot out those X values and we will also plot out those Y values now if we run it like this then it will somewhat work but there's going to be one big problem so let me run this and I'll show you what that problem is so if we run this we can see that we get an updating chart but you can see here that this is weird we're getting all kinds of different colors here so let me close this and I'll explain what's going on here so the reason it's doing that is because our plot method is actually plotting a brand new line every time but it's not clearing out the old lines so there are actually multiple lines getting stacked on top of each other there but they're just being covered up so you can't really tell so one way we could solve this is to simply clear out our axis and if we do that then every time it plots that new line from scratch we won't have that issue with the different colored lines because it's always going to plot the same color so declare the axis we can simply run the CL a method which I'm assuming stands for clear axis so I'm going to put that right above our plot so I'm gonna say PLT dot CL a and now if I run this then we can see that this looks better it's updating our plot with those random values each time our animate function is run okay so that's nice but using random data directly within our animate function doesn't really give us an idea of how this would help us plot real-world data that's coming from an outside source so let's take a look at how we would do that so to do this I'm going to plot data that's from a CSV and this CSV is going to be constantly updated by an outside source now this source could be data that you pulled down from an online API and put it into a CSV file it could be data that you're saving from a sensor anything like that it's very common to save data from sources like that to a CSV file now the source for my CSV data is going to be a simple Python script that's going to be continuously adding values but it doesn't really have much to do this with this video so I'm not going to go into a lot of detail how that's adding that data but if you'd like to see how I'm doing this then I can open this script up really quick and just give a quick overview so let me go over this really quick and show how I'm writing random data to a CSV file in real time so I'm importing the CSV module random and time we're setting some starting values here of 0 and then total 1 and total to equal to a thousand and then we're just using the built-in CSB module for this stuff so the field names here are going to be the headers for the CSV file we are opening that up and writing those headers and then we're saying while true so this is just going to continue Lissie continuously run we are opening up that data in a pinned mode so it's going to keep appending to the CSV and we are creating a dictionary writer and we are writing out this info here and the info is all of the data for those headers so we're writing that row and then we're also printing this out to the console so that we can get some real-time feedback in the console as well and then lastly we're updating the values so I'm just incrementing by X values by one every time now your x values could be anything they could be dates they could be you know anything and now for total one we are saying total one is equal to its old value plus a random integer between negative six and eight so I just wanted some variation there it can either go down or it can go up but it's got a better chance of going up total to is equal to total twos old value at plus or minus negative five to six so it can also go down or up but it's more likely to go up as well but there's also not as much variance in these random data's here so that'll just kind of mix up the data for those two totals there so that's how I'm writing in real time to our data CSV file so that's the live data that we're going to be monitoring but again that can be live data from any source it doesn't matter that it's coming from a Python script it could be anything so now let's see how to do this so I've got some code commented out down here at the bottom and this just reads in all of the data from that CSV file that we're going to be monitoring and if you've been following along with this series then we've done this several times but in just in case let me uncomment out this and I will show how to do this so first I'm going to put this into our animate function and then I will describe what's going on here so I'm just going to replace these old values that we were using just to do some tests there and now our new values here we are reading in data see s V and that's going to be the data csv file that is getting live data written to it on a continuous basis for our x value we are grabbing the x value row so that's going to be all of the data for the x values in that csv same for y1 we are just grabbing the total one for y2 so we're gonna have two lines on this plot that we are monitoring that is going to be the total two column so now since this is going to run that animate function every second that means that it's also reading in the data from that csv every second now we can simply plot that out and when we plot this out let's also give these labels so that we know which one is which using a legend so let's pretend that these are you know youtube subscriber counts and we're plotting the live counts of two different channels or something like that so below our clear axis there i'm going to plot out first i'm going to plot out our y1 so pass in X for our x-axis y1 for our y-axis then I'm going to give this a label and I'll just call this channel 1 and now I'm going to copy this and also plot out a line for y2 and I will call this channel 2 now when we clear our axis it's also going to clear our legend so we're going to need to recreate that and I will just do that down here at the bottom by saying PLT dot legend and make sure you actually have that indented here within your animate function now with live data coming in I think it's usually best to specify an exact location for the legend I've had it you know switch spots only several times because the data is being updated and it's a bit distracting because this tries to pick the best location by default but if the data is constantly being updated then it could be switching around on you so I'm just going to set a location here and I'm going to set this equal to the upper left of our plots and you can change that based on your data and so I usually put my tight layout method in here as well so that each time it plots it takes that into consideration so I'm also going to put that here within my animate function as well so I'll do taht look tight layout and run that okay and that should be all we need to monitor our CSV file now this file isn't created yet so let me open up my terminal and I'm gonna run that script that's going to generate data for that CSV file so I've got my terminal pulled up here so now I'm going to say Python and that script was called data gen dot PI so I'm going to run that and when I run that it creates that CSV file and now this is writing data to that CSV file and it's going to continue as long as we don't kill our operation here so if we go back to our script now that CSV is being updated with live data so let's go to the scripts and see if this is picking up those real-time changes so I'm going to run our code here and when I run this you can see that that's already been running for some time and that these I've already taken off now so we can see that it's actually updating as that data is being written to that CSV file now let me close this down for a sec and pull this back up in the terminal and stop this and let me start this back over because it was already kind of far long whenever we checked that last time so now let me run this again and now it's basically just started so we can see that it just started writing data out to this CSV file so it's picking up those live tank changes from that CSV file and plotting that data as it comes out and this can be updated on any interval that you'd like so if your data is only updated every 5 seconds or something like that then you can change your interval so that it's only updating when you expect new data that way you don't have to check every second so I was thinking that in the near future maybe I'll put together a YouTube API tutorial using Python and maybe we can actually see how to build a working version of something like this with real YouTube data that would be a cool little project to do so we can see here I think I gave my channel one a better chance of having better random values and it was behind for the longest time and now it's just constantly been going up okay so we can close this down but it's kind of fun to watch these live plots take off like this so this was a pretty basic animation that we put together here using this live data but there's a lot more that you can do depending on your needs so for example if you needed to run an initialize function for your animation that sets things up one time before your animation first runs and then to do that the func animation class has an an it func argument that you could pass in to do that or if you needed to pass in additional arguments to our animate function then our F or func animation class also has an F args argument to do that so there's a lot more that you can do with this and some of these animations can get pretty complex I've seen some people use these for drawings and math simulations and gravity simulations and all kinds of neat stuff now there is a way that we could have updated our plot without totally clearing out the axis but it takes a bit more code since we'd have to also manually check our x and y limits and things like that but it is possible to not clear out our axis here and just update the lines on the axis with the live data so there's more than one way to do things like this but I thought that this was you know good enough for this video I think that this works well with the amount of data that we were looking at now if anyone wants to see the other more manual way then I can also try to include that in these snippets for this video when I upload those to github so you can probably find those there if I don't forget ok so that is the basics of running these cool little live plots and we can see that now when I run this since we haven't cleared that out for a while that that data CSV file has just been continuously getting updated so every time I run this it's just updating with the newest ones but we still have over let's see I think that updates that every second so we have over 200 seconds of data here already okay so we're just about finished up here but before we end I would like to mention the sponsor of this video and that is brilliant org so in this series we've been learning about matplotlib and how to plot data in python and brilliant would be an excellent way to supplement what you learn here with their hands-on courses they have some excellent courses covering the fundamentals of statistics and these lessons do a deep dive on how to think about and analyze data correctly they even use Python in their statistics courses and will quiz you on how to correctly analyze the data within the language they're guided lessons will challenge you but you also have the ability to get hints or even solutions if you need them it's really tailored towards understanding the material they've also recently released a programming with Python course and they even have a coding environment built into their website so that you can run code directly in the browser and that is a great way to complement watching my tutorials because you can apply what you've learned in their active problem-solving environment and that helps to solidify that knowledge so to support my channel and learn more about brilliant you can go to brilliant org ford slash CMS to sign up for free and also the first 200 people that go to that link will get 20% off the annual premium subscription and you can find that link in the description section below again that's brilliant dot org forge / CMS ok so I think that is going to do it for this video I hope you feel like you got a good idea for how these animations work and how you can use that to plot real time data I definitely think this is one of the coolest things that we can do with matplotlib it opens up a lot of possibilities in terms of how we can plot data from api's or sensors and get immediate feedback like this now in the next video we're going to be going over subplots so so far in this series we've simply been creating our plots with the PI plot object and that works great for what we've done so far but if you want additional plots or sub plots then it's best to create your plots with these sub plots method so definitely be sure to check out that video but if anyone has any questions about what we've covered in this video then feel free to add in the comments section below and I'll do my best to answer those and if you enjoy these tutorials and would like to support them then there are several ways you can do that these ways to simply like the video and give it a thumbs up and also it's a huge help to share these videos with anyone who you think would find them useful and if you have the means you can contribute through patreon and there's a link to that page in the description section below be sure to subscribe for future videos and thank you all for watching you

Channel: Corey Schafer

Views: 227,980

Keywords: matplotlib, python, python matplotlib, data science, data analytics, data visualization, python plotting, python graphing, matplotlib tutorial, python real-time, python real time, python live, live data, live graph, real-time plot, data analysis, matplotlib (software), python (programming language), python tutorial, corey schafer, python programming

Id: Ercd-Ip5PfQ

Length: 20min 34sec (1234 seconds)

Published: Tue Jun 18 2019