Easy Arduino Data Logging and Telemetry

hi everyone I've recently developed this program that I thought I'd share with you guys so a very common thing that I've wanted to do when I'm developing firmware for a Arduino or really any microcontroller or even FPGA is to visualize data of course especially if you're working with sensors I do some robotic stuff and it can be really helpful and so I thought I would share this program and I will show you how to use it and it's also free and open-source so I'll show you how the code is laid out as well so this first demo I've got right here I have my little balancing robot off-screen kind of see here we have live data coming from the we have the accelerometer up here and we have the gyroscope readings over here I have the calculated pitch which is you know if it's the robots leaning over or falling backward here I have the magnetometer readings over here although they have not been calibrated so they're way way out and then I have some statistics down at the bottom and so yeah once again basically if I lean the robot up you can see we now have the pitch of around erection there you go around zero very good and false back it's not like that and so kind of see how the the accelerometer and the gyroscope also kind of react accordingly okay so um that's one and then I know that a lot of my viewers use Arduino and I saw I wanted to show kind of a very simple basic Arduino project as well so here I have a little Arduino Uno clone and I have three things plugged into it I have a light sensor I have a potentiometer and then I have a thermistor and so on the top graph we have the light sensor which is a kind of hidden right here and if I cover it up we kind of see it respond when you walk glide out or if you don't and in the middle we have the potentiometer and so if I rotate potentiometer we can see it respond accordingly and this particular potentiometer has D temps in it so that's kind of why it looks a little choppy because it is and then finally at the bottom I have a thermistor and if I put my finger on it and heat it up you can kind of see that hole to drag and drop and then if I take my finger off it will climb back up so I'm going to show how to write the code for the Arduino and then how to use the program and and then finally I'll explain how the program is laid out so if you know how to program in Java you can kind of poke through it and make changes if you want it's all open source all right so here we are at the computer I want to start by writing the code for the Arduino so go ahead and bring up the Arduino IDE and what you can do is you go to file go to examples basics analog read serial and we're going to modify this so I'll go ahead and type it all out and then I will explain what it all means okay so at the beginning of the video I showed you guys the Arduino and you might recall that we had a light sensor a potentiometer and a thermistor attached to it so I'm reading those measurements here and then I want to put that into a single piece of text so I created a C string or a you know character array called text I just made it size 40 because you need it to be big enough but 40 is actually bigger than we need and then we use s printf to basically printf into that string and you tell it you know which string you want to write into and then your format string which is a that's kind of the important part percent D means to write out a number the D is for decimal and so you write out a number and then you have a comma you write out another number you have a comma right at another number and then you have a newline character which is like pressing enter and then these three numbers correspond to three variables here which we read over here and so my my life sensor is sensor one and it is connected to pin 8 0 my potentiometer is sensor 2 and it's connected to pin a1 and then my thermistor sensor 3 and I have it connected to pin a 4 now one thing you don't normally need to do is this part because I have my thermistor arm already mounted on a little 3 pin device I'm actually going to use a couple pins a 3 and a 5 to power the thermistor so I'm using a 3 and I'm driving it high which is been basically providing 5 volts and I'm using pin a 5 and I'm driving it low and that's providing ground so I basically the the thermistor and a resistor form a voltage divider and that's why you need 5 volts and ground but yeah for your circuit you probably won't need to do that part just you know do whatever you need to do for your particular circuit but you'll need to modify you know put all your measurements here and then you know if you have more than three you know add another comma and another percent D and then you know you'd put whatever here you know whatever you have whatever else you have there so that's about it it's actually really simple you can see it sign out ten twenty lines of code go ahead and verify it and it will compile and then we can go ahead and I'm going to go ahead and upload it to the Arduino alright so we see down here that it is done uploading and if you go ahead and open up the serial monitor we will see we now have our text so you can see it's just streaming in live we get we're getting a ton of values the that first number again is my life sensor so if I move my hand in front of it we'll see that it'll get a smaller number in my hand away it'll get bigger that middle number is my potentiometer I'm going to go ahead and turn the potentiometer and you'll see that it will go between zero and ten twenty three and then that this third number is the thermistor and again if I touch it and heat it up with my hand you'll see that number slowly drop and if I let go now it'll come back up so this is what you've got to make sure you get right you need to have your numbers they need to have just the number and then a comma and then a number and a comma number and that's it so on don't have any any other information printed out don't have any you know spaces or anything like that you don't need to just a number a comma a number of comment so it's called a CSV which is comma separated values that's what you're doing here okay so that's really all it is for the Arduino you can see it's really easy not much going on and so if I then open up the the program so this is the program that I have a link to down in the video description and it's a jar file you'll need to have job installed all right if you just search for Java yeah that's all I want an area and if you go to the download link just click here and then download and install Java you probably already have it installed but if you don't you will need Java 8 so make sure you get Java 8 an older version probably will not work okay so um yeah it's a jar file that I have a link down to in the video description and if you just double click it it'll pop right up and so I'll kind of explain how it all works I'll cover these a little bit later let's start here in the bottom right so sample rate is how frequently your Arduino is sending data to the computer and so if we go back to the arduino sketch we have a loop and then we're delaying for 10 milliseconds so in theory we're basically executing this every or a hundred times per second because there's a thousand milliseconds per second now it's a little bit slower than that because we're wasting some time not wasting but we're using some time in the code here but it's roughly a hundred times per second so we can type in a hundred here and you'll see why that might be useful later on you don't actually have to type in a number here it isn't critical um so for the the communication types were using CSVs it's very easy there's also going to be a binary mode that I planned although I'm actually I think I'm going to comment that out in the code that I share because it's kind of broken right now and it's harder to use anyway so on always needs to ask UCS feeds those are the comma comma separated values we went through earlier and then pick whatever comport you want to use my Arduino is on comm eight and then whatever baud rate so I left it at the default around 9600 so I'm going to use 9600 and click connect all right we're connected to the Arduino we now need to tell it on what those comma-separated values are and so the first value which is column number zero that would be basically the first number here which I have a sensor one which I have as my light sensor so you give it a name you can pick color if you want I'm going to go and leave it as red and you can give it a unit um it is in calibrated I'm not going to type in a unit but if you if you had one you could and you could also do a conversion ratio here which I'll cover in just a second and then you click Add and so we've defined what column 0 is we'll go ahead and define column 1 which is our second column which is right here sensor 2 which happens to be a potentiometer and we can give it another color and this one I'm going to go ahead and scale it so I want it to be shown in volts and we know the Arduino has a 10-bit ADC which means the maximum reading will be 10 23 and I have a 5 volt Arduino so basically we're telling it that could take the number and convert it so that you know with the full scale of 10 23 turn that into 5 volts and you click Add and kind of see here is a what that what that all does and then finally we have the thermistor and I'll go ahead and pick that pick the color for that and I'm not going to try to calibrate it in two degrees so I will leave that alone we're not going to do any scaling and there you go and you click done all right so we have defined the data structure and we're now ready to visualize all of our awesome data so umm if you just click and drag anywhere in this grid area so click and then drag and then let go and now you can tell it what you want to put in that region that we just specified so you can do with a couple different or about five different charts on the first one which is the one you saw earlier is the time-domain chart cached and duration is basically how many samples you want to see at a time so this will show us the 1000 most recent and then we can choose to graph one or more values so if I click Add chart you'll see here we have the chart and I moved my hand up on the sensor and you can see that it just kind of jumps all over the place so there you have it it will this the cached version will auto scale and it will scale both the top and the bottom number to kind of fit everything and as you can see when all this stuff goes away it'll feel you'll see all the noise we're jumping up and down by one or two we might pick if I move my hand in front of it again there you go and let's take a look at the other graph type so if I click and drag over here on the time-domain chart the regular one you do the same exact thing so it looks pretty similar on you can see that it will auto scale the top to be roughly identical here but it has the bottom it'll always be around zero it doesn't it won't you know do that and if again with my hand in front of it you can see we get all that motion okay um so the the cached one which is here on the left um I wrote that because it'll let you if you ever want to chart a large amount of data like tens hundreds of thousands of points um use the cached chart because it actually um it slices up the chart into 32 pixel wide strips and it actually caches them it keeps the old data so you kind of see how all this data is just kind of scrolling by and instead of rendering all of it it will just render the new stuff and it will save the old anyway so for an Arduino or for many things you don't need to do that but if you ever want to cache a lot of data you'll find that this type of chart will take a lot of processing our a lot whereas the cached one you can you can do a whole lot more okay so let's look at a few more types of charts we can do a frequency-domain chart and again we have how many samples to look at and what we want to look at and this is doing a Fourier transform and this is the only chart where the sample rate down here actually matters so on if this value is not correct then the frequency measurements down here will be incorrect and let's see if I can I'm going to move my finger in front of the sensor approximately once per second you can kind of see it's not really a clear sinusoid so you're not going to get a good measurement you can kind of see we have a peak right here around to about two second or at two Hertz so every half second and then we have harmonics and mainly noise but this would be useful for like audio applications or stuff like that but right now we're kind of down in the noise can do a histogram which is really useful so right now we have just a few values and there's only a few bars I'm going to go ahead and try to create a nice little variety here there we go so on it's a little more useful for measuring like maybe a voltage rail or anything you can kind of get a feel for the distribution but yeah that's another useful chart on if we look down here at the grid size by default it's 8 by 8 and so it's um you know 8 columns and then 8 rows and you can make that bigger so I'm going to make it 12 by 12 and by D compiled in at not compile time pro by 12 is the biggest that I have currently set although you can change that code if you want and then finally we have statistics so we can we can take a look at you know what is the minimum reading the maximum the mean the standard deviation of all that we have down here alright so um it was kind of the various graph types so that was really all kind of boring I'm only showing one thing at a time we we can certainly do a lot more than that so if you want to you could could graph all three together and then let's see I'll turn the potentiometer watch is the one thing that I I forgot okay so I'm an important thing to keep in mind is when you graph multiple things they should all have the same scale and of course I'm scaling the potentiometer but I'm not scaling the midea they have different scales so the potentiometers in volts and the others are just in the ADC readings so that actually doesn't look very good um yeah let's see the light sensor and thermistor are not scaled and that should look better okay so here we can see if I move my hand in front of the light sensor and then if I if I put my hand on the thermistor you can see that kind of jumping up and then back or down and then back up okay so but yeah it does only make sense to graph those together because either totally unrelated but you might want to do like sensor measurements like an accelerometer all together or gyroscope all together like you saw at the beginning and and yeah so for some of the charts you can graph you know multiple things at once which would be the time-domain charts the others are only one at a time so you just pick from the drop-down box and that's pretty much it it's a really easy to use and yeah down here the reset button if you click that it will just get rid of all your charts and then you can I add a chart back in you can save the layout so I'll just do call it quick tests okay I just noticed that my screen recording software cut out I'm going to go ahead and bring the software back up I'm going to go ahead reconnect and then we'll pick back up rip so yeah we'll go ahead and create some charts again and we'll do a few different types okay so we can save the layout and I'll go ahead and put on the desktop and I'm going to call it quick tests so quick test and then save it and then now if we want to kind of restore this you know you can add a later point in time open up the program go to open layout and go to that file it'll take a couple seconds to connect and there we go so here we are we have the data I'll go ahead with my hand in front of the sensor again we can see it's all all working what all right so yeah it's really easy to work with in fact let me show you so if I go to open layout and I have the one that you saw at the very beginning of the video my balancing robot and I'll go and connect to that now that robot has a bluetooth serial module and that takes about five seconds to connect to so that'll take a little longer but yeah here we go we're back in so I mean I really was specially for you know when yet we have more complex things like this where you have multiple graphs open and you might have different settings for each of them and if pick up the robot you can see that it's all it's all working and slide okay great so I hope you find this software useful there's a link in the description down below to the both the draw file which you can run if you just want to use the software and also the source code if you want to poke around and play with it now that the second half of this video which will be you know starting right now I'm going to go ahead and cover that source code and kind of walk you through where everything would be located so if you if you don't want to program in Java or if you don't really care about the source code you can you can stop here hope you enjoyed the video and share it with anybody that you think might benefit okay so those of us that want to play with the code there will be a link down in the description below for a github repo and I'll kind of show how it's all organized so it's a it's called telemetry viewer and we have a whole bunch of classes right here it all starts in Maine Maine Java and basically in Maine I tell Java to use the the system look and feel so in other words and windows make it look like a Windows program and then I create a main object and then in the main object I just create a window and then in that window we have a Turk's region which would be this the bulk of the area that we're looking at here it's kind of where the grid was and we have a controls region which is a little panel at the bottom with all these buttons all the way down here this is the controls region and then I just stick them in the window I stick you know the charts in the middle the controls at the bottom I make the window maximized and blah blah blah so it's all pretty standard I'm not going to go into too much detail so let's take a look at the charts region again that's the bulk of the window that's where the grid is where all the charts are and it manages you know the grid size and when you're clicking and dragging to place in a chart it stores those grid locations in there and it'll register some event handler so it'll be notified when the like if the user types in a new grid signs over here the chart region will be notified of that change yeah let's not go into too much detail though I this is going to be an hour long video if we go to the controls region same thing we have a whole bunch of buttons and we populate them and we have event handlers so I'll go through one of them like the open layout button which is right here we create the button we add a event listener or an action listener as they call it and so if these are clicks on the button we create a file chooser and if the user picked a file we then get that path and then we call a open layout method in the controller which I'll get to a little bit and so the buttons are similar they do you know similar things and there is um you know fault checking so if you try to set a grid signs of I don't know a it won't crash hopefully and so that's the controls region so I have the whole program organized as a Model View controller so the model contains the data so that's where kind of some hard-coded stuff is located and where we save state and then we we have our data structure definitions are defined in here and that also contains the actual samples from the arduino or whatever microcontroller or even an fpga if you want and then we have the charts and then the middleman between the data which is in the model and the GUI which is what you're looking at is the controller and that's where all of the the logic is so basically the the code for the GUI handles just the GUI parts for the most part and then the controller handles all the logic and the model is all the data and it's a it's a big file it's a probably 909 hundred lines of code I could have broken it apart but I didn't and so at the top we have things related to the grid and then we have things related to the data sets which would be you know your data structure definition and then all the samples that go along with it and then we have our serial port methods to open and close and to receive data we have our chart listener stuff where we can notify code if a new chart has been added or for chart has been removed sample rate default line color sample count so yeah I don't want to go into too much detail but yeah it's all there if you have any questions of course feel free to ask them I'm happy to answer I just want to waste too much time and bore everybody okay so let me show you so if I remove all these charts go to reset so when you click and drag when you let go we have this window pop up called add new chart if we look here we have a new chart window which is that window it's this and so basically it you know it leaves it out on J panel and it queries the controller to figure out what possible chart types we have and then depending on what chart type you picked it'll know a valid range for the duration currently they all let you have really wide range but some things you may only want to have one sample of in the future I plan to add some of those anyway and then we have for church they can take multiple inputs we have checkboxes and for charts that only take one input you have a drop-down box and all that logic is in here so all right and so then when you actually add a chart all charts are a position to win a position chart and it's basically a J panel and all it does really is it's a J panel and it'll manage the resizing and testing to see if in areas occupied so if if I create a chart so if I resize the window that chart resizes accordingly as you would expect and that is what the position chart does for you kind of behind the scenes and then that region occupied code right here is so that if you want to create a new chart you can't overlap so it's a smart enough to to limit you like that and then we have the various chart types so time domain chart cached is the one that you guys saw the most of and it's right here and all the codes in here um this is really kind of an over the complicated one because I really optimized it to reduce CPU usage on the the regular time domain chart which would be these these two they were great for small data sets but when you start wanting to graph you know hundred thousand lines of hundred thousand points or a million points it slows to a crawl even on a modern i7 running at four gigahertz it can't keep up whereas the the chart want or the cached version that i that i wrote it can handle very large data sets and still be very quick because you know as you can see the line chart is kind of scrolling by and there's nothing to re-render the old data you only need to really need to render the new data so i save it all as strips of pixels and anyway so long story short it's it's really complex and convoluted but it's much more efficient and then we have time domain chart which are these two and that uses the j free chart library and most of the work is being done in get time domain shirt we have statistic shirt which is this guy right here and it's basically a J table and then I update that table yeah say area I update that table live and so we have I'm using the Apache math library the Apache Commons math library to do the statistics and basically I calculate them all here and then I have swing update the GUI with the new data okay so let's get going before this gets too long of a video we have the histogram chart which we saw earlier and it's also based off of the J free chart library and where the work is happening here yeah you can kind of read through that and check out the documentation for J free chart to go into details or ask me I don't answer if I am frequency-domain chart is a Fourier transform and it's a line chart based off of the the J free chart library I calculate the Fourier transform here I didn't actually use a FFT I just did a simple easy-to-understand DFT I'll have to optimize that later on but it works what next so the data set is when you define the data structure each one of those CSV columns is a data set and that contains all the information you entered and also the the actual samples that came in from the Arduino or whatever microcontroller so the the CSV column number is here the binary processor is not used yet that's for a future use with a binary packet the column name column called color the unit and the conversion factor is all right here that we saw earlier and basically we have a few methods to to access elements or samples that were stored or just to check how many samples we have and then to access the ones we have to add new ones or to clear the samples this optimizes a foul to ox so that helped speed things up a lot and to do that though when you clear the data set it doesn't actually free any memory I need to add some more code before I can do that without you know risking a nullpointerexception um but in any case with an Arduino you're not really sending a whole lot of data very fast so even if you leave it running overnight you're not going to use a whole lot of memory and that's pretty much it so the listener classes are well they're actually interfaces so for example the serial port listener the the controls region down here needs to know if we lose connection or if we gain account or if the connections of seeds so it will register a serial port listener and then the controller will notify it accordingly and same with the grid changed the chart listener and chart descriptor the binary processor isn't used yet that's part of the binary packet mode that I have commented out because it is really functional yet anyway so there you go that's kind of all the code um I'm happy to go into detail if you have any questions but I don't want to you know record a really long video and scare people away so ah yeah there we have it once again the program is free and open source and I hope you guys enjoyed the program and the video if you have any questions or comments please leave them down below and if you liked the video please share with anybody that you think might benefit from it

Channel: upgrdman

Views: 170,777

Keywords: Arduino, Data Logging, Data Visualization, Telemetry, Charts, Graphs, CSV, Java, Line Chart, Histogram, Fourier Transform, Statistics, JFreeChart, STM32, Gyro, Accelerometer, Magnetometer, CdS Cell, Light Sensor, Potentiometer, Thermistor, Temperature Sensor, MVC, Model View Controller, Open Source, Cross Platform

Id: lFZ26gD7OIE

Length: 34min 31sec (2071 seconds)

Published: Wed Sep 28 2016