Get Started with TinyML Webinar

okay I'm Katie from Baxter welcome to the getting started with tiny ml webinar with edge impulse so both cofounders of edenfalls Yann Yann boom and Zack Shelby are here today with us to share a little bit about tiny ml and walk you through some demos of it in action Yana will be doing the presentation today and he'll be running through some demos he's an open source developer and technology evangelist working on tiny amount as the CTO of Edge impulse over the past years he's gone up and down the stack from embedded to web and everything in between Yann is a strong believer of open source and has written thousands of patches for open source projects including Mozilla Firefox OS Jerry script and our friend Ben Zack is here as well he's the CEO of Edge impulse and he will be helping out with any questions you have during the presentation as well as helping to facilitate the Q&A at the end of the first half of our webinar if you haven't been to one before will be a presentation so you're gonna get an introduction to tiny ml followed by a hands-on tutorial so you won't need any hardware today to follow along but if you do have a mobile device you can get started with the platform using that and you can follow along on your mobile the last half of the webinar we will have open for questions so both yawn and Zack will be here to answer any questions that you have if you notice on the bottom of your screen there is a Q&A section where you can just click on that and it asked your question you have any questions throughout the webinar at any point and your questions there we prefer if you write it in the Q&A section rather than in the chat just because the chat sometimes a little bit hard to go through and make sure that we've answered everything so if you have a question please put it in the Q&A and we'll try to get to those by the end of the event today if for some reason we can't get to them all we will post a link to the edge impulse forums as well where you can go and ask any questions that you didn't get answered during today's webinar as an added bonus on this webinar we have edge impulses giving away 20 St IOT Discovery Kids to anyone who attends this event live with us so if you are online right now watching this you are going to be entered into that giveaway and so winners will be receiving an email after the event with more information on how to claim that prize and the great thing about that board is that it's fully supported by edge impulse so you can use what you've learned in today's webinar to get started developing right away with that and there's lots of examples and stuff on edge impulses website as well d you take a look at so with that I'm gonna let yon take over and we'll get started all right thanks so much and see how let's dive straight into this presentation it's it's wonderful to see so many of you here actually to do a webinar without any physical presence which is kind of nice Zak and I are pretty far apart normally as well so we founded a company that is remotes first so I'm in Amsterdam and Zak is in San Francisco so I think the whole world's working remotes actually works in our favor because we're we're used to it already so yeah in the webinar I want to I want to start off with a with a short presentation show you that what tiny mail is and what the relevance is of tiny mail and current markets and then we'll all go straight through demo and we'll actually train a machine learning model from scratch and deploy it back on a small IOT device and so yeah my name is Kjell noon boom you see me here on the left at a an event that I attempt every year which is data science Africa it's a summer school for mostly mathematics and computer science master and PhD students and for me was one of the first times that I I saw the combination in the power of machine learning and the Internet of Things because there's lots of really smart people that know how to find patterns in in raw sensor data it's very often they work on synthesized datasets they just download the datasets from somewhere they get it from somewhere and then they do their analysis over but rather for me as an IT developer at that point it was very much around okay how can I build a device that captures data but after I captured the data I don't really do anything with it anymore and be in one conference with data scientists and myself as someone who's doing the IT side showed the power of harnessing boats sensor data combined weights machine learning algorithms to derive much more value from these deployments so really cool and yeah sack is my co-founder is a CEO of edge impulse there we go see you on camera hopefully and second I met about four years ago a well were working arm one of the largest semiconductor companies in the world and what we saw is that they're more and more more devices at the moments every year there's 40 billion MCU CPUs shipping and most of these CPUs are not going in your your mech books or going into your mobile phones these are very very tiny and small microcontrollers that going in objects we use every day a typical car these days a 60 different processor sitting in them you don't really see them and Meccano you see they're as about six microcontrollers in it already plus two extras for the pedal and and we saw that there's much more and more and more of these same shoes being deployed they're kind of underutilized so about a year ago Zack and I founded a new company at simple you see the mark both on screen and on my t-shirts and to harness and unleash the power of all these sensor devices so if you look at the current state of where we are in 2020 a typical industrial sensor relatively expensive but but really nice high quality these can do amazing things we have now MEMS sensors really tiny and efficient sensors that can measure vibration a few thousand times a second are very precise temperature measurements measurements are waterproof and this one is even explosion-proof it was absolutely wonderful if you deploy it in a factory and and because technologies like Laura went in sick fox we can send data over ten kilometers of range with only 20 foot milliwatts of power it is also beautiful and a really powerful processor capable of running 20 million instructions a seconds but if you if you kind of look at what we actually do with these amazing devices it's very this device as you buy it from the factory the only thing it does is once an hour it sends the average motion what I saw on the accelerometer the peak motion and the current temperature over which is very little if you if you realize that the device can measure vibration at thousand times a second so we throwing away lots and lots of valuable information and only reason is because we need to run these devices off a battery and so we see very often is that 99% of sensor data it's currently discarded true the costs been cleared through power constraints and especially the bandwidth and power constraints are very real for IOT and embedded deployments and because that lots of interesting events that you currently capture already with your devices get lost here's a here's a plot from accelerometer data and if you only use the measurements that the device that I showed earlier takes which is the peak an average motion you're going to lose lots of interesting events so here with the only thing would capture is this little peak because that is the peak motion that we saw on signal but somewhere here in the middle is something that's probably much more interesting because there's deviates from the norm there's something weird going on and very often developers don't even see these kind of deviation these kind of interesting events because we don't have the processing power to actually send this raw data somewhere to a central location those we discard it's a second also single numbers can be really misleading and these are two plots from an accelerometer one where I'm doing up-down movements and another one we're making a circle with my hands with the accelerometer attached and the average motion in this signal actually for the up-down is 3.36 and four circles it's almost the exact same value so by only looking at least this very very limited amount of data we're throwing away lots of really interesting events and it's kind of sad because these amazing devices are so underutilized and so the only real solution here to actually starts if something more with these devices is not to send raw data over the Internet because then the battery of your device will deplete it within a day it's actually put some own device intelligence so the device itself should say okay well I've heard a vibration pattern that have never that I know is going to lead to a fault in about a week from now and or the temperature is varying in a way that I've never seen before or the Machine also lies different than all the other machines in the factory these are the interesting things that I want to know I'm not sending out sensor data to the internet because I'm interested in what the sensor data is I'm much more interested in these conclusions that I can draw from that and if you want to have four devices keep running off the battery without getting an end without getting a really big bill for my telecom provider I need to do this own device and then only thing in its end over is once an hour say hey my temperature varies in a way that I've never seen before that's it right so how do we go from here we know to be one's own device intelligence and we know that our normal sensor data is really messy because we have lots and lots of information an accelerometer might send data a thousand times a second on three different axes it's very hard to program something else to get this information out you know when they see that something varies in a way that they've never seen before now what is really great at finding patterns in messy data machine learning machine is really great if anything he can't reason about in Excel it's really great at trying to fit a curve through your data in a way that captures the very small variations between the different things that you try to detect kind of everything where you can't really draw it out in a graph and try to find this correlation yourself as something more machine learning might do very well but machine learning is also when I when I think about machine learning I think about data centers full of servers doing all the big churning if you've seen the alphago documentary it was on Netflix on how the team at Google actually built a go playing BOTS they had to develop their own custom neural network processor the CPU will steep you and put 5,000 of them in a data center and then the training on that and that's a lot of compute so it seems these two worlds are kind of incompatible on one end we we really really really want to do on the viol own device processing because then we get the interesting information but the thing that is really good at finding patterns in this huge amount of sensor data is something we associate with large data centers full servers now fortunately over the over the last year and a half there's been a really interesting development which is the development of tiny amount so tiny ml is is kind of the father of tiny mal as what's people see it as speed warden was I was working at Google beats together with Neil Tom kind of came to the same conclusion Neil was working at arm in the same group as me at the time and it came to the same conclusion that well these two worlds are not that different after all is just compute and four beats that realization came when he was looking internally in the in Google and I realized that the okay Google Keyword spotting the thing that runs on every single one of your phones is a machine learning model and a machine learning model that doesn't run on a big computes in a big data center no it's machine learning model it runs on the signal processing little chip that sits in your phone that is always on and and that's the reason why I can I can in this case to my phone I could say hey Siri and my phone will turn on even when the rest of my phone is asleep so you still have great battery life and that's amazing and then how do they do that so this tiny non movement is kind of focus on one and focus on inferencing so only on we want a response to events coming in and not about training the training of this model you know having the millions of voices and a millions of keywords that people say to distinguish between between these phrases between okay Google and hey Siri is still down somewhere in the clouds but then the final inferencing that actually running that can happen on very very small devices and we can do that because a machine learning model in essence if you're training it is just trying to find a mathematical function with lots of parameters that best matches the problem at hand that best matches whether I can have a bunch of input data some data from a microphone and match whether I just saw ok Google or something completely different so that training and getting all this little parameters right that takes very long time but after that is just a mathematical function means that at function I can just execute an anything it has a little bit of computes and as long as this function is not too large and that's something actually actually a run on a microcontroller then then from that after we after that the initial premise will solve lots of other companies including arm and started working on ways to make this faster and ways to make this run and even less memory so one of the most importance of things is that we realize if quantization where you trade a little bit of accuracy for a lot less memory because you say well I'm not that interested in all these like very tiny variations I'm mostly looking at like what the global what what the global say of this function is and that helps then we realized that we have very cool like vector extensions sitting in our processors and we can utilize them to make the execution of these functions a lot faster and ways to reduce the number of parameters and making it smaller and smaller and smaller up until the moment where we're really targeting battery-powered mic controllers so it's really made for tiny devices that run off a battery for a year or longer that need to deal with messy sensor data and not a Raspberry Pi that is always powered but are really interesting movements so this is really good for kind of everything that where you have messy high resolution sensor data so recognizing sounds from our mic controller detecting when when it cats is meowing and bio signal analysis where you have for example if a pulse meter sitting on your wrist like like an Apple watch and trying to determine whether you have good sleep or whether you're what are your behavior is healthy compared to others um or in a factory setting detecting abnormal behavior I have a an accelerometer sitting accelerometer gets really messy but nice high resolution sensor data and it's trying to find when something is amiss so not even and it might be even if even a form of anomaly detection so this could be something where I have lots of lots of normal data and when it deviates from the gnorm that is the moment that I'm interested in it and then I want to get a signal and these are even kind of the use cases that we can come up now as people like to say like sign email does not have this one killer use case there's all these like really nice like small use cases where we can apply it and then move up to higher value use cases and but it's also new things that we can start doing one of the most interesting ones for me one of the coolest things that I'm looking forward to and Accenture Fusion right now I can buy a I can buy a sensor to measure vibration but there's all kinds of stuff that is very hard very hard for me to measure if I don't if it comes just by like a normal sensor for it if the door in my apartments opens how do I know that I can I can build a very complex model with lots of different sensors and what's logic in between but what if I can combine a barometer and a humidity sensor together with a temperature sensor and maybe something that measures airflow and a machine learning model will combine all those parameters together and realize okay this was door opening so no longer is one sensor that a readouts but much rather multiple sensors multiple cheap of the Shelf sensors they can start measuring things that are much much harder to program out so and this is just a start this whole field is only a year and a half two years old and we're seeing amazing developments already I'm expecting this to be be even greater in the near future so right so how do we go from zero to model well first of all everything but literally everything in tiny mail starts with raw data raw data at the highest resolution if you can if you currently like a typical IOT device might get data every 10 seconds from a temperature sensor that litter that is not enough that is not enough if you get data from the accelerometer 10 times a seconds probably not enough as well if you're losing so much of the resolution that you have here and if you lose that resolution it means that we can't actually draw a good conclusions out of it maybe in a later point we don't need it as high resolution but always start with raw data get old get raw data and higher solute impossible straight from your device and how we do this for our development boards is that we either support getting data through serial cable or just directly over Wi-Fi but always keep this raw data don't down sample anything when you start collecting and second step is extract meaningful features the National view's is very dependent on your use case and how people see machine learning sometimes is I have lots and lots of data I trow it in one bucket I call a function and the machine learning model will magically find features this works but it also means that your machine learning models get really really big because they can take a very very long time to train and that's something we can't really use on our own our embedded devices for us as much and much and much easier if we already get a little bit of a hint and downscale the data as well because raw day there's lots and lots of raw data in real worlds just three seconds of accelerometer data at a hundred times a second not even a thousand times a second it's really nine under data points and a second of audio data is already 16,000 data points and what is really messy so having a way of cleaning up the data and reducing the number of features it's kind of next step so example of that for audio is you have the input speech signal someone saying yes and then I turned that 16 thousands difference the 16,000 data points that I get in my raw signal and then look at a spectrogram over time and for audio data for example we can go from 32,000 parameters here to 240 parameters after and get much better results after we throw it in a machine learning model so signal processing really key for anything you do on device and we see this as well if we if you look at like a plot of some of the data so here I was someone and almost fist bumping with a accelerometer attached I was doing up-and-down movements and I was ranking a beard and if you plot that raw data you can kind of distinguish the raw data is here on the left you can kind of distinguish between those three classes you see up and down there's much more movements in the in the y-axis bear there's some movements on the z axis because I'm moving with this I drink indeed lots beer and then fist bump is kind of in the middle there's lots of power like here in central location and if I put all of this data and graphing kind of sword guess it and if I have to look at a single little bubble here a single little orange bubble do I know whether it's beer up-down or fist bump no we need much more information feature effects you can make my life a lot easier so on the right we have the output of a feature extraction algorithm where we're gonna look at what is the most significant data in this signal and then remove all the other noise and here it's much easier here that's very this is just very clear clusters of blue green and orange data so if I have a single point I can pretty much determine what it is already straight away from this so a feature extraction very very important I mean an officer that let the computers figure it out so I see kind of three things I want to do one device one is classification what is happening right now do i if i see data from a sensor that it knows in transit is it on a bicycle is it on is it walking or is it currently attached to a car that is interesting anomaly defection is this behavior out of the ordinary if i let's say a train a model on all the normal sounds that i hear in a house and all of sudden I hear something completely different that is something that I'm interested in and lost forecasting like what will happen in the future and all of these things really good machine learning is really good at that there's not a single algorithm that is the best for this I know that neural networks are the hype right now and everyone everyone thinks about neural networks when I think about machine learning them but they're really good at classification but there are many more algorithms and pit and combining that in there in the right thing and picking the right outer animals kind of key after that and very often you might you might want to deploy two different algorithms so the classification has set neural networks are great for anything those classifications but there they have some downside as well why there is take quite a long time to train there pretty heavy and the kind of magic there a black box at somewhere you have a train neural network you found this mathematical formula with all the little parameter students but how do you know how do you actually know what is happening inside and if you're going to deploy the super minions of devices it's kind of nice to know like why didn't machine was making certain decisions for anomaly detection a more traditional machine learning approach like a clustering algorithm typically works really really well an unsupervised learning algorithm we just say we plot all the data that we have and kind of draw circles around it and then on lots of different axes it's kind of a easy visualization and then when new data comes in I just compared it to all the clusters if it's outside it's probably an anomaly if it's insights probably not an anomaly I mean lost if I've if I've deployed my model of extracted features then I want to deploy this back to device I want to take all the signal processing code the neural network and the anomaly detection code and then put it in C++ files or something something that can run on my device in the end and there's also really really good tools already coming out of here so Google stands for flood lights it's probably the most known so it's just a runtime for neural networks on device because signal processing pipeline is really important too for example for vibration data we have been with a nice signal pulsing pipeline our SDKs sitting for audio data you can use mfu C to extract data there's many many different ways of doing that and then loss for all their classical and no awareness you wanna have deployed cd+ there's a couple of copies here so because the device that we typically deploy are very underpowered they don't have a lot of flesh they don't have a lot of processing power available then have a lot of RAM you want to use the vector extensions or the hardware optimizations on your and see you as much as possible to make sure that this runs efficiently now the good thing is that Google and arm and lots of other companies in this in space are doing this already so kind of every every month your machine learning algorithms get faster and running less in less memory kind of automatically just because the amazing work that people are doing and and new processors are coming out so arm is doing the cortex m55 actually will have neural network extensions sitting on the within the processor so they have specialized silicon specialized instructions just to run neural networks much faster so we're only at the start we can do basic relatively small models now but once the the field gets bigger and and we see new silicon come out this won't get faster and better so getting started right I hope you're really excited we're gonna do this right after in the demo how do you how do you get get started and do something well first get some hardware this is it's much much much more fun if you actually look at real sensor data that you've captures and it also shows you like where are the hiatuses is there a problem with your data rate or not with the machine learning model and so well we recommend today is if you want to get a difficut i've laying on my desk here get the STI T Discovery Kids it's a really beautiful piece of hardware if you don't win one today and get it for about 50 bucks is widely available from lots of distributors and get next day delivered it's about an 80 megahertz processor 128 K a ram also Wi-Fi motion sensor microphone so really really fully equipped development boards to capture data and then deploy it back and also will show you bit than any modern smartphone you can use for this on every phone these days you have amazing quality sensors there is a really good accelerometer already on your phone there is already a really good microphone on your phone so as a way of exploring the world of tiny email the world of machine learning and sensor data just take your smartphone and and start working on it or loss if you already have a deaf boards you have something where you can already get data off it's really easy to post some data straight to edge impulse as long as you can get a raw sensor data maybe from the serial Bernard we know that you can get some sensor data off just post it to us and we'll help with that these kind of three options that you have today and hence what kind of impulse comes in so we do not want to be a competitor that tensorflow we are not gonna optimize the way that we have fosters gonna run harm is much better at that but we try to help with everything is hard around that we try to be kind of data science tools for people that are not data scientists so we have an SDK to capture data from your real sensors in real time we have my windows in a way we have a wave like building a data set that's really valuable and track that and share that within your organization then an richer data and generate them out process to doing both signal processing and the learning and the learning phase engineering model and then finally ways of like testing this so actually have real-time device flows and seeing whether the model performs nice and then it finally deployed a back to device so we can give you just optimized C++ code here and flash for both those flows we got really nice end-to-end tutorials both on vibration and audio at docks as impels I see then sitting Yaka one of our one of our guys who train a machine learning model on detecting whether faucets or sink is on in your house really cool it's just a small model you can build and it actually works like straight in your house even when you're locked up because of Coronas so with that I wanna I want to switch over to demo and we're gonna actually build a machine learning model here together so all right so this is edge impulse our homepage so what I need is a a dev board so I got the SDI T discovery kids laying here and I can log in that jumps me to the studio and within the studio you can capture your data build your data sets look at a way of doing signal processing and then finally training session machine learning model because I'm kind of bounce on my desk at this point we're going to train a machine learning model that can distinguish between four different gestures so one is me moving the dev board up and down the other is me moving the with the deaf boards the other one is the Deaf part going over my desk like like it's a snake and the last one is just laying idle and that last one is really important actually because a machine learning model only performs well if it kind of knows what to expect so ear actually having a having some having an idle state in the state where nothing is happening is really key so to get the deaf boards connected I'm plugging it in first and then I opened the daemon so there's a little piece of software that runs on my computer that connects over the serial wire to my deaf boards and then connects to each impulse I don't wanna connect over Wi-Fi right now and then my device shows up here and the devices tab my dev ward so with that I'm kind of ready to start sampling some data from it I did a bit of data acquisition already prior to this webinar but let's start with just some idle data at this point we send a signal to the device to capture some data while I'm not doing anything I'm touching my desk a little bit and it then sends it over here so basically flats a little bit of vibration while I'm working on my desk but that is all fine so that's one class so let's add some more some wave data all right so you already see me waving so you see kind of every weight that I do I can add some up-down data we're moving the deaf board up and down and the idea is that for a kind of an initial model and it's both for data that's coming from accelerometer and from the microphone you want about 10 minutes of data spray around evenly over those four classes I don't will just give you kind of a good idea whether your machine learning models gonna work initially and so if you look at misclassified that so I can change the label to have done data sanity is really important that you start labeling stuff wrongly then you're gonna have a bad time pretty easily but here we go so we have wave here in we see quite a bit of difference between wave and up-down right there's there's lots of noise here because he can't do the movements and generally is is different however the program to write some code to actually distinguish between these two things and especially in the way that many people can do stuff in multiple different ways is gonna be really hard so let's design a we call an impulse or we combine some signal processing and some machine learning to start distinguishing between these so we start there on the Left we take quite long samples so five or ten seconds but my gestures are less than those five or ten seconds so when a slice is up in little windows first so it slices up in to second windows and then he uses sliding window to go over the data so my five second of data I make many many windows of those and that helps a little bit with the amount of data I need to capture and then add a processing block so this I said earlier a signal processing block makes the amount of data that we need to process a lot less and can clean up the data as well and so we shipped with a number of blocks for repetitive motion for audio but you can also plug your own block since we got some tutorials on that so let's start with a spectral analysis feature and then a learning block like a neural network the first thing that I want to do if I'm looking at my data here is that my data is really noisy and all these like little vibrations of a little difference are not beneficial for me because if I'm moving this from left to right yeah there's some noise but I'm not interested in that so first I want to remove the noise and after that I want to kind of look in the signal and see where the interesting frequencies are and so set it up here so the very first thing is D noise the signal so you see in there after filter it's the same signal but D noise and do that by using a low-pass filter and then I do some analysis over the signals so I plots the frequency domain here and plot the spectral power over here and kind of the idea here is that when I compare this way for other wave data even though it might be captured by someone completely different the initial frequency domain and spectral power graphs stay relatively the same so there's almost a little bit of difference but the general movement is kind of the same and when I use it on up down for example the other movement is completely different again and I can play with these parameters over time bits but for now just just go straight forward so let's say I'm kind of happy with my D speed pipeline I cannot train this model by doing that I go over all the data I've in my data sets and then run this feature extraction step and then with that I have about five and a half thousand different windows two-second windows all with a label derived from is ten minutes of data and I can plot all of those in one graph to see if my features actually make any sense so let's take the height of the first peak the height of the height of the first peak and all the three different axes you see that my data is already kind of cleaned up quite nicely so we see a very big cluster here a wave data on the on the writes I see cluster up/down data here I see that snake is definitely different from Idol very similar to other things so by doing by choosing my features wisely I can start distinguishing between my clauses already very and if you see that lots of data is still clustered together and you can't distinguish it nicely yourself a machine learning model will probably also be have a really hard time doing this so you can you can put this data to help you doing it and so with that happy let's let's train a small neural network the the immediate step that you see here because the feature extraction pipeline is that we have lots less features so we have two seconds worth of data but at high resolution so 400 data points or so afterwards we only have 33 features so much smaller we make a very small neural network and let's strain that's but only train is for very very tiny bit you'll see that the network will perform very very poorly at the beginning because we're trying to find this this mathematical function that describes our data sets but at the moment we don't give it any time to adjust as parameter so it kind of things that everything is snake so very poor for accuracy kind of just guessing except for for all snakes so let's train it a little bit extra and then all I'll talk about own device performance so we see also what we expect the only vice performance to be so we think that the inferencing time here will be about 6 milliseconds and we see about big memory usage about 5 K and this is directly correlated to the number of neural network layers that you have here so if you add an extra one this will go up so you have a bit of a feeling of what its gonna be and these are device performance numbers running on the STI to discovery kids that's for a few after a minute or so we have much better performance but maybe a little bit too good we think right now with 100% performance typically if you see that in neural network or a new machine learning model that means it's over fit it knows very well about your training data but it's not very good it's doing stuff in the world and but yeah let's try it it out so let's see if it actually works nice or not so I have my go to life classification and then once again I can get data from my dev board we upload it to examples and we ask it to classify see what is happening so but do me some waving alright that was classify it's it's it took 38 windows 38 to second windows from these five seconds of data and I thought that all of those were waif so it actually works surprisingly well at least on this data naturally you need lots of data to also test whether your machine model works so now let's let's try something with the devout what the model can't do and and neural networks are really bad at encountering data they've never seen before at all so let me just shake vividly so this is nothing like up-down or wave or anything that we've done before we'll see what the network thinks so I thought all of this was up-down and it's very confident about this as well so it thinks it's a hundred percent chance that this is I've done but of course this is not true and if I and if this is a crucial part of my device deployments this means it might draw the wrong conclusions from this so this is terrible actually we can't trust the neural network here now fortunately if we look at the feature Explorer again and I have my F my four classes up down it's blue wave is orange idle is green and snake is red and then I plot my data that I just collected in purple I see that it's very far eyes outside of any known cluster and that's something I can take advantage of so let's add a second machine learning block which is a anomaly detection block or a cluster and then see if there is like any of the data we've seen before so now we have one signal processing block and to learning blocks we just train it on the three axes that we just saw just for clarity and after a few seconds we've we've known all the clusters of our data and so all the dark spots are data that we've seen before in our training set know the the circles are our clusters that we do we trust and when I compare this to the waving that it did earlier this fits nicely within the clusters so this was definitely not an anomaly this is that we we trust it but when I go back to life classification and once again shake vividly will now also run classification on the second book and here even though the network still photos up down we now override it and say well no here we don't trust the neural network is was an anomaly so we need to flag this as an anomaly so really flexible so we got a model now let's let's test it against some data that we've never seen before and see how well it performs so what we do here now if you go over all the data that's we've kind of put the site and it's in a special bucket and then we ask the model to classify this and tell us like what do you think that the accuracy should be and it is not just accuracy of the neural network is the accuracy of the signal processing pipeline the anomaly detection code and then the neural network combines and still doing really really well this is probably better than you could expect normally in the field and the fan-out works well and we've seen it a couple of anomalies are detected and that's something wicked like we had go in and fix because apparently we're missing some training data so from there let's deploy it so the easiest format that we have just deployed as a simple supposed library so this is a library without any dependencies that runs on kind of everything like small small mic controllers so in here we have our SDK which runs on basically everything we have our model parameters which tells like how long the sample how many layers you have and then we use tensorflow here so we give you ten flow light model for just a neural network so really easy to deploy if you have the DEF board sitting here I can use deploys a binary this builds a firmware package specifically for this def board and we can run it on device and test if the inferencing actually works there and then we can also see what the performance sorry so I save this and then I drag this file onto the the deaf boards to flash that's flashing yellow and red it's moments I stopped a demon and in a second when it stops flashing right all right so here we go so this is a view on the device so this is not running in a Jimbo's anymore where we're running without a connection and I can ask my device to run the same inferencing pipeline so I run ATM fills and so we see we have two seconds of sampling time with four classes we're in our DSP which is about 60 milliseconds then we're in the classification and then we were an anomaly so this whole pipeline two seconds of data full analysis with a neural network clock signal processing block an anomaly detection we can do in 18 milliseconds which is really really fast so let me do a movements up and down so we think it's up down now then let me do some wave and other musicians shaking so before it is you saw that we thought it was wave a hundred percent of the time so quite correctly and then last I did my my shaking so it's still thought it was up down but we override it because we say an anomaly score of three which is much higher than we normally tolerate so there we go full machine learning model now running on device if you have now naturally we can do this for much more than just accelerometer data and if you don't have the device itself we can also use our mobile phone to do that so let's go to the keyword spotting project here then I'll take my phone I also I'm just using a my ethnics s and here's a screen of my phone and let's add my phone actually as a as a device to do to get some data for match impulse so only devices tap I can connect a new device use my mobile phone get our QR codes and now my phone is here so phone ak8 j WM 4 h 4 is sitting there so what a heavier is a simple keyword spotting model so we have three classes yes no noise those are the three costs that I want to distinguish between and from here I can now sample data in exactly the same way as I can from my phone it's a good early err from the device so let's add a little bit of noise data and look at my phone on the left here alright so this is a little bit of noise data you can't hear it right now and I can add some yes sample here as well yes all right so now we have yes sir now very similarly we have a signal processing pipeline that consists of a way windowing the data so we look at a second word of data then we have an audio block to expect some features there and then another network so or audio block looks yeah we look at capsule coefficients here so you see for example no you see lots of energy here in the middle and then if you look at for example some noise data this is a guy meowing it's lots more stuff in the higher frequencies and then we have a small middle network a convolutional neural network here so we make this network is a lot bigger than what we had previously we can still run it on device so only vice performance here we think about 200 milliseconds for in second word of audio data so that's still a real time enough and a big memory used to about 34 K I'm so very trained that model and then when I want to use this on my phone I can click on the switch the classification melts what we then do is instead of building our signal processing and neural network libraries for the discovery kits we actually make a web assembly package and then push it to the phone so I'll switch then I go to classification mouths yes no circle misclassify let's know there should have told me when but yeah so here very easy built your neural network builds your signal processing pipeline put it on device whether it's your IT discovery kit or whatever it's your phone sitting here in a very very trivial manner just literally go take something that can get sensor data collected build this all data set out and then redeploy it back onto something that can run in real time on your devices so yeah last thing like really the only way that's that this makes sense then we'll switch to Q&A it's like get started yourself it's really cool like having sensor data in your hand and I'm training a machine learning model to run this on device is super super super cool and it's really easy to get started yourself and you really feel the power like for me running my first ml algorithm that was doing gesture detection on a real phone was from mica or a real device was kind of like the hello world again hello world in what I did when I was ten years old and doing Visual Basic 6 you get the power back into your back into your hands and so yeah go to edge impulse to get started so you can use your phone your IT discovery kids also one of my colleagues Dan brought together with Pete worden this book the tiny mouth book.com which is really great five hundred pages full of resources on a wide make sense what use kids are etc so really going it's are yourself so to recap the ml hype is really real it's very powerful technology and there's lots of amazing companies that are working really really hard in bringing this to any device and it's a perfect fit with sensors sensor data is raw and messy we throw lots and lots of stuff away combining some own device machine learning with all these sensors that is a really perfect fit and so literally start using the remaining 99% of your sensor data and if you do it on a impulse that's fantastic and we really want to hear about it so with that thank you know when I have to last 10 minutes for Q&A and it slides you can find a way it's a new one let's go everybody's out here so thank you for for the attention and the the awesome comments and questions on live during yawns presentation I've been busy typing furiously on my keyboard answering questions during the presentation so we've got 25 questions we've already answered please do take a look through existing questions now we've got 52 open questions and and what I'm gonna do is just I'm gonna pick out some some some good questions here and we'll talk about those with yan and one that's come up really often is about portability we've had some people ask could this run on an ESP 32 will we be doing new boards etc and young why don't you give a few words about that porting yeah and in fairness do yeah so we try to make it as easy as possible to do the porting here so we have a few libraries on that you need own device to catch your data to make it a little bit easier those are all written in very portable code so either C 99 or C++ 11 so you can run those pretty straight forward kind of the weight if you get as long as you can get data from a sensor interface into something maybe an SD card or of a serial you can get the data into a gym tools if you didn't want to run the machine learning model on device the only thing you need is a is a is a modern C full post compiler now for some part I'm not too sure I'm not too familiar with speed 32 but for some parts like running a small neural network really fast it's really nice to use the vector extensions on the device so we made extensions for run on any arm device to make it run really fast for example if a floating-point unit will use that to make it faster Darry might do need to do a little bit of work we'd love to support you in that as well and well we'll have some first party support of new boards coming out as well including aboard from attic compute which were really really cool low-power parts you know we're working on a new cool parts that lots people have come outs later she another question that's come up pretty often here it's about power consumption and how low how low power is this and do you need special hardware to achieve battery power yeah it's kind of goes with everything it depends how often you do it so the SD Board for example if we don't do anything so in the little inferencing demo we kind of we require it for two seconds and every sample and then we were in the machine learning algorithm you can tune this very nicely so that the board already goes about 50 micro micro amps yeah point zero zero five milliamps and in deep sleep mode automatically when you're not doing anything then when you capture data from the sensor you can do it at very at a low power setting and they only need to burst the moment if you run your machine learning model and that is in this guy is only ten milliseconds every seconds so you still need to do a bit of duty cycling but it's much much much more efficient than driving the radio so the guys at farm for example we're saying that they could do still in detecting objects is still images on a coin sell for about a year every five seconds before the battery was run outs so this is pretty amazing actually anything with the meta systems nothing is nothing happens magically automatically you still need to think about these are measurements you can go really really low with this kind of use case another great question was about being able to open up a model and a data set in a data science tool like a Python notebook or stage maker and then can you bring it back in again good question yeah so we have a couple of extension points so we don't lock any data in so any data we generate from the output of your signal processing pipeline to the tensor flow models that we generate you can just pull outs but also if you rather work with your own data science tools from the neural network block you can just go to the advanced menu say editors i-5 notebook and it will give you a Python notebook with all the data already pulled in from our API or you can start experimenting with different architectures data sanitation etc to bring it back we put all the interesting stuff for us in a single cell in the ipython notebook and as long as you keep everything there you can just copy and paste that back into a gym pulse and it keeps the whole flow in that so definitely definitely a way for how we're doing on time q got five minutes left okay so in a similar light quite a few people asked about specialized algorithms like can I do principle component analysis or what I have what if I have some really specialized data like ultrasound radar for example do our signal processing blocks in neural network machine learning blocks work for that yes and a little yeah most mostly yes so you can if you have a sensor if you have an awful sensor for example something with ultrasounds you can plug in your own processing blocks in any language and we have some examples available in Python into edge impulse so then we can do the feature extraction from there yeah however we don't have automatic like code conversion to run his own device then so it requires a little bit of work there but you can do the initial full onboarding on that sensor data from its PCA at this point we don't do image impulse but it's something I want to bring in very very soon and there's a couple of other really novel really novel techniques in reducing the number of features and some recent papers so that we're also gonna gonna look at and see what makes most sense from a engineering standpoint but yeah even if even if that's kind of the end point for examples you can always use it for data collection initial feature extraction and yours take the output there as an umpire Amen pie file is imported into your own pool and and keep working on that so very extensively yeah it's important to remember that we don't make all the algorithms the place that hosts and makes it all work so you can always implement a custom algorithm on it gem Holmes um assume the question came up about conventional ML techniques versus neural networks and what we think about that and do we support classical mo yeah yeah this point could do a couple of class canal blocks so anomaly detection is there is the most clear one based on clustering will add more depending on what we see is necessary what I really like about classic ml is that very often it is I can explain what happens with neural networks it's it's kind of hard if you're going to deploy a model and even on hundreds of thousands of devices that go into the wild and that then need to run without any supervision it's kind of scary to say here's a black box yes well yes it's gonna work so for us we always deploy it with a classical l ml over on the sides just sooo that's kind of a safety safety guard or something and no elaborate might do it better we like the certainty of things and actually know when something goes awry and so you can flag that and send it back so there's definitely a place for this in there great last one I think last one yeah so had quite a few people asking about cameras and will we do anything with images and you know is the same technique of tiny amount even applicable to images and you know where where does the boundary as well yeah video no not at this point at least still images not at among the temples but definitely something that we'll be adding in the coming year it also depends a bit what you want to do so if you want to do classification of ten different types of objects in still images yeah totally gonna work if you want to run full image net on your on your device not gonna work do full video full resolution video also not gonna work so you need to work a little bit with with the constraints that you have for still images stuff like the counting for example animal spotting definitely something that's that's going to be the pipeline yeah and keep in mind that you can run these algorithms also on higher powered computer there's nothing that limits this to MCU you could deploy on a cortex a a Raspberry Pi and you can deploy larger models and our infrastructure will support that so so there is there is a little bit beyond tiny amount that you can use this and there are also are great tools for full rate video and and facial detection and things like that so so we're not trying to replace those were trying to bring ml into all the other applications and low-power devices thank you everyone for attending um great session Thank You hackster and kv-4 for hosting us yeah Thank You Jana and Zach and thank you everyone this has been amazing there's so much engagement I had posted earlier the link to the forums and so if you didn't get your question answered you can always jump over to the I will also include that in the follow-up email that will be sent in in the next day or two with the recording as long as as well as the slides that Yan presented today you guys will all get that and be able to watch through this a couple more times and get some of those questions answered thank you everyone for joining us whatever time in the day it was I know a lot of you were late at night so we appreciate you staying up late for us and participating thanks back in town have a great day ready right

Channel: Hackster.io

Views: 6,119

Keywords: edge implulse, hackster, tinyml, machine learning, stmicroelectronics, stm32, real-tim sensors, machine laerning on microcontrollers, tensorflow, artificial intelligence, zach shelby, jan jongboom

Id: 7OCIm_gh55s

Length: 59min 42sec (3582 seconds)

Published: Fri Apr 10 2020