Hour of Code with p5.js and PoseNet

[Music] hello there morning welcome to coding trained hour of code special today's gonna be a little bit different I have a feeling that I just livestream myself saying test test by the way if that's the case please let me know but I'm gonna do things a little differently today in the sense that I don't expect that this is a kind of free this is less of a free-form session where that will get edited at chunks later this is going to be a one-hour standalone session so I'm going to jump right into the topic but hello I am Dan I'm a weird person on the internet with a coding channel that has train themed my life is very strange but what can I say I can't complain alright so starting off here sound level is low that's interesting I have a new mic and people told me that the sound level was low before as well um [Music] interesting I I did say test okay I'm going to do this real quick okay I think this should help the sound level and then I'm gonna check one more thing and I apologize to anybody who is now watching this as a recorded archive because if you're watching this as a recorded r-ky you're now watching me debug the sound and the sound is out of sync Oh people are saying sounds good I should never look at the chat I'm gonna I am going to I'm now looking at the coding train the slack channel I am going to snap my fingers and tell me how the sound is right after I snap my fingers so snaps use the word snap finger sound or something like that because everything sounds good alright oh uh hundred mill you know okay maybe I'll change this so people are saying this sound is out of sync by a hundred milliseconds I actually have a setting in here that puts it out of sync on purpose by 100 milliseconds because it was out of sync by a hundred milliseconds in maybe hopefully what was the opposite direction so uh since a bunch of people are saying this I am now going to go to just really kidding I am NOT I start over I'm going to go to the advanced audio properties here and I'm going to turn off the out the 100 milliseconds offset whoever said it was 100 milliseconds okay so I don't know if that changed it or fixed it but I am now going to proceed welcome this is the coding tray an hour of code special I don't know if there's anyone you know I always have this dream that I'll be able to do like these one out on one hour session like this and actually have a classroom of students perhaps in middle school or high school or some other community context tuning in live to code along with me I don't think that's the case right now but if you are a classroom tuning in life and seeing me with my magic mug please let me know say hi in the chat I will try to catch it alright so first what is the hour of code so the hour of code is an initiative put on by organization called code.org you can find a lot of different tutorials here at hour of code comm slash learn I think we've got a very interesting discussion about the pros and cons of Computer Science Education Week an hour of code but I am going to bypass that and just embrace the idea in my own way but I want to say that what I'm going to do today in this hour is I was a intermediate level so if you are someone who has never programmed before I might suggest starting out with a different tutorial you can find many again at that link a tower of code but a couple that I would suggest I mean I'm suggesting one that I'm involved with so I'm not suggesting that somehow my tutorials are any better but I know about them and I know that what that what's in them so I can suggest them ok so one is first I worked with a new education website called Publio on Twitter at pebbly o1 and if you go to our of code comm slash Publio art you will find the tutorial there which has me in the videos but the tutorial actually let me click over to it so here I'm just gonna skip this video you could watch it'll introduce you to the tutorial but if you click here and get started and just let's go hold on that's weird oh there we go okay I think was just taking a minute to load so here I'm at the pebbly Oh hour of code code inspired by artists you can see a bunch of videos here but also there is an inline editor that you can try the code you can follow each video one at a time read these instructions and by the end you can make your own composition and share it so this is this I would recommend for total beginners if you've never programmed before you can check out this Publio hour of code using p5.js alright next up there is also this is several years old now but a tutorial that I worked on in collaboration with other people from the processing foundation is also at oh my goodness I never put a little Miss tweet I didn't put a link to it but if that hello dot processing org so this is an interactive tutorial that will is very similar to the pebbly oh one with just a completely different interface but also this syntax is processing Java syntax it's running in the browser through this sort of the magic of kind of how processing Java code can be translated the JavaScript but that's another story for another time you can check that one out there's also a wonderful youtube channel called stem coding that is from like oh s you is that ohio state university oh my goodness osu stem coding let's take a look at this and christopher obon Oh Ohio State University I was right sure okay so you can check out a stem coding and they have also some nice intermediate level tutorials and probably some beginner ones with accompanying YouTube videos and is terrific okay so that's my quick just like send you off in a different direction if you don't feel like being here right now or maybe that's intermediate level isn't appropriate for you so I'm hoping this will take about an hour so we'll be done by 11 a.m. I might stick around after and do some other usual coding train shinanigans but what I'm going to attempt to build I'm gonna show you how to make this how come this won't play because I hit play so what this is you're seeing here is in the p5 editor I am accessing the webcam I am painting that webcam onto an html5 canvas in the browser and then I am drawing on top of the webcam feed with shapes and I am setting the location of those shapes tracking certain elements of my face so this is what I am going to show you how to do now what do you need in order to follow along and again I'm gonna try to do this a little differently in the sense that I might like type a little bit pause I'm assuming that you're coding with me and you're gonna try something maybe you'll ask a question in the chat we'll see we'll see how this goes all right so the only thing that you need is the online p5.js web editor you can find that at editor dot p5.js org and then I would suggest signing up for an account so I'm not gonna go through signing up for an account right now but you'll just you need an email address um you can I think you can also if you have a Google account already you can sign in through your Google account or if you have something called a github account you can sign through that but go ahead sign up for an account I'll wait and show you more of my magic bug because the coding trade is brought to you by invisibility I bet you you don't need captain disillusion to figure out how I did this trick with my mud you think that I'm a little bit too obsessed with cap dissolute you might be right alright so I'm gonna assume now that you've signed up for the p5.js web editor an account and you'll see now you might not look exactly like this to you in under in this little icon over here there's a Settings button and you can actually click here under under theme there are different themes light dark high contrast I have it on high contrast right now I think that looks tend to look best on on YouTube so so the first thing that I want to do is now if you've never I'm assuming that you come to this having learned a little bit of p5 or JavaScript already but just in case what we have are two functions function set up function draw function set up is calling this function called create canvas and that is making this square that you now see over here on your right or my right or my left right it is 400 by 400 pixels so I'm actually just changed that to 640 480 if I want to be like if I want to like just really live on the edge I couldn't click this Auto refresh button and then as I change things it will actually live update so now the canvas is 640 pixels by 480 and what I want to do also now is I want to hit say I did save I want to just change the name of this so I'm gonna call this coding train clown nose that's well that's basically the whole gist of what I want to do is just get put a clown nose on my nose on the coding train I'm looking for my bell here it is all right here we go now the first thing I want to do is gain access to the camera so the way that I'm going to do that is I'm going to declare a variable I'm gonna use the keyword let I have a whole video about why you sometimes say var why you might say let and why you might say Const in doubt just use let's so I'm gonna say let video so I'm gonna create a variable called video and that video is going to that variable is going to hold on to all of the information about my webcam that's on this computer and then in setup in setup I want to connect to that webcam so I'm gonna say video equals create capture to create capture function conductor yes I guess it should be Rost all in the in the live chat that's going on right now suggest that I should be putting on a conductors hat like a train person okay create capture is the function that will create a capture object that is in video form and now all of a sudden as soon as I add that here you could see suddenly it's asking for my permission so on a website this p5 sketch that I'm programming in the web editor he lives on a web server and I'm looking at it through my web browser a website a web page is not allowed to just connect to your web cam without permission imagine if it was able to you could on your web so you could publish a website that anybody that goes to it you suddenly connect to their camera and grab images and like save them on your server so you're going to want to opt-in here let down what I'm programming here your camera images actually never get sent anywhere I'm just giving permission to the browser to use them on your computer as it runs this sketch so I'm gonna click allow and then we should see down here look it's a green screen this whole thing operates with way too much shadow and the sound is bad in here and someday it's all gonna get better alright so now you can see there it is now the video is below the canvas right there's the canvas there's the video so this create sure actually creates a separate element on the page this over here is like the thing that I'm making this is the code let me zoom back out this is the code that's making the thing so create canvas makes this canvas create capture makes this video capture but what I actually want to do is I want to draw this video in the canvas because the canvas the place that I can put my own creative spin on stuff and change colors and draw shapes and do animation and that sort of thing so there's a little bit of a delay do you see that so I am now going to I'm gonna put my semicolon here I use semicolons I'm a proud semicolon user so why am I making this joke you don't technically you don't need to have put semicolons at the end of your lines of code and I don't know some people you know maybe I this is the thing this this will make me completely insane no semicolon here but a semicolon here I like consistency but you know you shouldn't be like me you should just use semicolons and not use semicolons and enjoy your life enjoy your life not worrying about semicolons okay now I am going to down here use a function called image image is a function that will draw an image to your canvas guess what image I want to draw I want to draw the image from that video so I will say video then I need to say where I want to draw it and where I want to draw it is the top left corner of the canvas which is the location pixel 0 comma 0 and if you're not familiar with coordinate systems again some of those other oh boy there I am a double vision here if you're not familiar with coordinate systems there's other hour of code tutorials that I referred you to will go through that in detail but basically that the top left is coordinates 0 0 there is a horizontal axis called the x-axis which has pixels going from now from 0 to 639 technically there is no 640 a--the pixel if you think about that right if you have 10 if the width is 10 but the first pixel is 0 0 1 2 3 4 5 6 7 8 9 wait oh no but I didn't end right I counted that's so confusing right zero one two three four five six seven eight nine I counted ten things but I ended at nine because I started at zero and actually counting from zero is a thing that happens it's just about all the time in programming so pixel zero two pixels 6:39 on the lung the horizontal axis pixels zero - what is it 479 along the y-axis now I don't need to have double vision of myself so I'm going to say video dot hide here so now I just have I'm creating that video element then I am then I'm hiding it and then in this draw loop this is the thing that's happening over and over again I'm drawing the image to the canvas and I guess at this point I don't need this background function because ultimately the video is sitting there as the background all right so I'm gonna pause and see if you everybody has this working if you're falling along is this working for you are you running into an issue please ask a question in the chat okay might give you like a few minutes here to catch up so mr. awesome size in the chat put intermediate level in quotes I don't I I assume that's some kind of like air quotes sarcasm type of thing and may be that this is like so beginner a yes this is beginner I should have said this is like beginner intermediate like the point of this is to be very beginner friendly and you really don't have to have done anything before but I am NOT starting you know I haven't explained everything every what every parenthesis means how is this what is an a JavaScript object all that sort of stuff alright in working for me I see it's working yes I can count to ten yes I should put my waiting music arm I see that Simon is typing Simon loyal viewer of the coding train you should all check out Simon's YouTube channel but Simon saying I think that is advanced ok great it may be intermediate level the sarcasm there was like this is so advanced like it's not intermediate I don't know all right now ah rayon ass I want to make Instagram filters using this can i yes oh this could have been a great hour of code activity to think about how to make an Instagram filter and in fact at a recent creative coding fest event creative coding fest is an event that's happened in several different cities organized by oh it's really been a half an hour already and hold on I'm just gonna click something here there was a workshop about making Instagram style filters in p5 and in fact I think I could do something like this right now filter gray look at that suddenly I have added another function called filter gray and it has after I've drawn the video it's filtered it and one thing that I want to show you here is if you want to go and you want to just stop watching this video and go off in your own direction which I encourage you to do where you're gonna find out how this stuff works is at the p5.js website now a real point of confusion and I see this constantly over and over and over and over again is what's the difference between p5 and JavaScript am i learning JavaScript am i learning p5 i heard i want to I want to be a web developer I want to be a programmer I want to make art I want to like make data visualization all these types of things what am i learning what should i 'la this is very hard there are no easy answers to this question and the more you try stuff and and andreen and experiment the more these concepts I think will become more comfortable for you but let me try to put it let me try to say to my own words right now what you are seeing right here is JavaScript the programming language here is JavaScript if you're writing this code you are learning JavaScript the keyword function the fact that there's a semicolon the fact that there's a parenthesis here these are all elements of JavaScript and this code is being interpreted in real time by the browser too because it understands JavaScript to show this result over here now JavaScript itself as a language does it just have some magic function in it that draws a video image no once you're in a programming language you need to use some library of functions that are made available to you the web browser provides a library of lots of functions in JavaScript that could do things like video and and and handle buttons and interaction and do behaviors and set Styles all that kind of stuff that's often referred to as vanilla JavaScript or native JavaScript it's it's really native to the browser those functions exist because you're programming javascript in the browser you can also use libraries that other people have written with additional functionality and that's what p5 is the goal of p5 is a JavaScript library to make getting started with learning to program in JavaScript to sketch and make art and be creative and experiment and leave you on a variety of different kinds of paths that's what p5 is ok so here I'm looking at what are the functions available in a p5 I'm gonna go under reference I'm just gonna do a find on the page for filter because if we look at filter now we can see threshold gray opaque invert so let's try some of these well I'm not sure what is opaque who wants to use my microphone now that's interesting okay hold on I don't know what okay but huh opaque actually I don't know what just happened there that was crazy let's listen let's go back to threshold I don't know why suddenly that turned my microphone on and then all this feedback happened oh no no no oh I'm like typing this in totally the wrong place oh what a what up what a world we live in where you can make these kinds of mistakes so by the way what is this this is like a keyword constant that p5 knows about and I'm making a particular kind of capture that is a video and I think by accident like just through some set of coincidences but key word opaque inside of Korea capture made it connect to the microphone which is not at all what I want to do I want to filter opaque huh interesting let's try threshold there we go threshold is a kind of an interesting one right do you notice what's happening with this image every single pixel is now being rendered as either purely white or purely black based on its original brightness and actually standing in front of a green screen you get kind of a nice effect here my shiny sweater maybe this is all my gray hair you're seeing here okay so we could keep going with these but I'm going to stop and I'm going to move on to do something different so in this activity what I want to do is send this video image to something else now I'm kind of like this is why this is intermediate level I'm going to jump leaps and bounds off away from p5 and I'm going to incorporate a library called ml5 I've used ml5 quite a bit on my channel ml5 is another JavaScript library to add additional functionality for you it's built with based on on top of its using the foundation of it is this other library called tensor flow chance so tensor flow j/s is a library an open source library for machine learning machine learning being a field that's kind of you if you haven't heard that term before you've probably heard the term artificial intelligence and these terms are linked but what machine learning can do is you can teach your computer she could teach an algorithm to learn based on some set of data so I can say to the computer here's a lot of cat images now you know about cats here's a lot of dog images now you know about dogs now I'm going to show you a new image can you guess whether it's a cat or a dog that is what machine learning algorithm is a classic example of what a machine learning algorithm can do ml5 comes with a bunch of sort of pre-prepared machine learning results for you to play with and I talked more detail about what I mean by machine learning results it's actually called a machine learning and model in other videos so I'm not gonna go too far into detail but I'm gonna make use of a particular feature that's in in ml5 called pose net so pose net is a machine learning model which allows for real-time human pose estimation in the browser what is human pose estimation it is basically the computer has been trained on a lot of images of people with the corresponding pose information means where's the shoulder where's the elbows where's the feet where are the eyes and you can see some folks here from Google Creative Lab so we just happened over here like I thought but it just I think it was a thing that's lit or whatever so you can see all these these nice folks from Google Creative Lab hello demonstrating that the pose net model is able to look at their image and kind of make guesses as to where their knees are where their feet are that type of thing so if you want to learn more about pose net this article by dan-o vet who is actually a current student here at ITP NYU where I teach and you can see also these folks from the Google Creative Lab who helped put this article together this will give you a lot of background but I'm just gonna go ahead and use it and how am I gonna use it I'm gonna go here to reference I'm gonna go here to actually know it I got a step we got to do one step first hold on we're going to go to let me go back to the ml5 web site and I'm gonna click here under get started so in order to use ml5 we first got to import the library so in the p5 web editor the p5 library is there automatically and available to you you can say create capture you can say image all those functions are there because p5 is available the p5 web editor is not going to know about ml5 that library so in order to tell it I'm going to go and copy this I want to go to that getting started page on ml5 JS org I'm going to do command C to copy this line this script tag this is a reference to the current version actually I'm going to change this actually should be at the time of this recording zero point one point three is the current version of ml five guess we have to update the website so I'm gonna grab this I am going to go back to my sketch now I am going to go up here to this top left where you see this little arrow this arrow shape indicates where I can edit other files beyond just my JavaScript code so my JavaScript code is in this file called sketch j/s but I want to click this and I want to go to index.html so I'm going to go to index.html click on that and you can see this is my index.html now just for a second I'm going to move myself away here and you can see look at all these links these are actually the references to the p5 library this is how the the thing you're coding in the browser knows about the p5 library these are put there automatically for you by the by the online editor itself now I just want to add one more paste it in here and you can see now I'm adding a script tag that refers to the ml5 library and I'm just gonna also change this to 0.3 because I want to use a more recent version libraries code libraries tend to have versions so you can see p5 is currently on version zero point seven point two and ml five is on version zero point one point three incidentally what these numbers mean this is something called semantic versioning these numbers go up as little changes little bugs little things are fixed in the library so if something tiny changed in ml five this new version might be released at 0.1 point for if something more substantial a new a new feature is added or something more substantial changes then this number might change and this number whoa hey something's really going on if that number changes and you'll notice that both ml 5 and P 5 their new libraries experimental in progress open source projects they're both on version 0 essentially but someday you'll see a 1.0 version and that bloom that will mean something will be an exciting moment alright so now I am going to go back up here and close this I hit save I'm gonna close this oh no actually I want to open it cuz I want to get back to sketch J s and then I'm gonna close it and then I'm just gonna everything's still running and I'm gonna bring this back over here so I can see it again so now the ml 5 library should be imported so now that the ml 5 library is imported I want to go here and I'm going to look at the pose net reference page so one thing I think actually I mean maybe by the time you're watching this in the future this will have changed I noticed recently that on the ml 5 website we have these documentation links as well as these example links and there's kind of a lot of redundant information on both and it might make sense for just each feature to have a documentation page with an example rather than be so separate example so I honestly I might like if I click on this example page by the way it's gonna ask to use my camera and we can see there is pose net running so I want to build basically exactly this so you can see here pose net the pose net machine learning model is guessing where my eyes are where my nose is where my ears are where my shoulders are you know in theory it's guess man now it's really designed if I can if I can demonstrate this it's really meant let me turn the camera this way and see if I can go further back it's really meant to work with a full body and you can see it's even having trouble making like a good guesses because it's only seeing me from the waist up has ever seen me from the waist down of the internet apparently but and so if I could reach so if you are using clothes that you probably want to get yourself in a scenario where you can back up and get a sort of full body view to get the sort of full power of what pose net can do but incidentally it's actually pretty accurate of tracking the face there are other libraries JavaScript libraries that can do more sophisticated face tracking with many more points on the face and there's something called I think it's called face api AAS but if i just want to put my clown nose or put on a little hat or something the what pose net is super fast it's gonna find eyes nose ears you can you can do a lot with just that alright so i think what i want to try to do now is go back to the documentation page which has a sort of some very basic code now this is going to be a little bit confusing here actually as this code on the documentation page is not necessarily written oh sorry i'm seeing some stuff in the chat which is a good which is some great questions i want to get to these questions one question was shouldn't i check to make sure ml v loaded properly yes and people are asking questions about Poe's net can you train your own model does it detect multiple faces so these are all really good questions let's actually make sure I'm going to answer those in a second let's let's go with I think was Alvaro suggested I should probably say console dot log ml five just to make sure ml five is loaded so yes so console dot log is a function you can call to log something to the console and this in the p5 web editor is the console down here I kind of want to make this bigger just for a second and you can see here look it logged something if you're seeing all of this that means ml five loaded properly right if I if I were to change this to like ml four it's gonna say this is what you're gonna see if ml five is not loaded properly so let me pause for a second it's ten thirty I have another thirty five to forty minutes of this particular tutorial let me pause that make sure everyone has ml five loaded and then I'll answer some of those other questions I think I'm hearing like some feedback okay turn that off okay now what were the questions so going back to here so first of all can you train your own pose net model now this is actually a really important big question so you can tune out for me for a second if you're trying to get ml5 installed or skip ahead a minute if yours net here is I'm you know I'm not really at the moment in what I'm doing like I'm not really practicing or learning or participating in the full story of the machine learning process I'm just using kind of affection of I'm using the most powerful bit in a way but it's the very last thing that happens in the machine learning process because what I want to do is make some interactive project that tracks my nose if you read this article and you'll find out more about what was the data set that the people who did train the model who participated in that full story use and so there was a data set I I think we can find it loan just give me a second here because I think it's mentioned here in the article yeah you might have to dig into these links to find out more about the actual data set I'm happy to kind of like go down that rabbit hole and provide some resources in this video's description later for that but what you have to realize is that there is a data set of that was used that has people mapped to their poses now what would happen if that data set just for example only had men in it and there were no women in that data set maybe when you again went to use the model that that pre that model that was trained on that data set it wouldn't work very well with women now I don't know if that's the case we'd have to like actually scientifically analyze whether that actually matters so much in terms of this particular model but this is something that you've probably heard about in the news or thought about or read about the fact that why do some machine learning models work with people of different races or ethnicities or genders this is really something that is important to be thoughtful and conscientious about as you're using as algorithms start to make more and more decisions for us in our daily lives now in this case for making an interactive project I find that the I haven't found a really significant limitation or problem with PostNet but you could imagine that if you if you really wanted the most accurate thing ever to train to recognize your particular poses if you trained a model on only you it might work really more accurately than pose net which is trained more generically I'm really going off on a tangent here now so why am I talking about this here's the thing though that process you need a ton of data so maybe it's not in my case it wouldn't be realistic like what if I wanted to train my own pose net model I guess I would have to what would I do I might take thousands hundreds of thousands of pictures of myself or I might go into like a motion capture studio and like put all sorts of like infrared markers on myself I'd have to create this data set of images with all the corresponding pose estimation points pose point there wouldn't be estimations maybe I would just do this manually with a spreadsheet or click on all the points in these images so this is what is sort of valuable and nice about being able to use something like the pre train pose that model in ml5 is I kind of get the results of that but understanding that full story is super important all right now so hopefully everybody has ml5 installed and working anybody have trouble with that please ask in the chat and Halle brine in the chat is making a good point which is you could crowdsource it so crowdsourcing is certainly an excellent way to compile a large data set but you've also got to think about you know how are you crowd sourcing you know are you are you just getting everybody's free labor whatever what is the copyright issues with all of the you know are you allowed to use these images that people are submitting you know are you only getting data from a certain segment people who live in a certain part of the world and you know there's lots of questions or and and I think things to think about in terms of thoughtful and ethical crowdsourcing there okay all right Andre as in the chat says I tried it but it doesn't work so I don't know if that means that your console dot log ml5 gave you an error if that's the case please ask and maybe there are also some moderators in the chat that can help out but I'll try to keep an eye on it okay we got to get moving here I have it even so now here I am oh right why is this confusing so this by the way is written with some slightly different syntax this is written without the p5 library in mind but it also has a video variable so I can actually do excuse me exactly this I'm gonna grab this and I'm gonna I'm going to say let pose net and I'm gonna say now I'm gonna say pose net equals ml v dot i think it's pose net video model ready and let me give myself a little bit more code here model ready is not defined right so this is the line of code now which will basically say I want to create I want to load the pose net model and connect it to this video that's a thing that takes time it actually has to go to a cloud server and retrieve all of the information in that pose net model so this would only work if you're connected to the Internet and also it's giving me an error because what model ready is I mean I could actually like skip that and it's working now working but the reason why I want to add this keyword this this this name model ready here is I want to have a function that tells me this is an event this is a callback it tells me when it's finished loading that model so I can say I could just say console dot log model ready so now as it runs you can see when it loads model dot model ready prints out down here okay so now I have video I have model ready oh no sorry I have video imposed in it both loaded video and post net are both loaded I'm gonna give this a pause to see I'm gonna see if there's any questions I don't see any questions alright so hopefully you have you're following along you have this working the next step is to actually create an event for when pose that has detected a pose wasn't mean so there's a moment it's sitting there pose that is connected to the video they're connected pose was always looking looking at the video and looking at the video right right now there's no pose no pose no pose no pose and a pose pose so I can write a function I'm gonna call it like got poses poses and this is an answer to another question which is can it detect multiple poses and if we go back to this article you can see actually yes pose net can see more than one person at a time now this is a like an added complication that I don't want to have to deal with right now so I'm gonna actually simplify it down to assuming there's only ever one person but you could certainly you'll you'll see some bits and pieces here that would allow you to do that as I continue to code this okay so I'm assuming there's this function called got poses I'm going to say console.log poses and then what I'm gonna say is now I'm gonna let me actually turn off this Auto refresh because it's kind of like reloading it every time I type is a little bit messy for this what I'm doing right now I'm gonna say pose net dot on pose I think this is what I say got poses let's just see if that works okay yes so you can see here what I'm doing this is pretty typical of JavaScript and i'ma hit stop that I create the pose net I load the pose net model by saying ml v dot pose net connected to the video then what I'm saying is hey there's going to be an event this event is a pose event and when a pose is detected the pose that this function got poses will be executed that's the callback and inside this variable called poses will be an array of all of the pose information you can see it's detected three poses two poses two poses right because I'm console logging every time it detects something so in theory maybe there's other kind of events maybe pose net can detect a pose or a something else I think right now probably if we look at the documentation the only current kind of eventually is the pose event but you can see that that this is like a model by which there could be other events as well okay so let's take a look at this now I'm gonna I'm gonna move this back up look this is everything it's detecting amazingly like I am just I am many oh you know I'm many people I don't know why but we can look at this and we could say look two objects right it's detected two objects the first object is a pose with a skeleton let's look in the pose score now this is super important this is super important machine learning models in addition to making some sort of prediction like where's the pose is it a cat or a dog they often also produce all this is done numerically so it can produce a confidence score up probability of it really being the thing it thinks it is so this is actually kind of low right it's only 38% sure that it's a that that it actually detected a pose and then in here inside here key points this is what we really care about so this is the pose that detected this is how confident it was and then look at this these are all these key points what are those key points well guess what here is a nice this this needs to be on the ml5 website but it's not at present but this is a mapping right there the pose net model is trained to find how many are here look that's number 16 how many are here say it with me because start counting from 0 right there are 17 key points let's go back and look here zero through 16 17 key points for example zero is the nose so if I go back here and I look here this is actually the confidence that have found the nose right super confident it found my nose let's see how confident it is it found my leg right it couldn't see my leg like left knee which is 13 so if I go down to 13 right left knee look it's 1% confident - found the knee right because I can't even see the knee so that's kind of interesting so we can see here that's its score for the nose it's part is the nose and then its position this is the pixel location as to where it guessed that nose is so guess what what I'm gonna do now is I'm gonna do something very simple I'm gonna put this down here I am going to make a variable called nose X I'm gonna set it to zero and I make another row called knows why and I set it to zero I am going to draw fill 250 this is a setting a red color then I'm gonna draw a circle at nose X knows why I'm gonna make the circle 100 pixels wide no diet you can see there it is there's my nose and what I'm gonna do now is I'm going to say nose x equals and this is going to be really crazy let's hit stop for a second and let's look at one of these poses okay so poses is an array I want the first pose zero then in here I need to actually go and look I mean let me give myself a little bit more room here I need to go and look at the pose so I'm in zero value to look at the pose so I need to say dot pose then inside of pose I want to look at the key points dot key points and then inside of key points I want zero because I'm looking for the nose and inside zero I know why it's I want the position and inside position looks sorry I don't know why this is keep scrolling I want the X okay dot X and now I want to do the same thing for the Y now let's see what happens let's now run this look at that it is tracking my nose nose I can come back and pick up my nose what Oh hold on oh why do I get an error why do I get an error what just happened I was working that it stopped it's no longer detecting poses I don't know what's going on that was interesting so I think there's an issue here I don't know why I didn't get an error but there's there's definitely a problem this could actually you could see it stopped right here technically this callback could be called when it detects zero poses right it can actually be running and it validates is greater than zero like I want to make sure there's at least one pose so I'm gonna put a little if statement there and do I not have my semicolons I'm gonna put a little if statement there to make sure and I also just gonna comment out this console.log poses because I don't really it's not really that meaningful me any more all right here we go my clown nose is way too big there we go hi it's me clown nose so now you can see I can leave and it I can pick it back up again yeah so this is interesting if I did this way if I was tracking my right hand for example I might be able to like have this idea of like sort of tossing and aside and picking it back up but I can go down here I can enter here really let me come come come to me little nose there we go and actually incidentally notice how it's kind of jiggly so the pose the estimation has had a lot of noise in it it's not gonna be you know and I I move around there's noise in the camera so one thing that I would like to add to this and there's there's a bunch of things I want to add to this but really quickly I am going to now change something and I'm gonna use a function called lurk lurk uh-oh as an hour past know I've got a little bit of time I'm going to write this function here lurk lerp is a function that takes three arguments 1 2 3 this is value 1 this is value 2 and this is a percentage really a number that ranges between 0 and 1 I'm sure you could actually use a number that's less than 0 greater than 1 to get interesting results but conceptually it's important that this number is between 0 & 1 so in other words if I put the number 6 and the number 10 and the number point 5 what this does is it says ok let's think about a number line with 6 here and 10 here and let's find the number that's 50% of the way that's halfway this would actually give us back the value 8 so lerp which stands for linear interpolation is just finding the point in between two other points along the line it's linear interpolation the this has a fancy name but that's all it's doing it's saying I have a number line I have the first number and the second number and I want to be somewhere in between so the reason why I can use this effectively is right now I'm saying knows X equals the actual like pose net knows X value but what I actually could do is I could say you know what don't set this directly equal to it actually lerp it maybe 90% of the way 10 percent of the way 50 percent of the way I'm actually plugged into using a number like point two twenty percent of the way so this will smooth the value instead of having it jump immediately to the new location it just has slides a little bit there so if there's a lot of noise it's going to to stay around the same spot so let's actually now go ahead and use the lerp function and the way that I'm going to do that is I'm going to actually create a separate variable here called X and a separate variable here called Y maybe I'll call it like new new X and new Y and then I can say knows x equals lerp its previous value with the new value 20% of the way there and knows y lerp its previous value with the new value 20% of the way there and now let's run this and see what happens look how it's like it's like blagging but it's smooth and what's fun about this if I move over here and then I can go underneath and then I'm gonna come back over here come to me nose there we go so I don't know what the appropriate value might be like I might actually say like I really just want I want to do it like 50% cuz I just want to like I don't want it to be so laggy but I want to smooth it a little bit so you can see here now maybe it's a little less jittery but the same idea okay so I should pause right and I meant to pause after I just got the nose to see people would get this working so let's see let me pause for a minute and let's see if you can get this working let's see if you have questions hold on I'm have a chat going over here right happy antlers for a holiday-themed coding challenge I think I might do this as like a coding challenge this is meant to be more like a one-hour tutorial activity so I'm mostly gonna stop I'm gonna show you one more thing that I think is kind of important I'm mostly gonna stop here but what but I think I would like to do a coding challenge where I actually like just go through this really quickly and like draw some we're designing thingies design you a technical term for taking this further as draw designing thingies okay so the idea is now you know that you would take this and and you have creative ideas you thought of drawing antlers or a top hat or earrings or you know a mask all sorts of possibilities right Austin writes it would be fun to use this to make a puppet like make a vector character and use Poe's net to control the facial features with your face yes absolutely so you could puppeteer something right there's no reason that you need to even see the video at all you could create a like a physics-based system that's a puppet and actually just almost like a marionette move it around with your body okay so okay seeing lots of great suggestions people are very marveling at my magic cup my amazing magic cup we will show you why it is amazingly magic in this is green here it is somehow purple over here this is very confusing all right all right so here's the thing my one thing about my clown nose is if I'm all the way back let me to move this computer away I'm gonna do one more thing one thing about my clown nose is if I'm all the way back here it's really big on my face and if I'm all the way up here it's really small that's not what a real clown nose would do it would be the same size how could I estimate distance from the camera think about this let's come back to the whiteboard the pose net is us has 17 key points there's eyes nose I believe on the face there's these two eyes nose to ears maybe there's more think about it if this is what it's detecting when it's at a certain distance that might see it this way when it's at another distance it might see it this way everything's going to be closer together so the distance between like the nose and your ear is going to be bigger when you're closer to the camera and smaller when you're further away so why not you have to calculate that too let's go back to here and that's let's use throughout the eyes since it's the next one and so I'm gonna say I guess I need to like lerp all of these things Wow all right let's add let I and I X let I Y and this is technically the like the right the left eye is it the left I remember well where was I left eye is number one so left eye ILX ily set all these values to zero as well so I'm going to add two points for that then I am going to say new of I'm gonna call this NX for nose ex4 eye I don't like what I'm doing here at variable naming is a thing so we've got the nose is key point 0 the eye the left eye is key point 1 and then I can also just lurk the now I have eye LX ILX ily to lerp IL x2 ily with this is NX and x and y hi e x ey so I made this sort of a little overly complex because of this extra lerp step but basically I'm getting now the left eye and the nose and just to be sure about this let's just draw the left eye as well and now you can imagine how I might do glasses right this would be breezy ting I could do a circles with no fill or squares with a line in between so many possibilities that would be a nice exercise for you to try let's just make that blue really quickly so now we should see that I have whoa that's an interesting eye what is going on I must have written I must have something wrong there must be a mistake in my code right il X ily ah look at this I have position X twice line 22 line 22 I see it so let me fix that and let me run it again we should say yes there we go so we can see I have both it's tracking my eye as best it can and my nose if I go really close to the camera you'll see how those are two different distinct distance between nose x knows y IL x and IL y and why don't we just try making the clown nose the diameter of the cloud nose exactly that distance it kind of makes sense right because if you think about it I mean I could use some math to it or a mapping to it but this is my nose and this is my eye and this is the distance this value right being the diameter would be you know something like this covering the nose but not covering the eye so what I could actually BLET so let's actually try that and run it and we will see small nose the nose the clown nose size is proportioned to the distance between my nose and the eye all right so this is this ends this one hour one hour hour of code tutorial we now have what have we done we've we're using the p5 web editor we've used p5 to get the web cam we've loaded the post net model with pose net with ml 5 sorry we've connected the video with pose net we have an event for whenever it detects suppose we find the noses position the eyes position we use lerp to dual add a little bit of smoothing and then in the draw loop we're drawing a circle where the nose is and the circle size is tied is tied to the distance between the nose and the eye so what could you do now so many possibilities right could you put glasses could you put a beard one thing I should say is you should remember that you don't actually have to draw stuff exactly at the position right what if you wanted in my example that I think I showed here at the beginning you know if I were to take this a little further whoops ah I'm in the wrong place if I were to oh come back to me oh this is the actual video of this is so difficult wow those are my socks turn you lose sight you just wanted to get that visual reference I will find it look at all these things on my Twitter there it is so for example there's no key point a forehead or a top of head or left forehead or right forehead but if I know where the eyes are and I know where the nose is I can figure out how to move something up based on those two positions to sit on top of the head so that's something you could think about what is a way that you could create a fun playful mask clown nose hat whatever on top of somebody's face using p5 and ml5 and I'm sure you'll have other creative ideas okay so I just want to a check check the chat for any last djinns oh where can you get this code right now so if I hit save I'm gonna this link this URL you can actually just start from my code itself let me just do this real quick someone could paste this into the chat actually I probably could but you have to be a moderator to paste a link in the chat but this is now the URL when once this gets archived this link will just be in the video's description but this is the put this down here this is the the URL to the code right now cover your nose and see what happens so interestingly enough right this isn't actually doing you know the way machine learning works is a topic for a long discussion but unlike some type of algorithm that is actually maybe looking for like color or some contrast ratio or some type of like edge detection to find where the nose is me putting my hands here it still could kind of estimate the pose it's not the nose it's not getting it super accurate obviously it's jumping around or if I put this train whistle on my nose but it still pretty much got it because it's making a best guess based on all of the training data and the training data is noisy messy weird data with probably like people with covering their nose and yet it still had a nose position perhaps in that training data so it actually still does a pretty reasonable job making a guess okay I thank you for people to me actually I think I can click over here and a woven coast I'm just gonna show one of them woven coasts has now posted that URL all right so Oh Bruno I don't know if you don't mind this Bruno has made crazy eyes well I would love try to make googly eyes right where the eyes like jiggle around as we move your head but let's look at Bruno's if I go to editor I this is going to take me a minute to type it in because I don't have the chat but this shouldn't be too bad capital B why RQL six you je let's see what we get here oh oh so sad what did I get wrong here Bruno ru CH IG ABY r q oh that's a seven let's look at this yes googly eyes oh that is awesome so already I really would love to see lots of examples of this if you don't know what a good what's a good hashtag for this if that you could share on Twitter or maybe or little animations really surprisingly fun is it scaled so you're gonna have to I would say oh no it is scaled oh my no no it's not scaled it's the same size right so that might be something you want to add is scaling that's really fun so please share these well I'm so excited to see things that people make so I I would I'm going to do something a little bit strange I'm gonna continue live-streaming but I want this video to live on its own as this like one hour video so I'm gonna actually turn off the livestream and create a new event on YouTube and then I'm gonna stream to that so I don't know if this is like a bad idea in terms of like losing the audience but I don't that's fine with me go please go about your day but I'm gonna come back and maybe do like one or two other things p5 mask coating faces alright let's do Co decoding faces is good I don't want to say masks because we could do coating pose net but that's when we should do coating pose net well the reason we don't say mask is I don't want to imply that what you make has to be a mask is it I'm sabores asked is it possible to add a 3d mask so the pose net model is making no three guesses about depth this is why I had to extrapolate depth based on the distance between the eye and the nose there I think if you probably look into current there's something called dense pose which is another machine learning model I'm sure there are people actively working on extrapolating the from a to now remember this is a 2d image extrapolating depth from but certainly if I had some type of depth sensor like a Kinect camera or the Intel I think it's called realsense or something camera there are cameras that using infrared can actually measure depth but could you extrapolate that this is hard that you have to have a training set for this right so how do you have a training set that has flat 2d images with all the depth information okay clarify let's try a coding pose net is boring snappy five chat that's kind of something I'm not good at these hash tag things now this videos gonna be like 20 minutes longer as I try to figure out a hash tag code your face I like coding faces actually I mean I don't know if that's accurate but it's sort of fun um you could also just tag at coding train or I think it's I think it's at the coding train or at Schiffman is what I actually see but like coding train faces why don't we could also just do we could do coding train faces is that googly eyes coding face Austin is typing maybe Austin's gonna have a great suggestion here whose videos like way longer just discussing the hash tag [Music] coding phases is cool there already is a little bit of use from May 2015 of coding phases all right we're gonna go with coding phases and I'm gonna shut off this live stream I'm gonna show off this live stream but I am coming right back it's just gonna be I'm gonna have to spin up a new event with a different title I just want this to like stand alone to the next and so we can change our mind later but for now I'm going with coding phases all right so hold on I'm hitting stop streaming goodbye everybody but I'll be right back but this is the end of my hour of code tutorial i hope and i hope to see things that people have made and i hope you found this interesting and useful thank you for watching and good bye [Music]

Channel: The Coding Train

Views: 38,878

Keywords: hour of code, HourOfCode, p5.js, p5, ml5.js, PoseNet, TensorFlow.js, ml5, webcam, interactive video, creative coding, tutorial

Id: EA3-k9mnLHs

Length: 67min 22sec (4042 seconds)

Published: Thu Dec 06 2018