AI Pose Estimation with Python and MediaPipe | Plus AI Gym Tracker Project

[Music] what's happening guys my name is nicholas renate and in this video we're going to be taking a look at pose estimation using media pipe and python we'll go through everything you need to do to be able to get up and running and at the end we might just put this into practice and build our very own bicep curl tracker let's take a deep look at what we'll be going through so first up what we're going to be doing is setting up media pipe so media pipe is our core dependency to be able to go and run pose estimation that brings us to step number two actually estimating poses so in this step we'll actually be able to estimate all the different joints and parts within our body then from that point on we'll actually go and extract our joint coordinates so we'll actually be able to grab the coordinate for our elbow for our eyes for our shoulders our hands so on and so forth now once we've got our joint coordinates we'll take a step into step four and this is going to allow us to calculate the angles between our joints so say for example we wanted to calculate the angle between our wrist our elbow and our shoulder what we'll actually be able to do is use a little bit of trigonometry to actually be able to calculate the angle for that particular joint now in order to leverage that we'll actually go on ahead and build a bicep curl tracker so what we'll do is we'll use the angle calculated from our arm to be able to determine once we've done a full wrap and we'll be able to maintain that history to be able to track how many curls we've actually done let's take a look as to how this is all going to fit together so first up as i was saying we're going to be installing media pipe with python and this is pretty straightforward so it's just a pip install so we'll do that for media pipe as well as opencv then what we're going to do is hook into our webcam and actually start making some detections on our machine so this is going to allow us to draw basic poses and then we're going to take it a step further so from there we're actually going to go on ahead and actually start tracking our bicep curls so we'll calculate our joint angles and we'll also implement some logic to calculate once we've done a curl as well as maintaining that state so we can have a running count ready to do it let's get to it alrighty guys so in order to get started with our pose estimation there's going to be five key things that we need to do and as per usual we're going to take this step by step and build up to our final end product so first up we'll install and import our dependencies so there's two key dependencies that we need there so media pipe and opencv then we're going to start off by making some lightweight detection so we're just going to detect our poses and not do anything with them as of yet we'll then start determining our different joints so we'll be able to extract where our elbow is where our wrist is and where all the different joints are within our body then we're going to use that data to be able to go on ahead and calculate angles so we'll be able to calculate the angle between any three points using trigonometry then last but not least with those angles we're going to be able to actually build up a curl counter so if you wanted to use this for a gym tracker device or something along those lines you'd be able to do that there so first up let's kick this thing off by installing and importing our dependencies as i said those two key dependencies opencv and mediapipe so let's go ahead and do it alrighty so those are our dependencies now imported so in order to do that written exclamation mark pip install media pipe and this is going to give us all of our media pipe good stuff so media pipes a really broad sort of pose estimation or machine pre-built machine learning capability library so it's got pose estimation it's got the holistic model which we did a video on previously object detection blocks tracking motion tracking so on and so forth so there's a bunch of stuff that you can do with this then our next dependency is opencv so to install that with an opencv python so the full line is exclamation mark pip install media pipe space and then opencv dash python so those are our two dependencies now installed now the next thing that we need to do is actually import those dependencies so we're going to bring in media pipe and we're going to bring in opencv and we're going to set up instances of those as well okay so those are our key dependencies now imported now what we've gone and done is we've written one two three five lines of code there so if we take a look what we've gone and done is written import cv2 so this is going to import opencv into our notebook then we've brought in mediapipe so to do that written import media pipe as mp so this is going to give us all of our pose estimation libraries and all of our different media pipe solutions so the different components within media pipe are normally referred to as solutions but you'll see that in a sec then we've imported numpy so this is going to help us with some trig later on so import numpy as np and then we've gone and set up media pipe so we've created two new variables so the first one is mp underscore drawing and this is going to give us all of our drawing utilities so when it comes to actually visualizing our poses we're going to be using these drawing utils so the line there is mp underscore drawing equals mp dot solutions dot drawing underscore utils so this is going to give us all of those drawing utilities and then the next variable that we've created is mp underscore pose so this is actually importing our pose estimation model so remember i was saying there's a whole heap of different models or solutions available inside a media pipe so face detection face mesh iris which is iris tracking a hand pose model a pose model holistic so what we're actually doing there is saying that we're grabbing the pose estimation model cool so those are our main dependencies imported now the next thing that we want to do is actually test out a feed from our webcam now this next block of code is going to be pretty consistent throughout each of our four steps so we're going to use this consistent feed to be able to get a real-time video feed from our webcam so once we've got that feed then we can start to build up apply a pose estimation and build out or calculate our angles as well as build our curl counter so let's go ahead and build this feed and then we'll take a look at what we've written okay so that is our feed now done i'm just going to add in a comment so this is going to be a video feed so before we run that let's take a look at what we've written so written one two three four five six seven eight lines of code there now if you've ever seen me do any content on real time estimation or media pipe or specifically the holistic model this particular block of code is going to be really familiar to you so first up what we're doing is we're grabbing or setting up our video capture device now think of your video capture device as anything that you're able to get a video feed from so it could be a webcam it could be a usb microscope it could be some other sort of camera that's connected to your particular machine now in this case the number that i've passed through here is the number from my webcam or the number that actually represents my webcam so you might need to play around with this number ideally if you get also if you do get a bunch of errors when you actually try to run this specifically around like image empty or image must have three dimensions you want to try to play around with that video capture device number so the line that we've written there is cap equals cv2 dot video capture and video is in caps and caps and capture is in caps and then we've passed through the number zero then what we've actually gone and done is we've written another line of code so this is kicking off our loop so we've got a while loop here so while our cap is opened we're then going to do something so this is basically going to loop through our particular feed we're then reading our capture so this is effectively like saying hey give me the current feed from my webcam so to do that we've written cap.read and then we've stored and extracted the variables from that particular line so ret is really just our return variable there's nothing in there that we're actually going to use frame is actually going to give us the image from our webcam so then what we want to do is actually visualize this and this is where cv2.iamshow comes in so cv2 i am show actually gives us a pop-up on our screen that allows us to visualize a particular image so in here or so to that particular function we pass through two variables what we want our box to be called or what we want our frame to be called in this case we've kept it pretty simple so media pipe feed and then we've passed through our frame which is effectively our image from our webcam so all in all these four lines of code are accessing our webcam starting to loop reading our webcam feed and then visualizing what our webcam is actually seeing to our computer everything down below here is really all to do with what do we do once we break out of our feed or what do we do once we clear our feed so this particular line here is checking whether or not we hit q or whether or not we try to close our screen if we do hit one of those keys or if we try to break out of it it kills off our loop using the break statement so basically this is going to break out of this while loop up here now the line that we've written there is if cv2 dot weight key and then we're passing through the number 10 and 0x ff which is checking for what key we've actually hit on our keyboard equals q so ord and then we'll pass through the letter q and then we've got a colon and then we're breaking so this is all to do with breaking out of our feed then if we've broken out of our feed we're going to release our video capture device so cap.release is going to release our webcam and then we're also going to destroy all our windows sorry i'm scrolling around there so to do that we're in cv2 dot destroy or windows so this is going to close down our video feed so effectively we should then be able to kick something off again and not have to deal with any existing windows cool so that is our feed video feed now set up so let's try running that and see what happens so ideally you should get a little python pop up to the bottom here and this is going to be our feed all right so that's looking good so we've got our feed so you can see that that's rendering appropriately so no issues there now right now we don't actually have anything happening there so we want to eventually build up on this and actually start using that feed if we wanted to all we need to do is hit q and that's going to close down that feed and as you saw it's releasing our webcam so i can see it's released and it's also closing down that window so that is our baseline code so we're going to actually copy this onwards and start using that to make some detections so let's go ahead and do that so we're going to copy this and we are going to build up on this so what we're now going to do is actually start applying our media pipe components so let's go ahead and do that and then we'll take a look at our new lines of code okay so that's our first line of code done so i want to take this one step by step so what we're doing here is we're setting up a new instance of our media pipe feed so to do that we're in with mp underscore pose dot pose so this is actually accessing our pose estimation model and then we've passed through two keyword arguments so min underscore detection underscore confidence equals 50 so this is saying what we want our detection confidence to be and then we're also specifying our tracking confidence so this is maintaining our state so min underscore tracking underscore confidence equals 0.5 so the core thing to know for those two particular metrics are if you want a more accurate model or if you want to be tighter with your detection so if your particular webcam doesn't sense something and you want it to be really specific you can bump these metrics up so if you want really high confidence you can increase that if you're not as concerned as having super accurate detections you can drop that down now there is a bit of a trade-off because ideally you want to be able to maintain your state and actually see your detections but if you set this to a really high level of accuracy or confidence you're not always going to get detections if your model doesn't detect a perfect body so in this case it's a bit of a trade-off i found the 50 tends to work pretty well so then what we're going to do is leverage that as the variable pose so this particular line is then going to be accessible via the variable post so the full line is with mp underscore pose dot pose and then we're passing through our two arguments so min detection confidence and min tracking confidence and then we're going to be leveraging it as pose so we can now work with this particular line as the variable pose let's go ahead and finish this out and then we'll start seeing some detections alrighty before we go any further let's take a step back and take a look at what our next components are so i'm going to break this down into three sections so first up what we're doing is we're recoloring our image so recolor image and the reason that we do this is because when we pass our image to media pipe we want our image to be in the format of rgb now when we get a feed using opencv by default our image feed is going to be in the format of bgr so if you think of that an image has three different layers or three different arrays one is going to be blue one is going to be green and one is going to be red now what we want to do is just reorder those different color arrays so that we're passing it through to media pipe in the right order so to do that first up we've written image equals cv2 dot cvt color and then what we're doing is we're grabbing this frame here so remember our frame is our feed from our webcam and then we're recoloring it or we're reordering it so cv2 dot color bgr to rgb so this is just reordering those color arrays then what we're doing is we're applying some performance tuning by setting whether or not our image is writable to false we're basically going to save a bunch of memory once we pass this to our pose estimation model so then we've written image dot flags dot writable equals false so these two lines are all about recoloring our image and setting whether or not it's writable equal to false then this line over here is the most important line so this is actually going on ahead and making our detection so make detection so you can see that we're accessing our pose model which is what we set up over here so you can see that we set our pose over there we're using pose down here then we're storing our results in an array so we're using our pose model so pose dot process our image so by processing it we're going to get our detections back and then we're going to store those detections inside of a variable called results then we're setting our image writable status back to true so image dot flags dot writable equals true and then we're going and recoloring it back to bgr because what we're going to do in a sec is re-render it back using opencv and again opencv wants its image in bgr format so we're going to do that so we're going to say this line is recoloring back back to bgr this is recolor image to rgb so these three lines or so these three sections are effectively doing our recoloring and setting our writable status to false making our detections then setting our writable status back to true and then eventually recoloring our image back to the bgr format but as of now we haven't actually gone and rendered anything so we could actually run this and what you're going to see is really nothing different right so what we could actually do is print out our results so let's try running that and see how we go okay so we can see that we've got our feed no errors there and you can see in the background that we're getting our results from mediapipe now we're not actually rendering anything as of yet so we want to take a step back and actually start rendering some stuff so we can close this feed again we don't need it running we can hit q close that down and then what we're going to do is actually apply our visualization component so we've done our recoloring but what we want to do is start rendering so let's go ahead and add those couple of lines of code to do our rendering okay so i've gone and done two additional things there so i've gone and added this line of code here and i've gone and changed the image that we're actually going to render so the first line is actually going on ahead and drawing our detections to our image so to do that written mp underscore drawing dot draw landmark so this is using our drawing utilities that we imported up here or we set up up here to actually go on ahead and draw to our particular image so the nice thing about mediapipe is that it's got this awesome visualization library that allows you to easily draw these results so you don't need to draw point by point to actually get this rendered you can just use those drawing utilities to that we've passed through three key things so we've passed through our image we've passed through our results landmark so you can see by accessing results dot pose landmarks this is actually going to give us all of our different landmarks now we can actually take a look at this so if i type in results dot pose landmarks you can see that we've got the coordinates for each and every landmark within our body so this represents each individual point that's represented as part of the pose estimation model but i'm actually going to show you what each of these represents once we get to step two right so then what we're doing is we're also passing through so in that drawing line we're also passing through the different pose connections so if i show you this as well so here you're actually going to see which landmarks are connected to which so in this case here you can see our nose is connected to our left eye inner so it's going to be this detect or this joint here so from here to here our nose is also connected to our right eye inner so from here to here our left eye inner is connected to our left eye which is going to be here to hear touching my eye but you can sort of see how it builds up so these ones are probably more appropriate so right shoulder is connected to your right elbow your right shoulder is also connected to your right hip so that's going to be that particular point there this just gives you a different connection so that's what we're passing through there then once we've actually drawn that image we actually want to render that to the screen so rather than having frame here which we did before we actually want to change that to display our image that now has our different landmarks drawn so let's go on ahead so it looks like we've got an extra line there so let's actually go ahead and run this and see how we go and we'll wait for our feed or for our little pop-up alright so it looks like it's closed let's try that again okay cool so you can see that we've now got our pose estimation so right now it's pretty basic so we've got our hands so this is really no different to the model that you might have seen inside of the holistic model and i can take the green screen down and you can see that it's tracking all of the different components within our body cool so that seems to be working pretty accurately so it's moving and it's quite quick as well but again i would rather this be in a different color so rather than having these standard colors which are what comes out of media pipe i'd rather go and change each one of these detection points so let's close this so what we can do is pass through two additional components to this draw landmarks line here to be able to go ahead and change our colors so let's add those and then i'll take you through them okay so those are our two new lines of code written so what we've effectively got here is two drawing specs now think of a drawing spec as the specifications for our drawing component so the specifications for how we actually want to go about drawing a particular landmark now what we can actually do is within this function we can pass through a number of different arguments so if i take a look at mp underscore drawing dot draw landmarks what we're actually doing is we're passing through a number of different things so first up we're passing through also so first up we're passing through our image which is that over there then we're passing through our landmarks list so remember this is going to be our results.pose landmark so all of those different coordinates then we're passing through our different connections which is mp underscore pose pose connections and then we're actually passing through two additional arguments so the landmark drawing spec so this first line over here represents the color that we want our different dots so that particular drawing spec is going to have this color this thickness this circle radius we'll come back to that in a second and then the next argument that we're passing through is our connection drawing spec which you can see there so landmark drawing spec then connection drawing spec so our connection is actually going to represent the different lines within our model cool all right now let's actually take a look at our drawing spec so let's actually bring that up so if we take a look at mp underscore drawing dot drawing spec you can see that we can pass through a bunch of different arguments so we can pass through the color the thickness as well as the circle radius so in this case here you can see we've got color for drawing the annotation so by default it's going to go to our green color the thickness for drawing the annotation as well as the circle radius if we're drawing a circle so what we've done is we've gone and specified mp underscore drawing dot drawing spec we've then passed through the color as a keyword argument and we've set that to 245 comma 117 comma 66 and i believe that is going to be in bgr format is it bgr format let's take a look rgb i can double check that for you hit me up in the comments if you want a detailed explanation but basically you're passing through a color spec there then we're specifying the thickness of the line and the annotation as well as the circle radius so thickness equals two and circle underscore radius equals two and then we've got our next line so remember our first line is going to be for our joint and our second line is going to be for our connection which think of it as your bone and again same line so mp underscore drawing drawing spec color equals 245 comma 66 comma 230 inside of a set of braces so this is going to be similar to a tuple but with three values and then we'll specify thickness equals two and circle underscore radius equals two so the thickness and circle radius are the same for both of these the only difference is the color here cool so i think that is now good so let's actually run this and take a look at how we're going so ideally we should see a similar feed the only difference between this and that first run is we're going to have different colors and there you go so you can now see that we again we've got the exact same pose estimation model just this time we've got different colors right so if i stand up move around you can see that again it's detecting my upper body it's detecting my hands and if i move bring my leg up let's move bring it down you can sort of see blocking it you can sort of see that it's detecting all my joints it's just now got different colors right got a feeling we're going to be bringing that green screen down a lot in this video okay so that is all well and good so let's quickly take a look at what we've gone and added there so in this section so remember we started off with our basic video feed what we then went and added is we set up our media pipe instance so we used the with statement there we then recolored our image feed from our webcam from bgr to rgb and we set a writable status to false we went and made a detection using pose.process and we passed through our recolored image set our writable status back to true so we can draw on it and then we went and converted it back from rgb to bgr so that it works with opencv we then went and applied our detection to that particular image so remember we used our image our landmarks and our pose connections and then we passed our two different colors our connection color and our also our joint color and our connection color and then we actually went and changed what we're actually rendering so in this base feed we're just rendering frame in our updated feed which is making our detections we're actually detecting our image cool so now the next thing that we're actually going to do is actually go on ahead and determine our joints now i've actually got a joint map here which is going to help render this or explain this a little bit better so if you actually take a look this is the architecture for our pose detection model so there are 32 landmarks in total and these are each of the landmarks so we've got our nose we've got our left eye inner left eye our left eye outer so on and so forth so you can see that each one of these numbers maps to a specific coordinate within our body so point 12 is actually our right shoulder point 24 is our right hip point 23 is our left hip so on and so forth but you can see that this is all the detail that's available when you're working with the media pipe pose model now rather than just sort of showing you this diagram let's actually go ahead and extract some of these joints so again what we're going to do is copy our code from step one so we're going to build up on this code step by step so i'm going to copy that and bring it into this cell here and now we're going to add in a couple of additional lines to be able to extract each one of these landmarks so let's go ahead and do that cool that's really it to extract our landmark so we've gone and written four lines of code there so this little block is effectively going to allow us to extract our landmarks now if we actually take a look at what we've written so we've written a try accept block so sometimes you're not actually going to be able to extract the landmarks sometimes they're not visible your webcam feed has a break so we ideally don't want to destroy our entire feed rather it's easier to just have a try accept block so if we don't make any detections then we can just pass so we've written try colon and then we've gone and set up a new variable called landmarks to hold our different landmarks and we're going to set that variable equal to results dot pose landmarks dot landmark so this is actually going to give us our actual landmarks then if we don't have any detections or if we have an error we're just going to pass through so we're not going to break the loop we're not going to destroy it we're just going to step out of it so this ideally is going to allow us to extract our landmark so i'm going to run this and we'll actually see our landmarks get created let's actually print them out print landmarks so we can actually see those results so let's go ahead and run that and we should be able to see our landmarks generated just make sure this is going to run that looks like it didn't uh run so let's try running that again so sometimes it'll pop up and then just close just try running it again and you should be good to go all right so we're good there and if we take a look you can see that we're getting all of our landmarks down here so each one of these represents a different coordinate or a different joint within our pose estimation model now again these are going to re-render as we're actually going through so what we want to do is actually take a look at one of these landmarks or start extracting some joints so let's quit out of our feed and let's actually grab a landmark from our last loop so if i actually go and type in landmarks you can see that we're going to be able to extract each one of these different components so we've got xyz and it's visibility and if you keep going ideally the length should be the same length as our model up here so we've got 32 points and again it starts at zero so you're going to bump one up and we've got 32 or 33 landmarks so again so this these landmarks are just going to represent what we've actually got here now what we can actually do is we can actually grab a mapping to each one of these landmarks so if we wanted to we can actually access mp underscore pose dot pose landmark and we can loop through it so for connection in mp pose actually this is just a landmark it's not a connection so land let's just call it lnd mark so you can see that this actually gives us our mapping for each one of our landmarks in this case pose landmark nose is going to be the first one which maps two nose if we go to the second one it's going to be left eye inner left eye you know so you can see that you've now actually got a bit of a map to work out what each one of these different landmarks are now what we can do is we can actually use this to be able to go on ahead and grab our landmark out of here so let's go ahead and do that oh so this line of codes are written for landmarks are lnd mrk in mp underscore dot pose landmark so this is actually an iterator then we've included a colon then print and then inside abrasives lnd mrk so basically we're printing each landmark from that particular landmark map then what we can do is as i was saying we can actually go ahead and grab one of these landmarks out using this map so let's try that okay so that actually shows you how to use that landmark map so what we've gone and done is we've gone and grabbed our landmarks which is what we set up as part of this loop so we're just grabbing the last set of landmarks from that loop and then we're accessing it from our particular array so if i actually extract this this is just going to give us an index so mp underscore pose dot pose landmark dot left shoulder dot value so if i take off value you can see that it's just going to give us our specific landmark as well as its index so by typing in dot value we're going to get where in our big landmarks array our left shoulder value actually is so in this case it's going to be 0.11 say for example we wanted our nose so type in nose you can see that our nose value is going to be point zero so if we actually paste that in we've now got the landmark for notes so you can try any of these right so let's try a different one so right wrist for example so right wrist is going to be point sixteen so let's try that and you can also just pass through 16 into here as well but i find it more useful to actually use the map so right wrist is going to be that landmark there so again and if we take a look this particular value should be what 15 16. so it's going to be so right wrist is 16. so you can see that you've got your mapping there so this gives you the ability to leverage each one of these landmarks now so say for example we wanted to calculate the angle what we might want to do is calculate the angle between our left shoulder our left elbow and our left wrist so ideally when you do a bicep curl for example you'd be looking for the angle that that particular joint is at right so let's go ahead and extract those three so 11 13 and 15. so we don't actually need those numbers for example we can just go left shoulder so that's our left shoulder there then we can copy that we can say left elbow and then we can do that again and grab left wrist so i get a lot of questions about how to actually extract these landmarks and work out which values which this is exactly how you do it so you grab your landmarks and then you use your mapping to work out which particular coordinate you actually need and again the same process would apply if you're doing the facial landmark recognition or the face mesh style recognition as well again so you're sort of grabbing your landmarks then you're working out which value within that particular landmark set actually represents your value now we're coming to the good bit so that is step one making our detections and step two determining our joints done so we've gone through quite a fair bit already now the next thing that we actually want to do is actually calculate the angle so for this we're going to be using a little bit of trigonometry so this code is actually based on the official ios code to be able to calculate angles using mediapipe i tried to find the link i couldn't find it i'll link it in the descriptions below if i do i've just gone and rewritten it for python so what we're going to do is create this function to calculate our angle first up and then we're going to try to calculate that angle between this particular joint here so we're going to calculate our angle for our effectively our left shoulder elbow and our wrist so we should ideally be able to work out what this angle over here is so let's go on ahead and write up this function so we're just going to create a couple of new cells and let's do it actually before we go any further into this let's actually take a step back and see what we're going to be writing out so i was going to write the whole thing but let's take it step by step so we're creating a new function called calculate angle so def calculate underscore angle and to that we're going to be passing through three points a b and c now a b and c are going to represent our first our mid and our end point so in this case it's going to be our first so point 11 our mid at number 13 and our end point number 15. so what we're effectively going to be able to do is calculate the angle between any three points so for that we've set up three arguments so a b and c and then we'll specified all pass through a colon and then we're converting them all to numpy arrays this is just going to make it easier to actually go on ahead and calculate these angles so we've gone and specified variable a so we're basically overriding a b and c and we've converted those two numpy arrays then what we're going to do is calculate the radians and actually convert that to an angle so let's go ahead finish this off and then we'll take a step back and take a look alrighty that is our calculate angle function done so let's take a look at what we've actually written here so the next two lines of code so we already sort of went through this top line here the next two lines of code are going to calculate our radians for our particular joint and then the angle and what we're going to do is convert that into an angle which is between 0 and 180 because our arm isn't going to go 360 degrees right so maximum angle we want is 180 degrees so what we've done here is we've written radians equals np dot arc tan 2 and then what we're effectively doing is we're subtracting our y values so y from our endpoint minus y from our midpoint so c1 minus b1 and this is effectively going to be extracting this value here so this particular y value minus the y value from here so say for example we're actually taking a look at our wrist so if we say our shoulder is or our wrist is our last point so we'd be grabbing y1 which is effectively going to be this or c1 minus b1 which is going to be this then we're going to do the same with our x variables so c0 minus b0 so it's going to be c0 which is this x value here minus this value then we're going to go ahead and do the same but this time we're going to do it with our first and our midpoint so these two values here so it's effectively going to be a1 which is going to map through to our first point which is going to be our shoulder so a1 which is going to be y minus b1 which is going to be our y from our midpoint so our elbow and then a0 minus b0 so x so x from our first point minus x from our midpoint then what we're doing is we're converting that into an angle so we're multiplying our radians by 180 degrees and dividing it by pi so the value pi and then we're converting that to an absolute value so this should ideally give us our angle but a 360 degree angle then what we're doing is we're mapping that through so rather than having 360 degrees we're just going to have a max of 180 then we're going to go on ahead and return our angle so what we actually want to do now is actually test this out so ideally what we want to be able to do is grab our shoulder or our left shoulder our left wrist or our left elbow and our left wrist value so let's go on ahead and grab these so i'm going to set up three new variables so our shoulder our elbow and our wrist to be able to calculate this so let's go ahead and do that and we're actually going to pass through only two values so we're going to pass through our x variable and our y variable our x coordinate and our y coordinate so let's go ahead and grab those values and then we can test it out okay so let's take a look at that before we actually go and do the same for our elbow and our wrist so i've gone and created a new variable called shoulder which you can see there and then what i'm effectively doing here is i'm just going ahead and grabbing the landmark that we had up here so you can see we've written the exact same line of code so landmarks and then we've passed through the coordinate number so mp underscore pose dot pose landmark dot left shoulder dot value exactly the same as what we had up here mp underscore pose dot pose landmark dot left shoulder dot value then to the end of that array we're specifying that we only want to grab the x value to begin with so effectively this line over here is the exact same as this line over here the only difference is that we're typing dot x to the end to only grab the x value so this grabbing our x value and then we've replicated the exact same thing over here to grab just the y value so effectively we're going to be writing this exact same line over here but this time we're just grabbing the y value so you can grab a number of values from this so you can grab the z value and i believe you can also grab visibility so as to whether or not it is a visibility as to whether or not that particular joint is visible to the camera at a point in time so that is for our shoulder so we've gone and grabbed our x-coordinate and we've gone and grabbed our y-coordinate so what we're effectively going to be doing is passing our shoulder in as value a which is going to be our first value our elbow as value b which is going to be our midpoint and our shoulder and our wrist as value c which is going to be our end point so these three joints are going to map to a b and c to be able to calculate our angle so let's go ahead and copy this and we can just make two additional variables so we're going to have our shoulder our elbow and our wrist and if we then go and update these rather than having left shoulder here it should be left elbow and it should be left elbow over here as well and then this is going to be left wrist wrist yep that's good that's good so now if we take a look at shoulder that's fine elbow that's fine as well wrist that's good as well so we've now got our three different joint coordinates now what we can actually do is pass this to our calculate angle value so if we do that calculate angle and we go and pass through those three values so shoulder elbow and wrist we've got our angle calculated so there you go so we've gone and passed through our three different joint coordinates and we've gone and been able to actually calculate our angle now what we could actually do is we could actually sub our different values say for example we didn't want to just calculate joint 11 13 and 15. say for example we wanted to calculate this particular coordinate house so how far away our elbow is from our core body well we could go ahead and calculate this angle so remember so that means our start point is going to be 23 our midpoint is going to be 0.11 and our endpoint is going to be 13. so let's actually test that out so we're going to need left hip then left shoulder and then left elbow which is going to be 13. so let's go ahead and change this right so rather than having shoulder elbow wrist so let's try this again so we can type in should be left hip then our midpoint is going to be our left shoulder so point 11. and our wrists so our endpoint this variable is obviously not named correctly but you sort of get the idea then our last point is going to be a left elbow to the left elbow right so we can do that so we've got different coordinates and if we calculate our angle it's 14 degrees so you can see that we're able to really calculate an angle between any three of these points so this gives you a lot of flexibility and a lot of power right so you can then go ahead and do a lot with this particular calc but what we actually want to do is render this to our visualization because right now we're not actually doing anything with it but we actually want to do something with it so what i'm going to do is i'm going to replace these with our shoulder elbow and our wrist value so let's go ahead and do that so if we now calculate our values again yep 166 and then as per usual what we're going to do is we're going to start off with our main detection codes we're going to copy this from step two so remember we had our detection code at step two let me zoom out so you can see that so this big block of text so we're going to copy that and we are going to paste it down here then what we're going to do is we're going to update this code to be able to actually visualize our angle so we'll do it initially so we'll do it for our particular elbow joint so this is going to give us our bice initial bicep curl code so let's go ahead and do that um foreign okay so we've gone and added a bunch of additional code there but most of this should be pretty familiar because what we've actually gone and done is copied our calculate angle code and then go and visualize it so let's take a look at what we've actually gone and done here so first up what we've gone and done is we've gone and grabbed our coordinates from our shoulder our elbow and our wrist so this code over here is no different to what we used over here so i've just gone and copied this down so remember we're grabbing our different coordinates our x value and our y values our x values our x coordinate and our y coordinate and we're storing that inside of an array which is going to be accessible via the variable called shoulder and then we've gone and done the same for our elbow and for our wrists then what we've gone and done is we've used our calculate angle function which we defined up here and we're passing through our shoulder our elbow and our wrist then the next line of code so this little bit is new so we've gone and actually rendered our angle to the actual screen now what we're actually doing is we're using the put text method from cv2 so cv2 dot put text and then to that we're passing through a number of different arguments so we're passing through our image which is going to be the image that we're actually working with from our webcam we're then passing through our angle and we're converting that to a string so str and then we've passed through our angle and then we're actually going and determining where our positioning actually is now to do that there's a little bit of math involved but basically we're using an array multiplication method so we're grabbing our elbow coordinate so the angle is actually going to be rendered right next to our elbow and then we're multiplying it by the dimensions of our webcam so my webcam feed is coming through at 640 by 480 so i'm just passing through those two values 640 by 480. so this is effectively going to grab our elbow coordinates let me actually show you some np dot multiply so if we grab our elbow coordinates and multiply by 640 by 480 what we're actually getting are the specific coordinates for a particular elbow for the size of the feed coming from our webcam so when we actually get these coordinates these are actually normalized coordinates so they're not coordinates based on the size of our particular image so what we need to do is a little bit of processing to get that available and so that's exactly what this line of code is doing there now because cv2 requires these values to be passers integers we're just converting it to an integer so as type int which gives us that and then we're converting it to a tuple because that's what cv2 expects so you effectively get that so the coordinates that it's actually going to render to are 531 and 591 so this actually represents where that particular joint is going to be within our image and that is that positioning there so that entire line of code is effectively doing that then we're just passing through some additional cv2 requirements for our image and it looks like we've got an extra one of those there so then it's just passing through the font that we want to use how big the font we want the color of our font so if you wanted to change it from white you could do that if you wanted it to be black you could just type zero zero zero we're passing through our line width and our line so cb2 dot line a8 should be a a cool now ideally if we run this code we should now be able to get the angle for our particular joint over here so let's go ahead and run this and see how we go so let's just say visualize angle so we've got good commentary let's run that so it looks like what have we done there so we've got that's fine we haven't closed our tuple up here so let's just close that cool so that's looks like it's running now if i actually bring my arm up you can see that we've now got our angle rendering so if i bring it to about 90 degrees that's looking pretty close so if we bring it in closer that's working open it up that's better so you can start to see that we've now got our joint angle calculated and if i actually step away let's take a look at this you can see that that's actually rendering in real time so if i make it almost straight so it's straight as possible it's getting to about 170 180 which is about right if i bring it in you can see that it's bringing back up to like five to seven degrees so 15 so if i open it up again you can start to see that that's calculating the angle appropriately now right now we're doing it for our wrist our elbow and our shoulder but what we could actually do is change this and calculate that angle that i was describing so from my hip my shoulder and my elbow so let's try that and see how that goes so what are we going to do so we're going to change the angle that we actually calculate so let's quit out of this by hitting q now remember how it's saying in order to do this all you need to do is change the coordinates that you grab so remember our first point is going to be our left hip so let's change that there and there our midpoint is going to be our left shoulder so left hip left shoulder and then left elbow so let's do that now if i run that again we should get the angle calculated for a different joint oh that's closed let's try running that again so sometimes it'll just close so all right so there you go so we've got our angle for our shoulder now if i open it up so you can see that it's now calculating this particular joint so if i go to 90 degrees you can see that that's calculating so you can do this for literally any joint so as long as you've got three coordinates and you make the midpoint the midpoint or the midpoint the angle that you actually want to calculate you can actually use this for a whole bunch of stuff so you can see that we're getting that angle there calculate it in real time you could also round this up if you wanted to if you didn't want to display all of the different coordinates but pretty cool right so that's our angle calculator i don't know i'm a kid in a toy shop so i love being able to do this stuff so we've got our angle calculated yeah we can do that all right awesome enough messing around let's go on ahead what we're actually going to do though is we're going to set this back to our shoulder our burn area so we can dive in left shoulder left shoulder and left elbow left elbow and uh left wrist we're almost there guys this is going swimmingly all right cool so we've got left shoulder left let's just test that out again make sure that works nope close try that again all right that's good let's bring up our arm so there we go so we've got our angle being calculated that's all well and good all right cool we're going well now we can also get rid of this print landmarks because we don't actually want that rendered so let's quickly take a look at what we did in this section here so we went and defined our function to calculate our angle and we used a little bit of trig we then went and extracted our three different coordinates our shoulder our elbow and our wrist and then we used our calculated angle function to actually be able to do that and then we went and applied this new section of code over here to be able to go and render that particular angle to our frame and again if you wanted to change the angle that you actually calculate you just need to be mindful of the midpoint but you can change this to different joints if you wanted to so we remember we did left elbow or left shoulder left elbow left wrist but we also did left hip left shoulder left elbow so again world your eyes to here you can calculate a whole bunch of different angles but on that note we are now up to step four building our curl counter so let's go ahead and do this now as per usual we're going to copy this block of code and we're going to build up again so let's paste it in there now what we just need to do is implement a little bit of logic to work out when we are actually curling so ideally what we want to do is when we're down at the bottom of our curl we're going to say that that's the down position and then once we pass a certain angle threshold we're going to say that we're now in the up position and that represents one curl so again you could apply this to a whole bunch of different gym use cases you could do this for shoulder presses you could do the do it for a whole bunch of different exercises as well now the core thing to note with this particular method is that it's limited to a single joint so you could obviously extend it out and do joint by joint but dev i've actually got a better method to do multi-pose tracking or multi-gym tracking for that particular case but again that's going to be in another video so do stay tuned if you'd like to see that so we're going to be able to use that particular code for body language analysis sentiment analysis and a whole bunch of other good stuff but for now we're going to stick to our curl counter so let's go ahead and apply this logic and have our core counter built so first up what we're going to do is create two new variables so we're going to create a counter so in this case we're going to set it to 0 and we're also going to create a stage and set it to none now our counter as you might have guessed is going to represent our running count for our different bicep curls and our stage is going to represent whether or not we're at the down part of our curl or the top part of our curl so down or up now what we just need to do is implement a little bit of logic to actually go and set these variables so let's go ahead and finish this off okay so that is our new curl counter logic so what we'll do is we'll test this out but before we do that let's actually take a look at what we've written so first up what we're doing is we're setting whether or not we are in the down position of our curl so basically if our angle is greater than 160 and i've given us a little bit of leeway because you might not always get all the way down to 180. so 160 is a bit of a fair compromise so if our angle is greater than 160 then we're going to say we're at the stage down so if angle greater than 160 colon stage equals down so we're basically saying if we're at 160 or greater our arm is in the down position then what we're doing is applying some additional logic so if our angle is less than 30 so around up here and our stage is currently set to down then what we're actually going to do is reset our state and say we're actually in the up position and increase our counter to 1. so this is effectively going to say that once we pass our 30 degree threshold and if we're coming from a down position we are then going to bump up our counter and the reason that we're checking whether or not we're coming from a down position is that we don't want our counter to keep spinning up once we pass that 30 degree angle so we're actually checking that we've come from a down position into an up position not that we've just passed into an up position so we should be able to run this now and see our bicep curl counter pop up so let's just make sure that we don't have any additional prints in here so we're looking good so just to confirm so what we've gone and done is created our new counter variables and then we've applied our counter logic so i think we're good let's test this out looks like it's closed let's run it again so if it pops up and closes just try running the cell again you should be good to go cool that's good now so let's try doing a curl so there you go so you can see our curls just popped up with one so you can see that that's actually running so if we go and do another one two three four five six pretty cool right so that's our basic curl counter done now i don't want to stop there we want to actually visualize this to the screen so let's make this look good so right now we've got our curl counter logic but we actually need to render this to a particular frame so this is actually reasonably straightforward to do we just need to leverage again cv2 to actually be able to show it to the screen so let's go on and do that and then we should be done with our curl counter okay so first up what we're going to do is set up a bit of a status box so cv2.rectangle and then we've gone and passed through a number of arguments so specifically this comma here so specifically the image that we want to apply to so this is going to give us a status box in the top left-hand corner so ideally it should be a place that we can render all of our stuff so we're specifying what image we want to draw it onto the start point for our rectangle the endpoint for our rectangle its color and by setting the last point or the line width equal to negative one it's going to fill the box with color as well so it should be a colored box so if we actually run this let's take a look again we're not actually going to have anything in that box as of yet but in a second let's try that again but in a second we should see a box in the top corner so you can see that we've now got our blue box so we're now going to render everything into that box okay we can close out of that so let's go ahead and actually put some stuff in that box okay so that is our rep data now visualized so we've gone and written uh looks like four lines of code but really it's two lines of code there so effectively we're just going ahead and using our put text method from cv2 again so cv2.put text we're passing through our image passing through the text that we want to display and it's start coordinate so in this case we're going to have a little label called reps that allows us to see where the reps value is and we're starting that at coordinate 15 comma 12 so in this case that coordinate there passing through the font that we want to use cv2 dot font hershey simplex the size of the text the color the line width and the line type and then again we're doing the same thing down here but in this case rather than just displaying the title we're going to be able to display our counter and again we've got a slightly different start coordinate so it's going to be 10 comma 60 and then again we're just passing through some additional details the font so font hershey simplex the size of our font so in this case it's going to be two the color in this case it's going to be white so 255 255 255 the line width we've set that to 2 and the line type so now if we run this we should get our status box with a little title that says reps and a counter in the bottom of it that actually represents the counter for our curls so let's go on ahead and try this and see how we go cool so we've now got our rep counter up here so you can see it oh so if we now go and make co make go make go cool so you can see that that's now calculating right looks like it was detecting my arm a little bit crazily before so you can see pretty accurate right and it's incrementing our counter so that's pretty good right so you're now at least able to calculate your different reps now the next thing that we want to do or the next thing that i want to visualize is actually the position of our arm so whether or not we're up or down so let's go ahead copy this over and we just need to make a few different changes to this particular or this these two sets of code to be able to display our stage so let's go ahead and do that okay that should be our stage data kind of done so what we've gone and done is effectively just copied this particular line or these two lines of text we've just changed the start coordinate so we're just starting the stage title at 65 comma 12 and we're starting our actual stage value so up or down at 60 comma 60 but everything else between here and here is really the same so now we should have reps and our stage so let's go on ahead and test this out now so i want to bring my arm down to the down position let's run it nope box is closed try that again cool so we've now got our reps and we've got our stage and you can see our angle is calculated just down there now if we bring it up one and you can see it's accurately calculating our reps as well as our stage so we can again keep going three four five six seven eight pretty cool right so you've got the ability to now go on ahead and visualize each of your reps now if i take a step away right so if i move back so let's go ahead and try this out again cell's closed run it again again you can see my arm you can see the angle calculated at this joint so it's 176 75 now as soon as we pass through you can see that's now gone up by one pass through gone up so we've now effectively got our rep counter and all we're doing is we're using media pipe calculating our angle and but you can see there's a whole bunch of use cases for this right so we can keep going up pretty cool right so that gives you the ability oh it looks like we've got a little bit of overlap so you can reset that and shift around the position but you sort of get the idea so you've now got the ability to go on ahead and leverage media pi for a whole bunch of different use cases so let's quickly take a look at what we've gone and done there so we've gone and i've finished off with a rep counter so the core changes that we did there are we set up our curl counter variables so we set up that and then we went and set up our curl counter logic now again you saw that once we got up to curl number 10 there was a little bit of text overlap so you can just reposition these values so if you wanted to move those around you could definitely do that really easily just by changing our coordinates or our start coordinates over here here here and here so you can move those around you could also make the font a little bit smaller but you sort of get the idea right so we've now gone and done a bunch of stuff in this tutorial so we went and imported and installed our dependencies we went and made our initial detections determined our joints so we were able to extract each one of our joints based on this map over here we then went and had those landmarks calculated our angle so we went and set up our calculate angle method which is really portable you could use this wherever you'd like and then we went and calculated and rendered our angles in this particular block of code and then we finished off with our curl counter so if we run our curl counter again we should be able to calculate some curls again so we're now baseline sage and if we curl you can see that we're now calculating our curls so that's sort of that so that about wraps it up for this tutorial so hopefully you enjoyed this we went through quite a fair bit again if you want access to this code it's going to be available via github in the description below or free you can grab that whenever you like and use it and if you get stuck or have any questions by all means do hit me up in the comments and don't forget to join the discord server on that note that about wraps it up thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get notified of when i release future videos and a huge thank you to all of you out there honestly from the bottom of my heart it means the world to me we just passed 10 000 subscribers and i couldn't be more ecstatic to bring you more ai videos and good stuff thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 168,379

Keywords: mediapipe, mediapipe python tutorial, mediapipe python api, mediapipe python pose, pose estimation, pose estimation python, pose estimation deep learning, pose estimation computer vision, pose estimation using opencv

Id: 06TE_U21FK4

Length: 64min 17sec (3857 seconds)

Published: Wed Apr 14 2021