AWS Nordics Office Hours - Machine Learning for Non-Data Scientists (and Knitters)

hi everyone and welcome to aws nordic's office hours with me guillermo i am a developer advocate here at amazon web services and this is your weekly show coming to you live from the nordics where i bring on guests to well teach you something new teach you about a service teach you about a feature show your solutions and let you our reviewers ask your questions and well hopefully we'll be able to answer them and if you have any questions throughout the show just post them in the chat and we'll we'll try to get to the questions throughout the show so don't worry if we don't answer your questions straight away we'll get to to them eventually so this week i am joined by chris howard chris is from the same team as i am the developer relations team and uh well you're coming to us from zurich was that right chris yeah i'm normally based in munich but uh i'm working out of the zurich office for the next three days and it's my first time in switzerland and it's it's beautiful so welcome to the show chris this week we're gonna talk about machine learning for non-data scientists so you're not a data scientist right chris no i i come from a web developer background a long time ago i was a front-end developer and always interested in machine learning and especially how i could apply it but i don't know about you i've seen a lot of talks at tech conferences over the years that say machine learning for for software developers or web developers and then you get there and it's just all calculus and i haven't done calculus in a really long time so so i'm i'm always on the lookout for for examples that i can apply that don't require me to be a data scientist yeah and there's an addition to the title as well that it's for non-data scientists and and knitters yeah yes i don't know the swedish word for knitting i know in german it's it's stricken yeah oh it's very similar okay yeah yeah um i am a knitter i i grew up i didn't grow up knitting actually it was something i i lived in australia for many years and when i moved to australia i took up knitting as a hobby um and so it's something i've i've just done off and on for a long time and anytime i get the opportunity to combine knitting and technology i i jump on it and i think i told you this you know there are a lot of similarities between the two knitting patterns are almost like a domain specific language yeah yeah you told me that and uh so well let's talk about uh the show a bit what we're gonna try to cover so machine learning is a very broad topic so we're not gonna talk about everything in regards to machine learning but we're gonna look at some areas within machine learning so can you tell me a bit about what we're going to cover sure i mean i i don't know about you gunnar but i find when i want to learn a new technology getting hands-on is best but i i a lot of times the intro tutorials are like build a to-do list and that's not interesting to me i like to try and solve a real problem even if it might be a slightly silly problem so my my angle on this was as a knitter you know i see people on the bus and i i you held up a beautiful jumper to me just a few moments ago and i'm always like cataloging the patterns oh i can steal that i can use that i can work that in and i in my brain can work out what the pattern is and it occurred to me that i wonder if computer vision could do it if i as a knitter could do it so i i over the last few years i've been working sort of on this little hobby project to try and teach a computer vision model to recognize different knitting stitches and it's not as silly as it sounds there are actually some researchers that at universities like mit who are also trying to solve this problem but i came at it you know from a very simple non-data scientist part of it and so i thought i'd show you my little project today i've i've used a few different aws services as well i started two years ago when i started this project with sagemaker amazon sagemaker but since then we have released recognition custom labels which actually makes it a lot simpler and so i can show you a bit about what i've learned about comparing the two very cool all right so let's talk about the inspiration to this then uh you you talked about knitting and you talked about a bit about patterns so let's go back i think let's go way way back to knitting from the beginning well i mean what like ancient egypt you're thinking like you know i'm thinking i've done a little bit of research on knitting because knitting patterns as i said like a domain specific language is an interesting thing you know for a long time knitting patterns were not written down they were just passed down and i mean obviously the nordics have a wonderful long tradition of of knitting and i've looked into some of the the norwegian and the swedish patterns they're just beautiful um but they would be taught in families and not written down until really only in the last sort of 200 years where they actually sort of formalized and the interesting thing is there's no standard for writing them everybody writes them differently they use different abbreviations some people use symbols use letters and and it's um you know i've seen examples where you can't just give a knitting pattern to another knitter and expect them to execute it's like assembly code you know you have to know what the actual terms mean in this in this particular language or framework and so that that's one area of overlap that i got really interested in and i i sort of developed a tech talk that i delivered to both computer science people but also to knitters about the similarities between knitting and computer programs because you know when you are ex when you were knitting from a pattern you were almost like a computer executing the pattern you know it have we have for loops and while loops and switch statements um they're just about knitting instead of about computer uh computer execution so uh let me just interrupt you there for a quick uh intro to the people who just joined us there was an issue uh starting out so we weren't broadcasting to the aws twitch channel but we are now so hi everyone welcome and thanks for joining us at the aws nordics office hours i'm joined by chris howard today and we're talking about machine learning for non-data scientists and networks so that's why we were talking about knitting oh they would have been very confused very confused uh so back to the knitting then and and the patterns yes so well how how can we apply machine learning to this well there are a few different ways you could do it i mean knitting at its at its basis they're really only two knitting stitches knit and purl so knitting's inherently binary you can encode data in knitting there are so many you know different analogies between knitting and computers um and depend depending on how you combine those knits and pearls you make different stitch patterns as we call it different textures and so i wanted to see if i could if i could read that if i could do it the way i do in my head and um there's this whole discipline called forensic knitting where knitters are actually trying to reverse this is reverse engineering reverse engineer patterns from photographs there's a very famous example if any knitters are listening the queen susan shawl which was a photo of a shawl from the shetland islands that a group of 30 knitters worked on online to reverse engineer just from the photograph you know so it's looking at it it's trying to trying different experiments to see if you can duplicate the stitch and now that they've actually recreated the shawl they've re-released it under creative commons license so i think that is a very cool example uh and so for my project i sort of my original thought was it's almost like reading handwriting you know like cursive handwriting so i thought maybe it's an ocr problem an optical character recognition problem like a scanner you know and so i i thought of trying to trying to approach it that way um and luckily we work with some very smart people at aws and so i chatted with the machine learning expert and she said no no no you don't want to do that go with go just make it just make it um just go with making it a classic image classification problem which simplifies the space quite a lot right yeah and uh to do that there are a few different ways of how to do that to recognize patterns or or objects in images and well you took i think you took one approach first so should we start a hard approach first yeah so let's start there and once again you're not a data scientist no i'm not established so i only learned as much as i needed to do this project um and so what i did is i found uh that um there's an existing project that some researchers are doing in mit um and i i i don't know if you want how you have the link there i think but um neural inverse knitting is a project these very smart computer scientists are doing where they're trying to uh print out a swatch and you're going to hear me use that word a lot a swatch is just a little sample of knitting you know knit one feed it into this model that's the neural inverse knitting there feed it into a model have it work out the pattern then feed that into a knitting machine and ideally spit out an identical swatch and they've gotten to about 80 percent accuracy using very big data sets they're using simulated and real knitting as training i i didn't have any of that capability so i decided to start very simple um and use amazon sagemaker to try and train a model that could just identify between a very few basic stitches and i also had the crazy idea to source my own data set um because there are there are open data sets out there uh the mit people have actually made their data set available but knitting has some problems when it comes to computer vision you know knitting is stretchy it's floppy it's fuzzy depending on the yarn so i thought oh i know i'll just crowd source my data set so that that was the first challenge this is not not like just seeing that it's a stoplight on an image or something like that exactly yeah yeah and and a photo of someone you know wearing their jumper that's not got the resolution you need you know you need quite up close you need to be able to really see the stitches quite well so yeah no it's not like saying actually recognition by the way can it does have knitting it can recognize knitting in it but it doesn't go any further than that it knows that it's knitting but don't you worry about it all right so uh yeah so you took the the sagemaker out trying to then source your own data as well for this uh yes how did that turn out oh well i put out a tweet um i put you should i share my screen i can show you the tweet that i put out with uh let me just pull it up here um all right get training data all right i'll show this so if i go uh share share screen this is live people yes it's live okay you can see that so that is actually the uh that's how the mit people did their swatches so they actually knitted knitted these with a channel to put steel rods into to try and lay them flat for photography which is a nice way of trying to solve the problem but as you can see they're their examples you know they do some lace stitches and different things and i thought that's really cool they had thousands of them i i'm not going to have thousands of them i need this to be much simpler so i decided to crowd source um i set up a little email pipeline i should mention as well uh i just used amazon's a simple email service i set up an email address because i knew i wanted all these training images in an s3 bucket and i didn't want to have to download them all to scrape them you know i didn't want people emailing them right to me and then i'd have to upload i thought let's just send them straight to the s3 bucket so i think this shows that yes you are a developer this is the developer route for the cloud of course you know there was some developer stuff in here so yeah there's a little lambda that would that would take the incoming images it would obviously check them for you know viruses and things like that but then um record them in a database and upload them to the s3 bucket so i got all that ready so i could handle the you know huge volume of of knitting data sample images i was expecting and then i put out a tweet and this was the tweet i sent out in october 2019 uh and i i had a blog post as well and i sort of said um hey send me send me your your you know send me your examples and it got retweeted like 350 times like it went you know it went viral on knitting twitter yeah and and people started sending in images i saw the s3 bucket started to get stuff in it and that was really exciting and then i looked at them and um despite putting you know very detailed as i thought instructions on what i want you people got very excited they got very excited and you know i got quite a wide range of images um you know some people sent me lovely pictures the whole garments which are good they're just not close up enough for me to actually get any detail from and then some people attach the wrong images or you know yeah what's the top middle one the lovely lady sent me a message and she said chris i think i might have attached a photo of halloween candy i said you did so all right so comparing this to the mit data yes exactly i thought i was saving myself effort by crowdsourcing data and i was not um yeah so i will i was yeah but and you know people were being so generous and wanting to help and i remember i i got a few messages from janelle shane who some people might know she does the ai weirdness uh blog and writes those interesting posts where she you know names colors or romance novels or whatever using a neural network and so she retweeted it and she's a knitter as well there's such an overlap between knitters and computer people um and also she's the one who actually pointed me to the mit paper uh among many others so yeah so i ended up spending a lot of time cleaning up the data right then and maybe that is a point in itself the the importance of data when it comes to machine learning if you take nothing else away as a as a non-compute non-data scientist data scientist told me you will spend 80 of your time cleaning data i did not believe them it is true it is 100 true that is by far the most time consuming part of the project right so you then had some source data cleaned up and and something that you were able to use yes and then i needed to label it so people were sending these to me i didn't want to label them all myself um and it was an opportunity for me to get hands-on with with uh sagemaker's ground truth um and i can actually show that if you'd like to see uh can i i might have to drag it into a separate window here and if i share that so sage maker is well it's something that it contains a lot of different services so sagemaker isn't one service right that's right it has a lot of services in it ground truth is one of those so tell us a bit about ground truth so ground truth uh we introduced um at re invent i think it was 2018 um but it allows you to create uh a labeling joe i'm like getting a credential error here refresh session expired always always okay um it's got a sagemaker is basically our end to end service for every facet of of machine learning and it's for people who kind of know what they're doing you know we have different levels of our machine learning stack and at the bottom of course is the role your own for the for the hardcore data scientists who want to spin up their own machine image you want to throw you know different technology at it um and then we've got atop the ai services which are sort of the pre-trained models and you just invoke basically as an api this is the stack in the middle and there's lots of services it's a complete end-to-end uh solution for for built building training data for doing inference it's got a lot in it and ground truth is something that we released which allows you to create uh labeling jobs that you can use uh custom workforces for to actually label stuff so if i open up here simple classification and you can see here the results um so i had these different images that people sent me and you can see there's a label underneath and what i can do i actually open up the labeling tool and show you what that looks like you can see this yes so it's actually really simple and kind of fun to set up so i put in the instructions here on the side you can customize the instructions for what the people will see and there's a few different options uh you can do you can have them draw boxes around shapes if you're trying to do identification of of what's in a scene but you can see what i did is i it shows them a random image from the training data and then they have an option they can select for the five different cases that i wanted to to try and identify and i also put in other because people sent me in ones that were still other um and so yeah you i had i think uh 15 different people you can set up a custom workforce for this or you can use mechanical turk i know some people are aware of mechanical turret where you can pay people you know uh to label your jobs for you but knitting's kind of a niche domain so i figured i'd better stick with experienced knitters for the labeling jobs um yeah yeah so [Music] you then and labeling means that you're adding a label of what the content of that image is and in this case it is one specific uh stitch stitch pattern yes exactly because then what we do is we take this training data set and we carve off 80 of it and we use that to train our model and we set aside 20 for testing the model afterwards to see how accurate it is um you don't want to test with one of the exact images that you trained it with no exactly it should know that yeah yeah so you could carve off uh 20 usually it's 80 20 i believe that that that you carve off um yeah and so i did a few different i tried different training jobs as i said ground truth lets you draw boxes so a few times people would send me images that had multiple stitches and so we we drew boxes around things but i think this this way worked the simplest for my use case which was to just label it and one thing you can do when you set up the job for ground truth is you can say each image has to be labeled a certain number of times and that helps you make sure that you don't have any mistakes you know so i said each image had to review at least three times so that so that i didn't get any errors in there right so then what's the next step when when you have labeled these images so yours well your source data has labels what's the next step then it's time to train your model and um so i used sagemaker for that so i'll go back to sagemaker here and if i go to uh training training jobs what you will see is this where you'll see i'm not a data scientist if i go back and look from you know it took me uh how many different training jobs until i got one that actually finished i think seven or eight to get one that actually completed and what you see here is i've actually okay yep so i just one two three four five six seven the eighth one actually completed because it's tricky like sagemaker is i liken it to you know it it's got all the bells and whistles if you know what buttons to push stagemaker gives you really great flexibility as a data scientist me i was working from some demos and some blog posts and trying to apply it to my use case and so there was a bit of trial and error to get it to actually work but i did actually get one that completed um and i set up a little front end for inference and i tried it out and the very first one was correct and i was like oh my god nailed it uh and and then quickly realized it was returning stockinette stitch for everything it was worse than a coin flip it was terrible and that's the other that's lesson number two your first model is always garbage um uh what do we say in australia it's like the first pancake the first pancake's never good so you know you have to make many and so that's why you see i made so many different models testing different um different configurations and yeah uh trying and just trying to get which ones had the best results and so if i look at one of these like this one take 14 it completed and sagemaker gives you lots of information it tells you i was using the image classification algorithm sagemaker has a lot of different algorithms you can choose from i just went with image classification i used a p32 extra large if people are interested you you can specify which instance you run your training on which makes a difference for how long it takes um you give it your s you point it to your s3 bucket that has your training data in it i also pointed it to another bucket that had the 20 validation images and it gives you some stats here as well and so yeah you you can get like i don't know what most of this means but i'm sure that antia could tell us yeah but but that's uh kind of the point with this as well to show that you can actually use it without being a data scientist and you're able to actually get results from it anyway oh yes oh yes you absolutely can and we have some great blog posts and the ramp up guides are really helpful for that and and i did eventually get a job that had quite good accuracy um one thing that i did that was very helpful that i didn't know about was hyper parameter tuning jobs that sounds very fancy um i have that that is in here as well hyper parameter tuning jobs so this is where i guess hyper parameters are all the little different options that you can tweak when you're when you're training your model hyper parameters sagemaker has a built in the ability that it will try out hundreds of combinations for you and rank them to find the best combination so you don't have to manually do that so i ran one as you can see i ran a couple of them here i ran one for 13 hours to find the best model um and that was a tip i got from our colleague gabe and that that really helped me fine tune and get the best the best model so question in the chats did you see it yes and i actually have a slide where i will show you some costs later at the end so all right pause on that i do have i do have that information cool so all right uh and then what now you you were able to actually get some results i should mention i also tweaked my training data quite a lot over the course of all these training jobs and the hyperparameter tuning um so what i did i ended up cropping all the images to a standard size i think with image computer vision that really helps i also ran them through image magic and rotated everyone 90 degrees which also has the nice effect of doubling your training data size so i ended up with about 900 images even though even though it is the same image you can actually rotate it yep that works um i made them all black and white to sort of i didn't want to have any influence of color like the color is kind of irrelevant so that's one trick i got so make them all grayscale i also introduced uh clutter which is a a specialized term i didn't know as a non-data scientist this is where you add in images of things that aren't what you're looking for you know because when i trade my first model i dragged in a picture of a cat and it told me that stockinette stitch i'm like wait what it's because i trained it on only images of knitting so to the to the model everything in the world is knitting so i added in pictures of animals and airplanes and musical instruments you can find these data sets online and that way you train it but not everything in the world is knitting so clutter is the term for that and um yeah and then i ended up getting some quite good results i got to oops let me just pull go back to my little slide thing because yeah so we talked about labeling it with ground truth ah this is where i started to d scope by the way because i realized for example i had moss stitch and seed stitch uh i discovered only halfway through that the americans and the british and the australians used them to mean different things so my dad it was unreliable i discovered that garter stitch in reverse stockinette stitch look really hard for even me to tell apart so i threw that out so basically i scoped it down inspired by silicon valley to just stuck in it or not yep so that's what i went with stuck in it or not um i trained the model we talked about let me sorry just skip through all of this because i've showed you this and this is my little web front end and as you can see this is where the cat came up as stuck in that stitch yeah um and that was my first my first training job 45 accuracy yeah terrible yeah so it was 41 sure that it was stalking at the cat yes it was 41 sure that it was stuck in that stitch for the cat and yeah so so this is me yeah so by training job 15 using all of the tweaks i told you that i made with hyper parameter optimization 900 images i get it up to 94. 94.8 um which is pretty darn good um and i was very happy with that um and then i see recognition custom labels and i'm like what it was literally like two weeks after i finished we brought this out always the way so let's start with amazon recognition first about what amazon recognition is have you played around with it at all i have yeah what did you use it for well i've used it to for the basic thing to recognize objects in images um that's so it hasn't been the main focus of what i've done but usually it's quite uh an interesting thing to add when you're building different kind of demos to to add ai to to whatever you're building yeah recognition is definitely one of those that people enjoy to see and it's super easy to use it can recognize celebrities and different options you know different objects and scenes but until now you were limited to the object it had been trained on and with custom labels what we what we brought out is the ability for you to provide your own custom training data and it would there's this concept of transfer learning in machine learning where you can you know take a pre-trained model but apply it to your data uh which is what i decided to try and do with the i already had that great training data set after all that work that i had in an s3 bucket straight from ground truth and i was able to feed that right into recognition custom labels and i was interested to see whether the results would be comparable and i will i have that in uh i can open that up right now recognition [Music] so while you're doing that everyone this is the aws nordic's office hours if you just joined us we're talking about machine learning for non-data scientists like chris and myself who perhaps doesn't know all the ins and outs of machine learning and well sage maker perhaps so yeah so amazon recognition custom labels yep custom labels here uh um it is so easy to use like i was blown away by this you just create a project um and so i created a project called knit ml that's that's my the name of my little project and underneath it each one of these is a model so to train a new model you literally give it a name you point it at a data set or you can even upload them straight into this ui you don't actually if i do create a new test data set and we'll call this nordic knit ml and i will create a new uh oh i have to have a data set set up so no i can't do it um you can actually upload uh if i go to data sets here let's do that create data set nordics ml you can point it to an s3 bucket you can use your ground truth as i did or you can actually just upload images from your computer so if you do this option it actually gives you a ui and you can actually just start yes i know how to use the tool so if i uh let's just add some images i have on my computer some test images here so let's just add five of these so you don't actually have to go through that whole process you can actually and theoretically you can actually start with as little as i think is it is it 10 images for custom labels i think it's like 10 i didn't get very good results using only a few obviously the more you have the better um but you can upload them straight from your computer here and then you can actually label them so for example i know this one here la mingus i can actually i need to add some labels so i need to add a label call it stockinette uh not stuck in it like you could do this project so quickly and so la mingus here is actually uh i don't think it added the second one i didn't add the second one did i not hit the button it's probably because my not stuck getting it there we go so la mingus oh assign labels here we go is stuck in it so you can go through and label your your data set very easily in this way you don't have to have the the hundreds of images and ground truths but if you do so much the better and then you can feed that into a project and you literally you don't have to it's so easy to train a model you you give it uh nordic's ml i'll just pick an existing data set for this just to show you you choose a training data set you choose a test a test data set or i can just tell it to just split 80 20 based on the the data set that you've given it and you click train that's it those are the only buttons you have to click to actually kick off the training job and i mean of course that means so much is abstracted away from you i don't get to choose what instance it runs on i don't even get to choose the algorithm it's going to figure out the best one to use and so you you lose a lot of the control you have that you get with sagemaker but it's so easy to do yeah it also means that you don't have to choose which instance to use you don't have to choose the different options absolutely it makes it definitely easier to get started with you doodle yeah i mean you lose a little bit of control over over how long it takes to train because we that's how recognition is billed by the hour for training and so because you don't really have control it's a little bit nerve-wracking because you're like oh most of the ones i did if i go back and show you i think mine all finished in uh can i if i click on them you'll see most of them finished in under an hour um 0.827 hours for this one you know i think there was one version that ran for four hours but everything else did in in well under an hour how big was it 900 i think 900 where's the one that went for four hours was it this one no that one on 900 images that was 0.827 and it's interesting because when you click on one of them you can see you know you can see your actual test results so it'll show you the test data that i carved aside it will show you whether it was accurate or not so this was one that was correct true positive this one here is see some of these are actually difficult for me look there's some clutter there's an accordion it knows that the accordion is not stuck in that stitch correct and so it scores itself based on this um and this one's pretty accurate actually so if you look at the accuracy ratings there the one that got the worst um 66 i think was the very first one i no this was this is the one that ran for four hours the one that ran for four hours had the worst results and it's where um i think i used the color images it was where i used the original color images and that performed the worst so actually grayscale worked better i found that's interesting yeah it's very interesting and then to actually um demo it um what we we have this great i have to show this because i think it's so cool um i found on our github on the aws samples github we have this great recognition custom labels demo and it spins up this lovely little website pointing to your custom labels um that you can just launch by just clicking a cloudformation stack and so i've done that and so i can actually we can try them out and i can show you what it looks like can you share that link with me so i can share it in the chat as well yes i will i will share that with you right now there you go thank you and so i i've spun this up and this is the other awesome thing about custom labels if i go back to that you just got projects and data sets models and data sets with sage maker you have to set up an endpoint you have to set up an endpoint configuration all of this stuff for doing inference you know for actually submitting an image and getting back a prediction you actually have to do a little bit of that scaffolding yourself recognition custom labels handles that for you like that is available for inference i don't have to host anything i don't have to do even with sagemaker i had to for example have a lambda as an intermediary because sagemaker endpoints are not exposed to the public by default so recognition custom labels was just so much easier and so here i have uh you know i did five different versions trying out different ones um and i've started these a little earlier this morning because it does take a little bit of time for the model to start up for you to start using but if i open i think this is the one that got the best results here and i will upload some test images i've got some test images so i will start with that one of la mingus we just saw um confidence stuck in it 100 okay it's it's pretty sure on that one um uh how about me this is a jumper that i was wearing stuck in that stitch um paulina it gets this one wrong it's not perfect um this is one image uh a woman's jumper i took a picture of it gets wrong every time it thinks it's stuck in it i don't actually know what stitch this is i think i know why because you can see stuck in that stitch kind of looks like little v's and you can see that here obviously i think that's the feature but i'm not sure can you open an image in a different window just hold it next to this that one uh what do you mean oh a stuck in that one yeah yeah just to compare well yes uh how can i share both i'll switch i'll do a stuck in that one and you can all right oh yeah yeah yeah i'll go back to the yeah where was it too many windows open i'll do a stuck in that one for you like here's just a very basic um so i don't know if this is a fuzzy one actually and it gets this one wrong because it's so fuzzy where's a good stuck in that one there's a good one you can see it's got little like columns of little v's and so i think that's why it gets the paulino one wrong but it's hard to say um so it's not perfect uh you know i know that there's this concept in machine learning of overfitting uh where you've you know you you and so i'm um yeah obviously i need to get now maybe some some advice from one of our machine learning colleagues and how i can uh test this even better but i'm really pleased and this demo is so easy to test and demo and it's so great but maybe more tests or um sample data of of that stitch that it doesn't get right yeah do you think that is the and it might be you know 900 images is not a big data set when you talk to some of these machine learning you know i think that mit group they used they used like 5 000 or something so yeah so i i didn't have nearly as many uh training images as some of those i have your jumper going on yes you you brought one yeah i did where is it chris asked me to bring you jumpers i brought one of their kids jumpers okay so let's see and you've sent me a little sample image yeah and it gets it wrong um so it thinks this is stuck and it's very see but this doesn't have the little v's so i wonder if it's because it's a very regular pattern uh yeah it's getting this one wrong so this is actually reverse stockinette stitch this is the back side of stockinette stitch um and it's something where if i had more i need a bigger training data set that has the different types of stitches um so i did i did do two different models here so the one here let's test it on this one uh so this model actually tries to identify which stitch it is not just stuck in it or not and it still gets it wrong as well obviously you need more need more samples of that particular stitch yeah yeah and you can see it's really cool you can see the request that's being sent you can see the response that's and obviously you know one of the things you get and i think developers need to understand about machine learning is you don't get a yes no answer you get a confidence spectrum you know and we have to know how to how to interpret that that's one of the things i i talk to developers about is regardless of whether you think you want to implement machine learning models you know in the future the apis you're interacting with are sometimes not going to return a binary answer and you're going to have to branch your logic depending on the results that you get back so it's important to know a little bit about how this stuff works even if you don't have a particular machine learning case you want to utilize i think that's a really great point that in in this case if i were to build an application using this this uh model i need to decide is 97.8 confidence in something good enough to have a binary answer to that in my code yes or no testing the cat the cat is definitely not knitting so that's uh you know but yes you're absolutely right you need to decide the threshold for the project and obviously this is a silly example this is determining knitting stitches this is not determining whether or not someone gets approved for a bank loan you know so there's obviously depending on the project you really need to know what those thresholds should be and look at them very closely and well you just said that it's a silly example but think about this but apply to other areas what what kind of examples could you see based on on the same principles well i can give you a really good example actually so one of our aws heroes who is in italy uh his name is luca bianchi luca wrote a blog post because he did kind of the exact same thing i did he had a model that he did in sage maker and then he fed it into recognition custom labels to to compare and so this is the blog post and i'll put it to you in chat as well so you can share it um but the use case here is i think if people want to know what is a practical use case for this this is using recognition or using image recognition um to automate detection for heating and air conditioning you know you want to know if it's dirty like for for maintenance purposes and stuff like that and so uh in this in this blog post you can read through it um luca did the same thing as i did did it with sagemaker and went through all the process of getting training data and all of that and got you know pretty good results and then tried it with custom labels and found out that it gave comparable results with so much less effort um and i can actually show you in terms of my own results if i change what i'm sharing here go back to where i have my results because that was the thing that blew my mind so i 900 images i got like 98 you know and obviously there are those edge cases as we said where it gets wrong but based on my training data recognition custom labels it's hard the the results it gives you here are not one to one with the results sagemaker gives you but it's it's comparable like in my testing it's comparably accurate and luca arrives at the same conclusion for the hvac example oh that's yeah that's very interesting and also just looking at comparing the two of course as you mentioned several times with sagemaker you have more control over how to train your models and what to do with it how to improve on that data but but the speed that you can have a train model with recognition custom labels it's it's yeah it's hard to compete with i actually think the next slide if i bring it up hang on oh no hang on let me just this is the one that compares the two so i kind of tried to do a little comparison so you know training time was ultimately with sagemaker faster but that's because you have control over what size instance you throw at it you know um cost is the big one where it makes a huge difference because recognition custom labels this is the important thing for anyone who's doing it um for every hour that you have that model turned on for inference you're paying four bucks an hour so really recognition custom labels works great for a use case like lucas where you batch up all the images you turn on the inference you run it across your batch and then you shut it down again so you would not use recognition custom labels for anything where you wanted to keep it ongoing like you know for a web app or something where you want it on 24 7. don't do that batch it up and run it through it so could you go there yeah go ahead yeah yeah i was gonna say cost is there as well um and really it's it's that it's that you the recognition cost is very simple a dollar an hour for training and four bucks an hour for inference it makes it super simple versus you know sagemaker you've got a lot more variables but i think i spent about 50 bucks on the training for sagemaker and that includes you know i ran 13 hours of hyper parameter optimization training you might not need to do that um but that yeah we we calculate them differently there's a question in the chat about uh jupiter and aws guides or tutorials and and there are a bunch of i mean jupiter notebooks i didn't go that route just because i'm you know that's i've seen examples and i did a couple internal tutorials at aws using jupiter notebooks um my python is probably a bit rusty these days so i decided i did all of this in the console because i wanted that experience of going through it sort of in the most lay person way possible but yes we we absolutely have blog posts that can get you and sagemaker now has like the quick starts and things uh that's my other thing i would like to try out and because you know sagemaker now have have some patterns that you can get started maybe a bit faster than it did when i was using it two years ago i posted a link to the documentation about using uh notebooks um so maybe that's a good place to start at least to learn more about that if if i ever do this muslin notebook i will share it but for now it's all just been done in the in the console really but what if you start out with recognition custom labels and then you want to have more control well once you have that training data you can absolutely feed that in sagemaker you know you can reuse it so custom labels was getting the training data is the hard part you're not gonna you know custom labels only saved me the headache on the training the actual training and inference part in terms of getting a clean data set we have not yet come up with the magic bullet on that you know ground truth is helpful but you've still gotta you've still gotta do the hard work yourself there right so sorry and that last line there ease of use is really where it came down to for me that um you know i those five models i did in recognition custom labels i think i did that in about two hours on a sunday morning you know it's just so simple to do yeah that's really interesting um for sure so there was a question earlier on about any architecture links to share do you have anything to share in regards to what you've what you've created so i have a yeah i have a longer version of this talk that goes into more detail on some of this stuff and so i can uh share that with you i don't have it available on a github repo i have some blog posts on my site but like for example the ses architecture i'm not documented and i need to do that so that is a good call out and i will do that i mean most of this as i said was done in the console so there's not a lot to actually share but i should document it all in a really in-depth blog post so people can recreate it yeah for sure can can you perhaps share a link to the blog post at least so we can get that in the chat yes you talk i'll find it yep that's fine so i i know also you had there's a great link on the machine learning ramp up guide let me post that as well so useful which is something from aws training and certification a question in the chat why not use the knitting training book images yes so there are a lot of i guess like a stitch guide is that what you mean sri there um that's an option so there are various books people have published which are just like you know have basically a little image like i had and have them labeled part of it was just because uh the legal rights for doing that you know i didn't want to be scanning someone else's book um so especially because i knew i wanted to talk about this in a few places and i i wanted to make sure that i had the rights to use the data and the easiest way to do that was to get my own um but yeah that would be an option um but really really that book only has one picture for each stitch pattern and what you want is you want to have multiple so that you can train it um so yeah that at least is a starting point you could use that as a starting point but it's not got everything i'm going to put in here a link so that is the full talk you know i showed you a few of the slides but if you want to see in detail that's probably the best place to send you if you go to that talk there um that i've just shared gunnar that was a recent conference i did back in australia uh that i went through the whole project in detail but i will i will if people follow me on twitter i will write a blog post and share it on there all right cool so i'm interested to see a bit more on sagemaker and on ground truth so are we able to skip back there you want me to go back to ground truth yes please all right you've got an idea for something to do with it no no well i i want to see uh if you're able to just show from the well how you get started with the ground truth absolutely yeah so okay go back there we go so ground truth so what you set up is you set up a labeling job and so if i go to create a labeling job i'm going to give it a name nordic's ml um you can give it the s3 location your s3 bucket where you've got uh see they've changed this since i've done it so i'm like i have to look at the ui a little close um manual data setup yep so this is where i can actually point it to my s3 bucket uh and then you give it a task type as you can see there are a few different types is where i told you like i explored with we're drawing bounding boxes um you can do image classification with multi-label single label and you can even get them to draw really closely around different images if you're trying to you know identify people versus cars or whatever so my knitting skills are well they are below basic so can it well kind of does the jumper have multiple stitches in the same yes so the jumper you showed me on the front has multiple stitch patterns yes so if you yeah that one has multiple patterns on the front um hold it right up to the camera i think there you go there you can see it yeah that's got multiple stitch patterns on it so would that be the next level stuff and then to be able to recognize multiple stitches that is the next one definitely so that's where i was trying to do bounding boxes so that people could draw a box around this bit is garter stitch this bit is stuck in that stitch um but i think yeah i need to just go back to my data set and make sure it's really clean and and in order to get that to work so that that is another future iteration is to go back and try and identify multiple in the same garment right so you then choose which type of job you want to do yep and your feet yeah make yeah yeah so you give it can i give it a bucket let's just give it a bucket let's give it uh sure whatever it's been a while well anyway we'll just yeah you you give it a bucket you give it this and it will then set up it's not going to do it because i haven't given it the bucket um i'll go into an existing labeling job as i said there is a way you can put in some custom instructions um for your actual uh for the specific domain you're doing so you know if you're identifying different birds and you want people to draw a box around it or whatever you can actually put that in there and so that's where if i go and view labeling tool um that was that bit on the left i did where i put in the instructions right so you can provide some custom it's a little bit of html basically you can specify for doing there and you can also specify that's a good point so the workforce experience so um labeling workforces so you can set up your job to use amazon mechanical turk you can have a private workforce or you can actually even have a vendor be your workforce for you i set up a private one and so whenever it sets up like the necessary uh cognito identities and all of that and so whenever someone volunteered that they would do it i would set them up in there and it would actually send them an email with a login and a ui to actually so it's very nice from their point of view they it's just a little web app they go to and they get logged in and it tells them you have 50 images to review and they click through and do it and that's that's the dirty secret of machine learning is ultimately there are a lot of human beings clicking a lot of buttons to make it awesome there are and you and i spoke before this session that none of us uh we haven't used mechanical turks so it would be interesting to know if any of the viewers had tested mechanical and what's your opinion on that then i mean i think if you were like drawing boxes around a basketball or something sure you know people know that but i didn't want to rely on on you know if i was finding it tricky to identify the different stitches i i knew that a a random workforce would not be able to do it no probably not yeah cool so with uh who did you use for for workforce by the way did you just be new yeah i had about 15 knitters um i actually have you know i should do i should show you my my thank you screen because you know that's that's uh i love to give credits to these folks who help me out on on doing this um so yeah i had there's you get some stats on your ground truth job so that those stats there actually come from ground truth so you can see it took about 17 seconds per per labeling job um and because i did different versions of the bounding boxes and the labeling about 3 000 different you know labels got clicked uh so that's that's quite a lot um and 14.5 hours and so i've got here some of the people who i think had about 15 people who labeled and about 50 people who sent in images so it was quite a big team in the end very cool yeah yeah and gabe at the bottom yeah well gabe gabe helped me out with some code for um a little web app for doing the sagemaker inference i didn't get a chance to show that off but if people watch the full version of the talk they can see that a little react app for doing it doesn't you know i couldn't find a nice version like the react recognition custom labels one but i i built a little react app based on what gabe gave me for that is that similar to his bird recognition i haven't seen it does he do bird recognition it's probably the same it's probably the same he had it on github and share i think it's gone now but uh yeah yeah it's very useful all right well very cool so we're approaching top of the hour so getting time to wrap up but if you have any final questions for chris uh hurry up and post them in the questions have been great i was so worried no one would ask anything now there's been some really good questions throughout the episode all right so chris let's try to summarize this then uh if you were to start from the beginning how would you do this now with everything you know hindsight hindsight well always i think if you're trying to learn a new area of technology find a real problem to solve i think that's the most important part and especially with data science you know you get a lot of demos and things where you download some sample data it doesn't mean anything to you i would have never been motivated to see this project through to completion if i wasn't really kind of interested and curious about the results so start with a problem you really want to solve see if they're in existing data sets and you know kaggle.com is a great place to find data sets um it's very interesting that we're working i'm working on a talk for re invent right now which is related to this which may involve different types of beer and so we've been looking at data sets for that and maybe trying to source our own but try and find a good data set and then just know that you will spend a lot of time getting clean data keep in mind what i said if you're doing image recognition do the rotation trick add in some clutter make them gray scale all of that is going to give you much better results in the end and don't be afraid to d scope start with a really simple prod you know problem is it's one thing or another and then you can build on more complexity on top of that but recognition custom labels definitely uh if that had existed i probably wouldn't have even gone the sagemaker route back then but having looked at sagemaker and gone through that process i think you probably learned a lot that you are then able to apply yes absolutely absolutely and things like hyper parameter optimization like i now have hands-on experience of what what does that mean and what a you know and things like how how do you know that a model is accurate it's not just yes or no that's correct you know did it get it right that it was the label did it get it right that it was not the label that whole like true positive false positive like there's a lot of nuance that goes into that that i wouldn't have really known about from just reading a blog post right very cool so this has been another episode of aws nordic's office hours this week i was joined by chris howard and we were talking about machine learning for non-data scientists and knitters and knitters well thank you so much for having me it's been really fun well thanks for joining us chris and i hope everyone watching enjoyed it and we will of course be back next week and next week uh we will be talking about cdk pipelines for exciting well for for everyone not only for non-data scientists and not just for knitters no no no exactly so thanks everyone for watching my name is giuliana girosh i was joined by chris say hi chris all right bye bye everyone see ya

Channel: Gunnar Grosch

Views: 21

Keywords:

Id: pkVHLNjb5H8

Length: 59min 4sec (3544 seconds)

Published: Mon Sep 13 2021