Machine Learning Models Deployment with Flask and Docker

thanks everyone for joining my name's nathan cheney i am marketing manager at new science dojo and i have alexi and alexey i tried to pronounce your last name yesterday and i probably butchered it so forgive me um gregorette no close enough close enough okay but i have alexey here with me he's a principal data scientist at um olx right and um and obviously we just mentioned he lives in berlin with his wife and son and um he also runs data talks and which is just a place to talk about data so alexi i'll let you go ahead and get started with uh deploying machine learning models with classic and docker like i mentioned earlier if you have questions use that q a tab um help us keep everything organized we'll we'll make sure to answer questions as we go and then if you're on a live stream ask your questions in the chat we'll pull them in the zoom for you so lexi why don't you go ahead and get started okay thanks a lot for having me so i don't think i will be able to pronounce your last name to you so okay so let me start so i will actually use this thing so this is a book i wrote so it's called machine learning bookcamp and what i want to do today is open chapter 5 and this chapter 5 is about deploying machine learning models and i just want want to work through with you today through this chapter you don't need to have the book to follow this so this chapter is quite standalone we will have however use a model from a different chapter so here in this talk i assume you already know a bit of machine learning but if you don't it's fine then you can maybe come back later and check some of the materials for today so let me start i will share my screen now so i'm sharing my screen so um yeah actually speaking about the book so there is a course based on the book it's called machine learning zoom comp which is kind of funny the book is machine learning bootcamp and the course is machine learning zoomcamp right so the course is available on this link a mail zoomcamp and everything i will say today is available in the course repository so in the course repository we have this zero five deployment and this is the code we will use today so we will actually this code so we will write we will now create something similar to what we already have here so later when you want to come back to the materials of this webinar that we will have today you can come to this module 5 and check and check it out and of course you're welcome to sign up for the entire course it's free but yeah let me start actually with saying what exactly we will cover today so first we will take a model so this is a code for let me enlarge it a little bit so this is the code for producing a model and what this model is doing it's predicting if a customer of our company will churn so we can pretend imagine that we work in a telecom company and we give different services to people like i don't know internet services landline land phone services and things like this right and we have some clients and some of these clients might decide to leave our company and go to competitors and what we want to do is we want to identify those customers customers that are about to leave us and go to competitors and perhaps propose them some discounts or somehow convince them to stay with us so let's imagine we have a large customer base and we want to score every customer on this customer base and identify those who might churn and then those who churn we want to send them some promotions and convince them to stay so for that we will use this uh code we'll use this data set that actually i think i took from kaggle tell call data set cargo let me try to find it yeah and this is the data set so quarter so in this data set we have different let me enlarge it we have different uh characteristics of customers like gender of a customer whether they are senior or not whether they live with partner whether they have dependents like kids how long they are with the company and things like this and at the end which is something we don't see here is at the very end we have this thing called churn so and which is yes if they left and no if they didn't so based on this data we want to train a model for predicting customers who will churn in the future who are likely to churn i already have the code for this so we will not talk exactly how to train this model so this is something that you refer to module 3 i think yeah this classification model of the course and the book too to see how exactly this model was trained so we will not talk about this here so here we will talk about the moment when we already have a model and we want to go and deploy it and start using this model and for that so first we will execute this code and save the model we will save it to pickle pickle is like a special format for saving the model then we will load this model from a different process we will see how to use this model and we will serve this model with flask flask is a special web service for for python that could be used for serving the model too then we will see how to manage dependencies and create a virtual environment we will use a tool called pen for that after that we will package everything in docker so we will have a docker container we will put everything inside this container we'll create a docker image and then at the very end we will take the docker image that we created and publish it to the cloud so we'll actually deploy this and it will make our web service our model available so that's a lot of things to go through so let's get started so i just want to take this notebook and execute it you don't have to understand what's going on here but you can always refer to this and see the code so what i'm doing here is i'm training a logistic regression model so this is the logistic regression model and i'm determining what is the best value of c so c is one of the parameters of logistic regression model and yeah so here i am uh doing cross validation here i actually don't know why i need to do this so let me just remove this so at the end i want to train the model so i use this train function i use the c value of zero i see a value of one right so i train the model and then i evaluate the quality of this model and this is the quality right we have a model so this is model so this is the model we created here with c equals one and we want to save it so for this model we actually need to save two things so we need to save the model itself here's the model and dictionary vectorizer so this is the thing we use for pre-processing data so if you don't know what it is we will see this later how exactly to use this uh don't worry but yeah so we need these two things to save and nathan can you please mute yourself because i think i hear you it's a bit distracting yeah thank you now and now so we saved it and now i want to show you how we can load this model and then if you don't know what dictionary vectorizer is you will also see right now what exactly this thing is so i will start with importing the pickle file the typical model so this is the model for loading like saving and loading python objects and i will load the model that we just saved so this is this is the file that we just saved and i will load it in the reading mode and in the binary mode so rb then this is how we open files in python and then i will just say so pico lot and this file f in and then it returns so this thing returns so if i just load this it loads the file and returns something and returns it returns what we saved which is dictionary vectorizer and the model so i load them so now we have the dictionary vectorizer and we have the model right and now we need to figure out how exactly we can use them to make predictions so let's say we have a client who is our customer right and we want to identify identify if this customer wants to leave us churn or not and i will just take a random customer i don't know customer number four from our test set so i will save it as a dictionary or show it as a dictionary so this is the customer who did not churn so she stayed with us for 10 months and she still stayed so and this is uh so this is the description of our customer right so i now will take this so it will be our customer and yeah so we have this customer and now yeah we know that she actually didn't turn but let's pretend we don't know if returned or not right so we have here in our customer database and we want to understand if she will churn next month or not or what is the probability that she will do this next month and we have this dictionary object so what we need to do with this dictionary object is somehow turn it into something we can put into a machine learning model and for that this is exactly why we use dictionary vectorizer so dictionary vectorizer can take our dictionary and can turn this into a something that we can put in our machine learning model so then in terms the dictionary into zeros and ones right and this is exactly what we can put in oops we can put in our model and then i do model predict so i do model predict probably because i want to predict probability and then i see that for this customer the probability that she will churn is 23 so i actually need just this number right so this is the probability that she will turn and i can just say that this is y prediction right and then i can do something like red just print it right so this is how we save the model this is how we load the model and this is how we take a customer which we want to score and this is how we score the customer and then we can implement some logic based on i don't know whatever rule let's say if this is higher than 50 then we can send a marketing email to this user right so this is what we've done so far let me take a sip okay so now we've we've done that and now what we will do is we will create a virtual environment in which we will take this thing and put it into flask into a web service so i'm going to terminal right now so this is my terminal um and this is the same location where i where i open notebook right and what i have here is yeah just a bunch of things and i already have p-pen so p-panth is the tool we will use for managing virtual environments and we want to create an environment where we will have all the dependencies we need for this particular thing so to create an environment we do pip and ins people and install and we need to install a bunch of things so the first thing we need to install is scikit-learn right because we use this model and this dictionary victorizer they come from scikit-learn so we need to install scikit-learn and we need to be careful because we want to make sure that this version of scikit-learn that we have in our virtual environment is exactly the same as we used for creating our model so i will just do quick import i will import circuit loan and i will see what is the version of circuit learn that i used for um for creating this model so this is the version so now when i do pip install scikit learn i specify the exact version that we need and then we also need a bunch of other things so we we need flask for serving it and for now let's just leave it at this so just take it learn at last yeah and i don't know what's happening here so i think uh because in this folder i already created something so i'll do prepend rm okay yeah because i think in this folder i did a workshop some time ago so typically you will not need to do this ppm actually another thing that i might need to do here is specify the version of python so for that i will do something like python i will use 3.9 for this so we can specify the version of python we require for the project here when we create two and you see that it says that we use 3.9.7 because i also on this computer i also have python 3.10 and 3.10 is a bit too fresh so i don't know if all the libraries will work here and also in this folder i have visual studio code so this is where we will create code right now and the first thing i will do is i will take the notebook that we created here and i will turn this notebook into a script into a python script so first i will rename it so i'll call it i don't know predict so this is the name of the notebook we have it here and now i will turn this notebook into a script so do jupyter and be convert so this is a common for converting an output to script to script and then what's the name was predict all right so now what will happen is we will create this predicted pi file which is here and now i'll just clean this file a little bit and that's basically what we did in the notebook right so we have customer we have uh this bit of code so loading the model applying dictionarizer applying the model and then pre-printing printing the prediction so i think that's pretty much all it does right and now i want to execute this code from our virtual environment for that i will do pip and run or i'll do peep shell so this kinda activates the virtual environment so you see we now have this deployment workshop here i think and if i do which python it tells us which exact python is used for that so it's python that is that lives in this virtual environment okay that doesn't matter so like it's not important maybe if you're not really following what i'm saying here like here just i am making sure that we use a separate python of the system python for now for what we are going to do here anyways so here now i will um execute there are some questions right nathan yeah so let me just quickly execute this thing and then we will cover the questions so i'll just do python predict so we're doing this in a virtual environment and we see the predictions and yeah what's up um so uh this person is just wanting to know if this is a virtual machine or not it is a virtual machine yes um because i am on windows you might see uh yeah this uh windows uh and i um yeah i just had a virtual machine in aws so which is this virtual machine you can see the api i hope you don't use this ap in any way but it's a virtual machine doesn't mean you have to have a virtual machine too you can just do everything from your so on windows i use git bash for everything so this is just you know it's like a linux like environment but for windows or alternatively you can use so on windows i have vsl which is windows subsystem for linux that is like usual ubuntu but on your windows but for different reasons mostly chrome eating all my ram and not having enough ram for doing a workshop and then also at some point we will do some stuff related to aws like um uploading images it's a lot faster to do from a virtual machine that's why i use it but for you you can pretend you don't know about this and i'm doing everything on my local machine yeah yeah okay are there any other questions by the way i see there are a lot of things in chat but i don't know if there are here's one other question but i think we should wait till the end to answer it okay okay so yeah so we have the predict script and now we want to turn this predicts clip into an actual web application so something that we can use for making requests and getting responses with predictions and for that we will use flask and i just want to quickly check yeah so i want to look in the book to make sure i don't forget anything so yeah so let's turn this into a flask application and for that we already imported flask we already installed flask and i'll just do import flask here sorry not like that from flask we will import flask and a bunch of other things i will need to take a look what exactly do you have nathan are there any questions there are more questions now yeah so what's up so uh while you're taking a look is oh and i'm not going to say this right right is pipe nv like virtual envy yeah it's exactly like that so p pen pipe so it's p pen so p panth is uh a wrapper around virtual and so it just makes it easier for you to interact with virtual environment but it's nothing else as like a wrapper around virtual environment so i actually wanted to show you this file too so this is what p pen created when i did p pen install so when i did this uh here right so when i did this it actually created a file this file where it specified dependencies so it is similar to requirements.txt very similar except that it has a bunch of other things so for example also python version and more importantly it has this pip file lock file right and here we it locks specific versions so for example for flask it expects that the version of flask is this particular version of flask and these are checksums of files coming from flask item if something happens like when you try to install and the file is corrupted bpen will tell you we'll let you know about this so this is quite useful so even though we when we say here so when i say pip and install flask um when you usually do this pip usual pip by default just fetches whatever latest version is available pip is doing the same but it locks the version so when you use this pip file lock the file lock then it will install the same version for flask as i did which is very handy and then like if it works on my computer it will also work on your computer and i will not be able to say but it works on my machine because it will work on your machine too hopefully so so does the same deployment process work for deep learning models as well to some extent but yeah i'll have to make a shameless plug here sorry but in actually in the course we cover so deep learning is a bit special because the libraries for deep learning their behavior in principle you can follow the same process as we show here but let's say if you use tensorflow then you can use things like tensorflow serving and we actually have two modules about that serverless and kubernetes which is which are about deploying deep learning models but again this what i show here will also work but there are better ways okay end and then we'll do one more question and then let me get back to it um can we use uh some free platform other than aws to predict yes of course so everything i show here except the last part we is platform agnostic and on the last part i will show you how to deploy it to aws elastic bean stock but this is just one of the options so with a docker container you can deploy to pretty much every cloud that supports docker container deployment and there are many of them you know so aws is one of them but you can also do it with jcp or i don't know i don't know or have your own local kubernetes um with hiroku it's a little bit different so you don't need a docker container but most of this stuff will apply to heroku too for example or to python anywhere or i actually don't remember if i can deploy docker containers to hiroko but most of the stuff that i will show here today is pretty generic it will work for pretty much everything except for maybe last part where it's very able to specific okay should i keep because i see that it's already like uh half of the webinar so i need to i need to carry a little bit so now we need to import a few things request jsonify so here what i do is i now want to turn this thing into a flask application and i just need to import a few things for that so first of all flask will let us create a flask application so i'll call it up and we can call this application term i don't know if it's really important to give a name then we will not need this because we will send this in a request i will just copy it to a different file and then here this thing we need to put inside a function so call this function predict and this function will just do this thing but instead of simply returning the prediction so we will call it result and then in this result we will say churn probability probability and let's say it will be wipe red and yeah and at the end i will just do return jsonify so jsonify just when if i turns the dictionary that we have into json object so this is the format we use for communicating with between clients between web services so the client uh sends a request in json then we the web service gets this json parse it turn it into the python dictionary then execute this and then convert the result back to json and send it back to the client and yeah as i said the customer is also uh we receive this customer as json and for that we need to do something like request dot get json i think so i don't have auto complete here i hope i'm typing incorrectly and then now we need to say that this function is actually an endpoint endpoint is like we just want to say at which address this function lives so i'll just say app dot route if you ever created an web service you probably know what it means if you did not and you feel lost it's fine because later like in a couple of minutes we'll show you how exactly this is going to be used so here i just write so this is address how exactly we will access this function and the http method so there are many different methods so we will use post so if you don't know about them you can google http methods gets get post put delete like things like this and then you can see what exactly it means so for this one we will send request our customers using the post method using this address and that's pretty much all we need to actually start serving our clients to start serving our model uh with flask and there is just one last line when we need to start our flask application so we do up run debug true debug 2 helps us let's say if we make some changes here in this code because if we run it in debug mode it will automatically restart it and then we will also need to say host so host is zero zero zero zero so it's important that it's not localhost but zero zero zero zero um because like there is a distinction between localhost and zero zero zero zero uh when it's zero zero zero oh it's just too many zeros anyways when it's like that then um yeah it will serve like all the clients okay i i think i like if you're really interested in what's the difference you should google it and there is a good stack of answer about that and then port so this is the port that is listening to you it's 96.96 and then if you know python you need that if we want to say to run something then we need to do things like this right so this is our main block in python and i think that's pretty much it right so i think so we import flask we don't need json so we import flask we import pico we load the model we create a flask cup and then we put all the logic here in the predict function and then we turn this predict function into an endpoint and then we start our web service now let me run it so to run it what i will do is just run this again python predict and you will see that it starts a flask application on this address and on this port so now i will go back to our jupyter notebook which is here and we'll use a special model called requests for making the request to this web service the url which we will use for sending the request is http uh it's localhost 9696. so this is the address where we will send our request and the actual address is predict right so this is the cost this is the port and this is the endpoint that we use so this will be our url and we will send a post request so we'll use method post then this is the url and this will be the customer in the request that we will send to our web service so copy it so this is how it should look like post then we send this data in json customer equals json and i don't think we need to do anything else so i just execute this and yeah so it gives us some response so actually i need to save this response to your variable so we can later see what the response actually was and response was in json so we need to parse this response and yeah that is the response right so this is how it works okay yeah that's uh that's it so we were able to send a request and get back the response i'm actually surprised it works i was expecting it would fail here but okay that's good what's up oh we have a ton of questions yeah if we have a little bit of time um so uh let me get to this is from bernardo um there's been some rivalry between the ben and poetry due to dependency management do you use defense in production have you had issues with it so we do use bipenthe in production so we're at work where i work at alex we every projects uh every new projects use p pam at some point we just decide okay we just need to stick to one tool um but yeah we didn't have any issues the only thing i don't like about pipe is about sometimes it's low so when let's say you already have 20 dependencies and you want to add 21st one then it's sometimes slow right other than that yeah i didn't have really any issues with this okay and uh this is from mehdi is pickle always the preferred way to save the model what other format would you recommend yeah actually it's not always the preferred way so with pickle you should be careful so you really need to trust the source so if you know that this pickle file is coming from your job then it's safe to load because with pickle it's possible to add some arbitrary record that will be executed so you don't you shouldn't unpickle any random files you found on the internet right so don't do this um so that's kind of not safe but if you know that you produce this file then it is okay so that's this warning apart from that pickle is just easy to use it comes like it's the built-in package that is everywhere like you can every python has it so i don't see reasons not to use it apart from pickle there is job leap which is pretty much the same thing but i think yeah you can also use this so it's instead of pickle you would just import job deep and then use job flip for saving the pickle file there is like slight difference i don't remember which difference is it and then there are formats like pmm8 or something like this when you actually save the model as xml file not pickle which is like more cross platform and then it definitely has no like if somebody wants to hack you they definitely have no way of injecting any random code in this pmml file uh but yeah pickle i think is just the easiest one and can we add an emailing function in flask not in flask so actually so here in this example so this is the client i know custom customer and client is confusing so here by client i mean let's say we have a marketing service the marketing service uses our churn predicting service so for each client we have in our client database it's for each customer it has it sends a request to the churn service and gets a response and then this marketing service can decide okay if the probability of this or actually what i should do instead here i can have a variable called churn which is if y spread is higher than 0 5.5 then i return churn churn right so i will now if this thing returns true then the marketing service will send the email so it's not the this service that sends the email but the service that uses this service i know it's confusing but it's not your flask's job to send emails but the users the people the other teams who use your service it's kind of they not responsibility but like this is what they should do so they use your service for predicting churn and then they do what they need with this thing and now let me try to i see there is an issue i think i might need to restart it now when i do this we see okay now this is the problem i set we should have because now i need to turn this into i will explain you what i'm doing here but let me just check that it works and then i will explain what what's happening uh okay live demos are always like that okay cool it's working so we see that it's churn false here it means that this service here we assume that we are marketing service should not send an email but if if it's if it was true then it would send an email so we don't do this from here and then coming back to this problem so here we saw this we're weirder object of time bull underscore is not json serializable so if you see this for the first time it might be super puzzling like okay what the hell is going on here and the reason you see this is because this bull underscore object is coming from numpy so numpy is something that psychic learn uses internally for like this actually so this y y print is a numpy number not the usual python number and yeah just json library that we have built in python it just doesn't know what to do with these numpy numbers and we need to help it a little bit so we say that it's actually deluxe this is a usual number relax this is the usual boot type so we explicitly convert it to built-in python types and then it knows how exactly to serialize them so that's that's what's happening here so we turn this into a float instead of using numpy float it's a python code so these are different plots and then this one actually this one should already be a python float so like there should be no need to cast this here i'll just in case keep it here do we have any other questions we do but i want to make sure that we have time to okay so because now we actually we already have a web service right now we can interact with this web service from our let's pretend this is our marketing service right so we can already interact with this from our marketing service and we can make decisions here so for example we can see as i said if churn true then we send emails we will not write code here for sending emails but yeah you can do that too if you want and yeah so we have that and now let's deploy this to docker um maybe before that maybe before that uh we should also like why it's doing this and before that i want to talk a little bit about this line so you see it says this is a development server do not use it in production deployment use a production vsji server instead so what it wants to say to tell us here is that flask comes up comes with built-in server web server but it's very limited so you can only use it for testing your application but if you want to deploy it it will not be able to handle your load and it says that yeah for testing it's okay but if you want to deploy it use something else so what this something else is is like there are actually many so let me stop it yeah i stopped it so there are many there are a few vsgi servers we will use here um g unicorn which is one of them so if you're using windows unicorn will not work and you will need to use a thing called waitress waitress python it's pretty similar to genicorn we will use unicorn here uh and later on like even if you're on windows because later we will put this in docker and then in docker it will be linux so you can just skip the part where you test it locally if you're on windows and move directly move on directly to windows so i will install g unicorn here so we pip deep end install g unicorn and then yeah it will take some time to install so this is the part about pens that i don't like so it's doing all this resolving dependencies and sometimes it's taking a bit of time right yeah when it's just a few dependencies it's not that bad so now i want to show you how to use the unicorn to serve our model and again if you're using if you're doing this on windows it will not work it will complain about some missing dependencies so just you can skip this part and later we will do this in docker where it will work so unicorn i think bind we say which address we want to run it so it's the same host and port like we did previously and we now need to say which where this app lives so it lives in the python in predicted pipeline in predict dot pi file and the up uh the variable that will let me just open the code so here we need to point it to this thing so we need to say where this app lives and this app lives in predict.pi file and the name of this app is up so this is what we're saying here so go to predictedpie and take a variable called up so now let me start it and yeah so now it's telling us something that it started the unicorn and it's running on this thing so now we can go back to this thing again and i will send another request and then yeah we get a response so this is our response now it's not coming from flask but it's coming from the unicorn that is running flask but not from the flask development test server okay next step i i don't know like maybe a quick uh there is one question i can quickly answer before i'm moving on to docker sure wait a new one um let me find i had one lined up for you um so maybe that one's saved for when you do docker um could we use and i'm i'm hoping i'm saying this right as well can we use onyx instead of pickle yes you can use so in an x instead of pickle when x is just i think a format for saving models i usually hear about online x in context of deep learning models i think you can also use it for simple models like the one i am showing you here like for linear regressions the linear or for logistic regression i have personally not used only next for this thing but probably if it works for logistic regression then you can use it as long as you can use something to save it and then load then you're fine you can use whatever you want but it should be able to save the model and then it should be able to load the model and the rest is not important yes okay so let me go to docker so docker i will not be able to give you like uh a good introduction in docker into docker it actually should be docker file so for docker we need to define a docker file and this docker file explains what exactly we need to install for our for our application to work right so we just need to specify all the dependencies we want to install and usually when we define a docker file we say like what is the base image so there are base images that we use that contain something so for example here we use python 3.9 so we will take a base image that already has python of the version we need so i can just write here from python 3.9 so python here means the name of the image on docker image and 3.9 is the tag and you can just google python docker images to see what kind of images are there so this is the right link and if you go there and click tax uh you will see that yeah there are quite a few of them so for example this 3.9 13. so it can be like a minor version and then there is this bullseye bullseye is a version of debian which is a kind of linux here actually it has all these things here so what i typically use is i use this miner slim thing so i add minus slim here and slim here means that i think it's just smaller image compared to python 3.9 i don't know how exactly it's smaller and why what's the main difference between them but i go with slim because it's smaller and this thing has only python it has pip but doesn't have anything else so now i need to install p pen i don't think i showed you how to install pen i will show you now so i will need to do something like pip install p pen and it will install p-p-pen for so now after executing this thing we will have p-p-pent in our docker image and now we can copy our pip file and pip file lock to our docker image and then install all the dependencies we have in these files so we'll do copy so this copy command commands the copies things from our directory here to the docker image and since we have two files i will use this syntax the first this is the file we want to copy pip and then we want to copy two files it's pip and lock and then the last one is where exactly we want to copy these two files so we copy them to the current directory which reminds me that we need to define what the current directory is we use workdir for that so i will use workdir app so the working directory is the directory where we execute everything so for example right now my working directory is this one right and like that here we say that the working directory for this image this is the place where we will copy these two things is up so it will do something like it will do cd slash up and it will copy these things there right so we have this and then so we copied this pip file and we want to install these dependencies from there so i will do now run pip and install and it will install the dependencies but in docker so i didn't really tell you what docker is i kinda assumed you might heard about this so docker is a way to isolate the application you have from the rest of the system so it's like you have your so i have my ubuntu running here and inside this ubuntu i have a container that is isolated from everything else so it's kind of its own thing so it has its own ubuntu or i think this one has debian its own dependencies that this specific application needs and nothing else right so now we need to make sure that we install the dependencies we want which is flask and circuit learn to this specific docker image if i can interrupt you for a second um someone is is up asking uh should you copy pip file and not pivot right yeah that is a mistake indeed good job thank you yeah so i would discover this mistake later but thanks for noticing uh because otherwise later we would lose a bit of time here so people and here i actually also have a type one and here and so many typos so run people install but as i was saying so this thing is already isolated so here we do not need to ins to create a virtual environment so here we have a virtual environment which is kind of like having uh dependencies just for our project but in docker it's one more layer layer of isolation so we don't need to create a virtual environment there we can just install everything to python to this python to the system python inside docker so i hope i didn't lose you here but what i'm trying to say is we don't need to create a virtual environment here we just install everything we need to to this docker image and for that we need to use two flags so one flag is deploy and the other flag is system i don't remember which is what is the difference between them but the main the main thing here is that when we do this it does not create a virtual environment and it uses the system python for installing the dependencies right okay now we install all the dependencies that we needed the next thing we need to do is to copy the model file and the predicted pi file so i will do again copy so first it will be model c 1 2 bin then the next thing will be predict dot pi and then the last will be again like the current directory so what we did so far we copied the pip file we installed the dependencies from this pip file and we copy the model and the predict file now we can run it so for that what we do is we do entry point so entry point defines what exactly how what we what exactly we should run when we start this docker image and here i will just go here and copy this thing so this is what we should run so we should run g unicorn and i will edit this a little bit so here each thing each token should be like its own like this is just how we need to do so we should each token in this command should be its own work and then we separate them like this i don't know why we need to do this but yeah we just need to do this and also we have this port right so this is a port 9696 this is the port where our application is listening for incoming requests and inside docker so in docker this docker container as i said it's a thing that is isolated from the rest of the things in our host machine so we need to explicitly say that on this container this port should be open so then we can send requests to this port and for that there is a special instruction that say expose it means that this port should be open so i think that's all we need to do for this docker file now i want to build this and hopefully i don't have many typos so it should work so just go and do docker build then i do docker build minus t so i need here i need to specify the tech name um so this is how we will know how to defer to this image we are building so i'll just call return service i guess sure this yeah and then the tech could be like 0.1 something like this and then dot so dot here means that use this folder where we are currently in to copy like all the files that we here specified like pip file and so on so now what it's doing it's first downloading the model this python 3.9 slim the base image now it's installing path so yeah it's installing vm then it copied the pip file now it's installing dependencies from the pip file from pip file lock and it is installing it to the system python so it's not creating a virtual environment and the next thing should be it should copy the should copy these two things right expose the container and now it's finished so now we can try running this so this is how we tagged this server this image so i'll just do docker run and then this server it's actually this this name so this is not enough i need to add a few more things so i need to add minus i t uh we want to make it interactive and this is needed if we want to stop it from the terminal otherwise we will not be able to stop it and then i will also add rm rm means when we stop when we finish when we stop our container it should remove all the temporary files we created so let me run this and now we see the same command as we saw previously but now the difference is it's not running on our host machine it's not running on our in our virtual environment but it's running inside the docker container so this thing is self-contained and it has all the dependencies it needs and now let's test it so i'll just go to this thing and the url is still the same because it's running on our localhost and they will just send the request and something is wrong so let's see so there yeah so what i forgot to do is i forgot to add port marking which is pretty important we did expose a port already 9696 but we didn't map it to a port on our localhost because i am still sending requests to a localhost right to our host machine and docker needs to know that if i send something to this 9696 it needs to forward this request to the docker container and for that i need to use this syntax so it's 9696 the port on the host machine and then the port in the container so we want to run the port on localhost 9696 to the port 9696 in the container now let me run it hopefully now it should work let's see okay now it worked and it gives us the predictions right so let me see how much time i have so i don't think i have actually time to show you how to do uh cloud deployment now or it's okay if i take a bit longer to do this um as long as there's time in your schedule um it's okay with us okay so what i'll do so i will not take questions right now so i'll quickly do this thing and it takes some time to actually deploy it so while it is deploying i will take questions and it should take 10 minutes max i hope so let's see so i want to use elastic bin stock for this so now we have docker container and with this docker container we can deploy this model everywhere anywhere we can anywhere we want and like this anywhere of course should support docker images so it should support deployment with docker images so i will go to my aws account and i will go to elastic bin stock so this is the simplest way to deploy docker container in aws maybe there are other ways but this is what i will use and for this i want to use a special tool uh special cli tool which is called it's called elastic install see an elastic beanstalk cli so i will maybe i have it already oops eb no i don't have it i'll do peep install actually i should use ppnf install i will not use it i will just just one second i want to make sure that i install the correct thing so it's it should be aws cli aws elastic bean stock cli that's quite a name right so i will install it using people and install pip install so ideally i should use p pen for install so you can pretend i actually use p pen to install for this um because we want to make our virtual environment self-contained we want to make to it to contain all the dependencies we have but if i do this i will change this file i will change this file and this file and then docker will see it and it will need to rebuild the image so i want to avoid that and that's why i just use pip here and now so i have elastic bean stock so this is the the command we need to use so this is elastic bin stock cli command line interface and let me quickly check the version so this is the version i have and now i can say that i want to create a new project with elastic bin stock let's say b init minus b so this is the type of the project so init initialize the project minus b type of the project so for me it's docker and then the name could be let's say churn serving serving yes so this will be the name of the project so i start a project and it created a folder elastic bean stock here with config and then yeah there is just a bunch of things so what is interesting for us for me right now is this default region because i uh i want to use instead of esp2 i want to use ireland so i'm now in so here oops so here i am in ireland so i want to use the same region and it's geographically close more close to where i am now so i will replace this to eu west one which is ireland they will not change anything else here and yeah that's the config so this folder was just created by this command and before i deploy i want to quickly test it okay locally so i'll do eb now local oops sorry local this local run and then port 9696 so it seems to work now let me execute it just to make sure it's still working it is still working good so now i will stop this and i will do eb create uh churn uh serving n so this is something that we will create now in elastic bin stock so i will execute now this thing so right now this will do a bunch of stuff on aws which maybe is not super important to know what exactly it's doing but in essence it's just taking the model we created this the image the docker image we created and it's pushing it to aws where it will be served so it will take some time so now let's take the questions in the meantime sounds good so um can can the same functions work on a mac as well yes everything that i show here will work on windows with a slight uh thing that i said that ginicorn does not work on windows but apart from that it all works on linux windows and macos and what can happen if we use flask run instead of unicorn in production so what will happen is with the moment it gets two requests at the same time it will not be able to serve the second request so only one request at the time and it's slow um yeah basically it will just be it will not have the performance you expect to have so it will not be able to handle the load with unicorn it's more flexible you can say that okay i want i don't know eight processors to handle this request so then it's yeah you can read some documentation about like how to tune unicorn but in essence it's just a lot more performant than just plus flask and uh will the code be shared i yes it is already shared so you go to ml zoom comp and in this it redirects you it redirects you to this repo to this folder and then you go to this five deployment folder and then here you have code and then all the code that we discussed here is there so for example let's take a look at dockerfile it's i think it's exactly the same let me see except no i think like it's identical like i don't know how i mention oh no like this one and this one the order is a little bit different but yeah it's and the version of python is different but yeah like the code is there maybe with minor things but yeah check it out okay and um where can we watch the recording you can watch it on youtube at um data science dojo's channel as well as on our website online updatesciencedoctor.com um how can we perform model monitoring for the cases where the data isn't labeled okay that's quite a packed question so uh i will do another shameless plug so we have a course which is about envelopes so mlops is like a set of tools and practices or it's more like set practices and then tools to do to predictionize machine learning and then one of the things there in envelope space is like after you deploy how do you actually monitor your models and then in this course we talk about envelopes and different components and practices and one of the models there in this envelope zoom camp is monitoring so check it out it has answers to your questions i don't think we have time to actually answer it right now it's a whole other webinar yeah exactly or two yeah um so uh this is from jacob alexia i was wondering if there are any other follow-up resources that you would recommend checking out um jacob is in college and free is preferred please prefer it yes so this ml zoom camp is quite good uh of course because i am the author but not only uh so one of the things you can check out so here i like i just showed you this is how you use pen this is how you use docker right i didn't really talk about what pip envy is what docker is so then like this is the same module in the same course i have already mentioned which talks about you know what pickle is like what is flask what is pitman what is docker in more details like there it explains like every line what exactly they are doing so here i couldn't cover all that but check it out if this is something new to you then if it's not new to you you can check like serverless for example this serverless module is where we deploy a deep learning model to aws lambda so this is pretty cool and kubernetes is also something i recommend checking out so this is like you have your docker image how can you deploy to the kubernetes and then it also explains what kubernetes is what kind of things are there and how to take the model and deploy it to kubernetes so check this out apart from that i don't know check out our data docs club which is a cool community and we have a lot of events like like similar to data science dojo let me actually check if sorry i just see that it deplo it's deployed so what i will do now is i will show how to use this it will not take a lot of time so i just take this url and i will go to our thing here so i will put it here so this should be http here we don't have a port elastic bean stock by default uses port 80 so we don't need to specify the port so i replace the url i keep the customer the same i sent the request and i get back the response so now instead of going to my local machine it actually went to elastic bin stock so this is how you can deploy it and it took let me see so it finished it at six minutes and we started so it took like three minutes to deploy it which is not a lot but also not immediate and now i want to show you how to actually turn this off because remember this is actually paid so you are paying so this is not a free service maybe there are some parts that are free but let's assume that it's not free like there's the thing called free tier but i don't really know how it works so like if you don't turn this off you will be charged at the end of the month you don't want to have a surprise at the end of your month so please terminate your environment and this is how you do this so there was a thing called terminate environment and you just terminate it and you will not be charged at the end of the month with a lot of money it will not be a lot but still like you don't want to pay like 50 dollars for something you don't use right okay i think that's actually it for me so it took 10 minutes longer but i hope it was still worth that yeah definitely all right i can take a few more questions if there are any sure let's do maybe five more minutes of questions and no no okay so um can this be applied to neural networks or computer vision models yeah we talked about this yes and no so first of all tensorflow for example if you use tensorflow it's a gigantic it's a huge library it's not very easy to just take it you can do this you can take the flask up that we created here and just put tensorflow there but it will be super slow because tensorflow is a huge it's like two gigabytes library so you don't want to carry this kind of library around what you can use instead is tensorflow lite which is a lighter version of tensorflow which only has this serving part and this is what you can put inside flask and then again so this is something i talk about in [Music] here in serverless but it's not only about serverless it can also be applied to flask up that we created here like this tensorflow lite part so for example if you want to use flask so you can check this tensorflow lite thing like just google it you don't have to watch it here you can also watch it here if you want but it's pretty straightforward so you take your model you convert it to tensorflow lite format and then you can put it inside your flask application okay and do you have any opinion on fast api i unfortunately do not i really need to try it like everyone says like why do you use this flask like it's not cool anymore like cool kids use fast api so i'm not a cool kid but yeah i've heard many good things i just need to try it i am just out of habit use flask but it doesn't have to be flask it can be fast api or any other web service and then a question about your book does it cover the deploy the deployment of tensorflow yes it does okay and um why waitress over unicorn us why so waitress is like uh like if you want to do this on windows you use waitress right so if you don't do this on windows um yeah you don't have to like waiters kind of works everywhere then the question was uh like let me just open it so i make sure i really understand it was it the uv corner or gui unicorn because it's like unicorn yeah like uh uv corn also works on windows i guess uh just didn't test it i heard it's better but like to me they all look the same right so i don't know what are the differences between them i didn't do any benchmarks to say that one is better than another i know that you g unicorn works i know that waitress works i know that uv corn works all of them will work with lusk so yeah there are might be reasons to prefer one other over another i don't have them you know okay and is it possible to train is it possible to train the model with updated data also in the production model and not just predict with already pickled or saved models yeah so again i would refer to ml ops zoom camp where we show how to constantly retrain a model so the process for doing this is called like you create a machine learning pipeline so in this process you can easily retrain your model and then you can update the model that is running in production so for example let's say we have we have this model here right but let's say we have some machine learning training pipeline that uh trains a new model and then we can automatically deploy our thing like our service when a new model is created so again this is more like domain of envelopes where we automate these things but like you should have very good reasons to implement this because it's not easy it's difficult to set up this process it's fun it's uh like we engineers like tools experimenting with tools but like the question is whether you actually need it maybe it's okay to use like this model and update it manually every now and then so yeah and if you ask this question or also have this question uh we actually have a webinar next week called putting analogues into practice it's at noon on august 31st with a senior cloud solution architect at microsoft um so if you have this question or also was like an answer to it i would i would suggest joining us next week yeah they will probably show how to use azure right probably um i'm not playing anything against azure actually at microsoft while we are talking about this they have a really cool article which is called ml ops maturity levels i think maturity model yeah so this is the first link you have is the article from microsoft machine learning operations maturity model so check it out it's really like it outlines what exactly you can have at different maturity levels what kind of automation levels what kind of people on the team you can have probably that person will tell you a lot more about this um but yeah it's a cool article maybe to prepare for that webinar take a look at this it's a really cool one awesome okay alexi i think that's about as much time as we have um so thank you so much for being here and and uh giving this this uh this webinar for for the community and thank you everyone for coming i just want to steal the screen here for a second and actually talk about next week's models or models uh webinars um so i already mentioned um putting ml ops into practice this is august 31st at noon um and this is with and i'm going to butcher his name as well pythian string hold and i kind of apologize if i if i said your name incorrectly but uh he's a senior cloud solution architect at microsoft and he's going to be talking about putting in a lot in the practice and then we also have um nick yes barrett's with us for the seven ingredients in every great chart this is friday september 2nd at noon pacific time and nick is an independent educator and author specializing in data visualization dashboard design um so join us next week for these two um especially if you have that question and maybe uh my team we can send an email out with that uh article that um alexi just shared with us about uh model maturity um so thank you everyone for joining alexi again

Channel: Data Science Dojo

Views: 9,159

Keywords: machine learning models, machine learning model deployment, ml model, flask, docker, pickle, Pipenv, AWS Beanstalk, Jupyter Notebook, deploy machine learning model, docker tutorial, flask tutorial, ml model deployment, model deployment

Id: KTd2a1QKlwo

Length: 73min 0sec (4380 seconds)

Published: Thu Aug 25 2022