How to Schedule & Automatically Run Python Code!

hey what's up everyone and welcome back to another video so in this video we are going to look at different ways we can schedule and automatically run python scripts and we'll start very simple we'll look at how we can run our code every minute every five minutes every hour every day on our local machines and that will work with whether you're using mac linux or windows then we'll take it a step further and see how we can automate those scripts on a cloud server so maybe something like amazon aws or microsoft azure and then finally we'll look at serverless architectures to automatically run our scripts that's a little bit different a little bit trickier to set up but these different ways are are really valuable to kind of know and know the pros and cons of each the ultimate goal of this video is going to be to take that analytics report we created in the last video and ultimately write a python script to generate that report and send it via email each day before we get started i want to say thank you to this video's sponsor and that is skillshare skillshare is an online learning community with thousands of inspiring classes for creative and curious people they offer classes on a wide range of subjects with topics including animation business analytics personal productivity and much much more i'm always trying to improve my video editing workflow so i recently took a class called advanced video editing with premiere pro 2020 by jordy vandeput this class taught me all sorts of new things such as different keyboard shortcuts i can use to improve my workflow as well as cool animations that i could incorporate into each video that i make membership to skillshare costs less than ten dollars a month when you sign up with an annual subscription but the first 1000 people to click on the link in the description will get a free trial to skillshare's premium product definitely something exciting that's worth checking out all right so what are the different ways we can run scripts automatically well the first technique i want to highlight is running scripts locally if we're using linux or mac we can use cron jobs to automate our tasks and if we're using windows we can use the built-in task scheduler to automatically run our scripts both of these techniques are very very simple and straightforward and they benefit from kind of like we can set up our machine however we want and we kind of utilize all of those personal customizations and it's definitely the most straightforward to set up the con of running your script automatically on your own personal machine is that it's dependent on your machine always being up so let's say we schedule a task to run every day well if i turn off my machine for days at a time that task is just not going to run so that's kind of the downside of doing it locally next we have that we could run the same types of scripts automatically on a cloud server so think uh amazon aws microsoft azure google compute engine or something like digital ocean droplets this kind of has the same benefits of our local machine like we can set up our cloud machines however we want and then we can do the same type of thing usually your cloud machines will probably be linux based so we can use cron jobs to automate our tasks on our cloud server the benefits of this is that like we can keep our server running 24 7. it's not going to have that same unreliability of us kind of turning off our machines on and off so it's going to be more reliable the one downside here is that cloud servers can get costly i looked at digital ocean droplets kind of a little price comparison guide you're looking at at least five dollars a month to run one of these machines with very very basic computing capabilities and that's just going to keep ramping up and up as you get to more powerful machines so this might not be the most uh effective method if you're only running this machine just to automate like one task if you have a bunch of tasks and a bunch of other things using the cloud server for maybe this is a good option but uh if you're just using it for a simple like a single task you know there might be other options that work better finally let's talk about serverless architectures and so when we talk about serverless architectures there still is a server involved but basically it means that we can package up all of our code and upload that to some sort of storage and what works what happens with serverless is instead of like always having that on a dedicated server we load in that packaged code only when we need it so if we're running a job once a day we're only going to load in our package code once a day when it needs to execute and so the benefit of this is that we only use the computing resources that we need so it's going to be way less costly i personally had a little fun with this and i wrote up a script to automatically send myself text messages of cute dogs and i've been doing that for the past month and it's cost me zero dollars because i still fell into aws lambda's uh free tier um so it's very cheap because you're only using the competing resources you need but the downside of the serverless architectures i would say is that you're constrained with how much things you can package up into your code and it's also just a harder setup process so all these different techniques have pros and cons all right to get started open up a text editor and save a new file with the name something like cron test.py so we're going to start out by showing cron jobs which works best on mac and linux but i'm actually going to demo it on windows and i'll show you how in just a second but this is going to be a very simple script that will append to a test file so to do that we can write something like with open uh how about we do cron test.txt we're gonna open it in append mode as f and we're just gonna do f dot write something like hello world i'll add a new line just to make it a little cleaner so if we run this code we see we now get this context test text file and i can run that again and if we refresh this we see it has two hello worlds so we're gonna write a cron job to automatically run this script every minute to keep appending hello world so how can we do that and i'll be testing this with windows subsystem for linux if you want to download that on your machine you can go to the microsoft store search for ubuntu download this and then also i recommend probably getting windows terminal and using windows terminal to launch the bash window by default so that bash window is what you get when you download windows subsystem for linux alright so i'm going to open up my windows terminal and you see this opens up a bash window and make this bigger so you can see also keep my code up in the background okay so what are we going to want to do to make a cron job well first off we can do ls to see what files we have here i'm in a folder called code so i'm going to go to that folder and see what i have there so i have the crontest file and this chrometest.txt file well i can rerun this contest file through the command mine by doing python3 crontest.py and if i refresh this contest file we see that there's now three hello worlds so that's cool and if you don't have 3 on your terminal you can do something like sudo apt-get install python 3. i think that should work um all right so now let's get into the cron job how do we automatically have this run well we could edit so if i do cron tab dash l and this is going to work on mac and linux but if you use windows subsystem for linux it will also work on windows we see we get like this kind of it's all comments right now so there's nothing in this uh cron tab that we have here if we do so we can start adding to that if i do cron tab dash e that allows us to edit what script we want to run in our cron job edit this file to introduce tasks to be run by cron and so with cron jobs there's this asterisk syntax that we're going to use so i'm going to write five asterisks down and the first one is going to be the minute the second one is going to be the hour the third is going to be the day of the month fourth is going to be the month and the fifth and final one is going to be the day of the week if i want it to run every minute i can do all five asterisks and that will say every minute of every day of every hour etc so i'm gonna do five asterisks for now and just have our script run every minute and then i'm going to do python3 and then i'm going to do that file cron test dot py and we can escape that and save and now we can do crontab dash l again and we see we have that command we just added to it and we can also check to see if cron is running by doing service cron status and it's not running right now so we can do service cron start it says permission denied so you might have to use the sudo permission so sudo cron service cron start we have to type in my password and now if we run that command service command again we see it's running so we expect in a minute's time that this should um keep running so let's wait a minute okay it's been about a minute and we don't see anything updating so why what did we do wrong well our issue right now is that if we look at our what we wrote it's just kind of looking for this crontest.py file but we don't actually like we're not running our cron tab from the directory that we're working in it's running somewhere else so what we're going to want to do is actually pass in an absolute path to this file so we can do that by we see that our path to working directory is this right here so i'm going to copy all of this i'm going to edit my cron tab and i'm going to paste in that path so now where a file actually is is this path and now it will know where to find it exactly no matter where this python script runs so that should then work and the one other thing that we're going to need to change is that same thing it's not going to know where to write this it's going to write this wherever the cron job is running and i'm not sure where exactly it's running from so what we're going to want to do here is also paste in this path to make sure it's very clear where we want it to run so i'll save this and now it should have updated the script and now it should actually start running every minute and i'll just look at the cron tab again see that's running python3 and it's running it in the right file location so now it's just a waiting game to see if this updates we had three cron tests before we should get four if it's working properly and see look at that we just got our fourth in a minute from now we should get our fifth etc while this is running let's uh duplicate this file and show how we can do the same thing using the windows syntax so we're doing it with windows subsystem for linux right now but the one thing that's annoying is that normally with these cron jobs you can have it run and if i close out this window it would continue running but because we're not using an actual linux machine once i close out this window it will stop the cron job so let's do something similar with windows so i'm going to just save this file as now i'm just going to call it windows test dot py and now instead of using this path i'm going to just make this a variable here so path equals so this was our linux path but now it's going to be different for windows for windows i recommend getting the path through windows explorer so i'm in the code directory here and we see we have windows test.py file and we could even add like our own windows test.txt file to get the the full path in windows we can double click on this folder and copy this and now i'm going to paste this in and maybe we want to and i usually recommend using the um forward slash instead of the backslash backsplash is going to escape characters and we don't want that so forward slash forward slash forward slash and then we're going to want to do let's just call it windows test dot txt and to make it clear that this is a different script i'm going to just say like yo yo yo yo yo and then new line and so we're going to paste in this path and start running a python script to edit windows test.txt here so if i run this file we see we get windowstest.txt so how do we schedule this well we can search for the task scheduler app in windows we're going to want to do go over here on the right side and do create task i'm going to call this schedule code it doesn't really matter what you call this but schedule code i'm going to have a trigger this is when we want the task to run so i'm going to do new and i'm going to have it all right so it's 417 right now i'm going to have it run the first time at about 4 18 and 46 seconds and i'm going to have the task repeat how about every one minutes i think i can do that oh okay it seems like it doesn't let me do that so i'll use the smallest window here i'll do five minutes for the duration of let's say 30 minutes so this is like just like the cron syntax but it's kind of a gooey interface to do the same type of thing and i'll actually make this just to be careful so that i don't miss the run time i'm gonna do 419. okay so one time 419 uh we're gonna have that script run we're gonna have the action we need to actually point to the script so i'm gonna browse for the script it's in code and it's windows test.py windows will know that this is a python file so it will run it with python accordingly as long as you have windows installed on your windows machine all right python installed on your windows machine okay and so now if we scroll down to this viewer we see we have schedule code that is ready to go so it should execute once we hit that 419 time let's go back and now look at what's happening with our cron test that has been running every minute and we see now we have eight hello worlds that looks great we're running it every minute what if we wanted to run it instead of every minute what if we wanted to run that cron job every like five minutes you could experiment with how you do this but if i wanted it to run every five minutes i could do asterisk slash five that would be every iteration of five minutes oh i could do every ten minutes like this etc if i wanted it to be every five hours i could do something like maybe zero here so on the zeroth minute every five hours we want this script to run and you can get very particular with this if we wanted it to run every day we could do like we could specify maybe we wanted it to run at eight o'clock every every day so we could just do asterisk like this would be every day on the zeroth minute at eight o'clock exactly that's how we do that static so you can play around with the syntax i recommend a site called i think it's called cron tab guru to help you write your cron expressions this this like if i do star star at every minute we see if i do star five at every fifth minute every tenth minute this crontab guru site can help you figure out your syntax and make sure it's what you want so if i wanted to do like at every minute on the day of the month seven so the seventh day of the month it would run every minute if i wanted it to run at exactly eight o'clock on the seventh day of the month we see we can do it like that so crontab guru can help here we're doing a lot of things at once so hopefully this is making sense windows you could use task scheduler linux mac you can use cron and i'm kind of showing them both at the same time i'm going to just exit out of here though i don't want to edit that right now because i like it running every minute so i stepped away for a little bit just to see if things would run and we see that if i look at windows test we scheduled it to run six times over 30 minutes and it did that we had a couple extras because i ran a couple other times but that's cool and then we can also see that uh for the past like 30 minutes that have been kind of gone we can see that we've written a lot of hello worlds using our cron tab with that windows terminal all right that is running code locally on linux mac and windows note that the cron jobs that if you were to move this to a server i would recommend using cron jobs on a cloud machine the nice thing about that is you can kind of have it always running you can leave the machine on 24 7 and it won't depend on you like closing a window or shutting down your machine that doesn't matter so now that's transitioned to seeing how we can do something similar with aws lambda which is a serverless architecture so we'll kind of do our own version of this hello world test there and real quick i'm going to actually just kill the cron tab so i could do that in two ways i could you know edit it and delete the line here alternatively i could just stop the tasks if i wanted it to just stop i could do pseudo service cron stop and that would stop it running every one minute so that's something good to know all right moving to aws lambda and serverless architecture let's open up a web browser and we're going to navigate to aws.amazon.com and you can either create an account or if you already have an account just sign in to the console um okay i'm going to sign in using lastpass okay and so you're going to get to this console here and what we're going to want to do is you can look up lambda to run code without thinking about servers that's what they call it so let's click on lambda and go and create a new function so i already have some functions here but we're going to click this orange button i'll make all my stuff on my screen bigger so you can see and we're going to click create function and i'll just call this schedule code cool we see we also have the choice to do a run time uh we want it to be python so i'm going to say python 3.8 whatever python version you want to use and that seems good create function all right we successfully created the function schedule code so when you're here i don't want to even go to full screen so you can really see stuff uh one of the things you'll see is that we actually have this um code editor available to us within the lambda console so if i scroll down we see that we have this lambda handler function and we're going to want to leave that but we can kind of implement whatever we want inside the lambda handler function so i'm going to actually delete the stuff that they had here and i'm going to just do a good old print hello world here as well i'm going to save that and the next thing we need to do is deploy so i'm going to hit deploy that will actually get us our code you know ready to be executed and i can test the code if i click this test button you're going to create a test event it doesn't really matter if you just want things to run it doesn't really matter what you pass in as your kind of event so i'm going to just leave this as is i'm not even going to just call it test create and now if i run test we see that we get this hello world down here and the logs so that's cool but how do we schedule this to execute on our own and and one thing i want to stress is that we can put anything we want in this lambda handler function so we'll get more sophisticated as we go but how do we schedule stuff how do we schedule this to execute using the serverless architecture well they make it very easy to do that or relatively easy maybe it's a little bit more difficult than like a cron job or the schedule on our local machines but we can click add trigger and we're gonna look up um cloudwatch or yeah cloudwatch events eventbridge here and we will create a new rule and basically using this rule we can like pick what we want to schedule how we want to schedule our tasks so i'm going to create a new rule i'll just call this like schedule code rule optionally i think provide a description and so we're going to do a schedule expression right here as the rule type and notice one of the options that they give us is a cron here so we can do cron and they have a little bit of a different cron syntax than we just saw with the five asterisks uh you can look up like cron aws to see exactly the syntax um so they have six fields and they give you all sorts of examples i'll link this page in the description but if i want it to run every minute i think they might put this somewhere run every 15 minutes this is basically what i want but i want it to run every minute i can do an asterisk here and then i just have to put a question mark here because it doesn't allow you to have both day of the month and day of the week defined you also get to define a year here so let's go back and do cron star for every minute star for every hour star for the day of the week um star and then i think it's question and then star so let's see if that looks right so i was basically trying to copy this one except make this every minute star star star four stars then a question then a star so four stars question than a star add that rule so now this is going to run our print hello world every minute and we see that it did execute once this might have been when i previously ran at hello world so how do we prove that this is running every minute well i can go to the event bridge here schedule code rule this is going to take me to um amazon event bridge let's see i actually want to go to cloudwatch so let's see how i get to cloudwatch from here i think if i click logs it will bring me to cloudwatch so we see in if i click logs we're in cloudwatch now if i click log groups we see all of my lambda functions it has all the logs for them and we see the most recent one we created was schedule code and we see that we have log streams here this is each time we deploy a new update we're going to generate a new set of log streams so if this is running properly we should start seeing hello world come in every minute so i'm going to just set this to auto refresh and if this is working properly i'm not you know running anything behind the scenes uh it should after a minute execute that function again and we should see another hello world and look at that we got another new hello world here and if i kept this open so i'll leave it open for one more minute i'll speed up the video behind the scenes all right and we see we got another hello world here cool so it is running properly now that we've seen kind of our hello world for the three different methods we're really kind of working with let's start building something more sophisticated let's start scheduling emails to automatically send and real quick if i want to disable this from running i can go to events rules we see we have schedule code rule i can just go in here and i can disable this so now if we go back to events and rules we see that it is currently disabled so that would allow us to stop our stuff from running as well all right let's go back to our text editor and create a new file with the name something like send email dot py because the focus of this video is on scheduling our python scripts i'm not going to dive into the details too too much of how to send this email in python if you're curious to learn more about that i really recommend corey schaefer's tutorial on the subject he shows you how you can set up your gmail to kind of be programmatically triggered via python and it was a great video it's what helped me ultimately write some of this code so i kind of took what he showed and then kind of tweaked it to my needs so i really recommend that the other resource that's great here is that you can look up the standard email library in python and see kind of the examples that they provide there so i'll link this documentation to the email library in python as well both great resources but i'm not going to dive too too much into the details of what's happening behind the scenes here all right so i'm going to real quick copy in some code to send an email i will put this file on my github page which will be linked in the description so you can get all this exact code if you go there but basically what we're doing is we're importing the necessary libraries to send a basic email in python i'm defining the email address i'm going to send from i'm defining the subject of the email address you know this is where i'm sending from i can put any contacts here so right now i'm just sending an email from myself to myself kind of silly but we could add in more emails as we see fit by just adding items to this list we have the content of the email and then what i've already done behind the scenes and what corey shafer's video shows is that i've set up a token here to log into my gmail through python so that's what i'm doing here and then finally i am sending the email so if i run this script we'll see that it's finished in 2.7 seconds if i go to my inbox we see test email this is a test email from keith kgmit18 gmail.com2kgmit18 gmail.com so that's cool that was just us sending an email very straightforward so how would we schedule this well we could go ahead and do something very similar and you know write a cron tab to do this so if i do crontab e to edit it we had our cron test before now i could just show that we could do send email dot py and that will schedule an email to send every minute and we want to make sure that we are starting the cron service too so i'm going to do sudo service cron start so we expect that if we wait a little bit we'll start seeing these emails and we can do the same type of thing using the task scheduler in windows as well if you don't have cron a machine that can easily do cron so i could also have created a new task here jumping ahead in time a bit we see that it's like been 50 minutes or so and not a single email has been sent again so what's wrong with the cron tab right now well our issue is that if we look at our file again there's the environment variable and so i set this locally on my windows machine but now that i'm in windows subsystem for linux i need to make sure that that is set on my you know my in this linux shell so we could do that by just doing export email password equals whatever the password is but that would be kind of like just for the the single shell that we're running we want something to be accessible by the cron tab whenever uh the the command executes so we're going to do something a little bit different to do this so we're going to set the variable by going into our profile i'm going to go through the very bottom of this so i'll go to the top yeah very bottom of this i'm editing with them there's not much in here but basically to set this variable i just want to do export email password and then i would copy in my password i'm not going to show that to you i'm going to do it over here because i don't want you to steal my password and then i'm saving it by doing wq okay so now we've added the email password variable to our bash profile and now what we want to do is actually edit our cron tab to just make sure that it reads that profile before it executes the command so we can do period space home this is going to be our home directory that's kind of where the profile is stored home slash profile and then we'll execute the command so these two things will ultimately run that file we just edited and then this command will run the send email as we expect and it will now have access to that environment variable and now we want to check to make sure that chron is running micron is running so now that it can read that variable we expect if all is going well to get a new email here within the next minute and look at that we got another test email and if we wait one more minute we can confirm that it is running as we want it to so i'm going to just wait one more minute it's 6 26 and we see we get another new email so i'm gonna kill this um before i get too many of these emails so i'm going to do sudo service cron stop and that has now stopped it so i'll stop getting these emails but let's make this email a little bit more useful so right now it is sending a basic email but let's like spice it up a bit what would we actually maybe want to like have automatically sent well i love puppies so i'm thinking let's add a fun little dog picture to our email and maybe schedule that to send daily okay so this is fun little api called the dog api and if we type in this basically use this path it will pull this information in kind of hard to read but it's just like a list of single like list of one and ultimately has a url to a cute little dog pic and if i paste this in we see we get a fun pick and each time we hit that api endpoint we get a new dog so i'm going to go ahead and write some additional code so i'm going to change the name of this instead of send email i'm going to call this send dog email and i'm going to just make some changes so we're going to import requests and then we have the dog info is equal to requests.get well we want that url that i showed that gives you the dog info so we'll get that grab the json from that and then we want to get the first list element because it's a list of one we just want to get the actual like json inside of that and then to get the url we can do dog url i'll just say dog image url equals dog info and then it's the url property of that so that will give us the dog image url now we want to say like our subject will be like doggo or dog of the day and instead of saying this is a test email we'll say uh here is your dog we'll do a new line and then we'll paste in the url oh sorry the url would be dog in or dog image url and if we make this an f string we can just paste it in like that so that's great but that would actually just give us the link we want to actually have html and embed this image in our content so we can do that in the email library doing message.ad alternative and basically we're now writing our html version of the email so if we have a email reader that can accept html it will or if it doesn't accept html email it will take this it will use this one but if it does accept html we'll use the second one so we'll start with the same thing here's your dog do a new line and then instead of pasting just the length we'll use an actual image tag image tag in html and we'll do image source and i'll use big quotations here pass in that same url and then we need to make sure to close off the image source and then we need to make sure that we close our body too look at that so here we are just embedding an html tag with the dog image and to make this a nice size we want to kind of define the size i'm going to just say that it's equal to 300 pixels so i'm also going to define that in the html here okay now we can send that and there's one actually additional thing we want to do is just to find that the type of this is equal to html and i also will link this file in github so that you can easily find it so this is what our final code looks like i'm zoomed in a bit just so you can see easier but we have now we've added a little bit complexity to this we're sending a dog email and it is 6 40 p.m where i am eastern standard time so i'm going to edit my cron tab to just send this one time so i want this now to run send dog email and i want it to run at let's say 6 45 p.m so how could i have that happen i'm going to go back to that crontab guru site and so i wanted to run it 45 want this to be 6 and then i want it to be i guess the day is 22nd the month is 11 and the day of the week i don't think that this matters because we're already supposed to die the specific day um okay so we could copy this in 45 6 22 11 star and that should run it at 6 45 pm uh and as long as we get that set up in here in time we can make sure that that happens um okay so paste i'm gonna paste this in okay okay that looks good and one small thing if i actually set it up like this right now it wouldn't quite work properly because our linux systems linux mac might not do this but on linux it actually defaults to utc time and even though it's about 6 45 eastern utc it is about 11 45 p.m so because i know it's going to default to the utc time i'm going to account for that and make this 11 45 p.m okay and i also just real quick want to make sure that i actually get a dog email when i send this i didn't miss anything so i'm going to real quick run the script see it's finished let's see if we got our dog email dog of the day here's your dog no it broke what did we do wrong see this is good that we tested it um okay we didn't close off we put this in the wrong spot this should have been here now let's try running that again look at that here's your dog and i'm gonna just add another new character here maybe i'll do a html break tag this should get it on a new line so i'm going to do that instead but i think we're good to set our cron tab to run and i'm going to do this quickly because i'm almost out of time sudo service actually let's just change the time a little bit because i'm getting really close so i'm gonna change this to 11 47. and now i'm going to do sudo service cron start see the status okay so at 6 47 this should run and let's just make sure i'm going to run this again behind the scenes one more time i don't mind seeing extra dogs so this is fun look at that that looks good this is kind of the format that i wanted and you could spice this up if you wanted to maybe put a date in the dog of the day here or change up the text kind of make it more customized but so if all goes well we have our cron tab that looks like this that should hopefully send that dog email at exactly 6 47 p.m but we accounted for it being running in utc time so that's 11 47 utc now it's the waiting game one more minute 647 oh wow i uh goofed a bit here i think it actually will read the right time i need to edit that cron tab i was like thinking in am pm terms but we need to think about 24 hour schedules when we're thinking about this stuff so now i'll make it 11 49 and i'm going to try 18 see if it actually works if not it will be uh 5 hours plus or 23 that we want here so hopefully at 6 49 this now runs i think it might not be like because i typed in date and it actually reads um you know the current eastern standard time i'm thinking that maybe it actually wasn't defaulting to utc so 649 now is our crossing fingers time 18 49 24 hour clock hey look at that cool here's our dog of the day awesome so we just showed how we could schedule a specific time for a script to run and if we wanted to edit this even further you know maybe we don't care uh you know what a day it runs on maybe we wanted it to run every day like play around with this cron tab guru to see when you get things to run it's like if i wanted to run 645 every day it would be like this all right i'm going to stop the crown all right let's go ahead and do the same exact thing but now move our code to aws lambda and see how we might execute this there so if i go back to the code in lambda we initially were just writing hello world well all we would need to do and i could say the dog email but let's just start with our basic email i'm going to just copy all this code control c and paste it into my lambda function i want my window to be a bit bigger okay we have all the code there we need to indent it so it's inside of that i'll remove the hello world um import json well we had a couple things we imported we actually didn't need to import json but we did need to import these three libraries so i will do that at the top instead now okay so that's all there all of our code is here the one thing now that we're on aws lambda getting this environment variable will be a little bit different so to do that we can go down here to environment variables we can edit them add environment variable it's the same name as the other ones email password the value is whatever your password is i don't want you to see this so i'm going to blur it out when i edit but i would just kind of copy it from elsewhere paste it in and then save and now we see if we scroll down we see our environment variable again i'm going to blur it out so you can't see mine but it will be able to get that so i can go ahead and maybe we do test email and like just call this like lambda test email just to make sure that we it's clear that it's coming from lambda and i deploy this then i can go ahead and test this see if it runs it looks like it executed properly and if i go to my email we see we got lambda test email from aws lambda so that was pretty straightforward we're now sending emails from lambda and just like we can kind of schedule our cron we could in lambda edit our cloudwatch event so if i go to i think we stopped it so let's go to our logs and go to rules for events we see how it's paused right now so i'm going to go ahead and start that back up and so this will run as we said it before every minute so what we expect now is that we should get another lambda test email in one minute look at that we got another lambda test email and so i i'm going to wait just to confirm that it's running every minute i'm going to wait one more sec if we refresh this we should see our next one should be coming any second now we see we got our lambda test email cool and again we can stop this we could always like go in go to our rules our schedule code rule and we could either edit so maybe we wanted it to run less frequently like every five minutes and it's kind of nice with the lambda here it will tell you like when it's going to execute but you could play around with that but really right now what i want to do is actually just go ahead and stop it so i'm not getting that email every five minutes okay let's continue building our lambda function so before you know this was great we we created our function it sends that basic email let's copy in the code from the dog email so i'm going to copy all this code here um and we want to paste that into our body man it's really like tricky to get all this like the window just keeps getting out of place so that's all in our body we want to indent it so it's there so that is exactly how it was before in the sublime text window and i'm also going to import requests we're going to have that update on the server or i guess not like we're going to redeploy this code to our serverless architecture deploy and now let's test sending the dog email so i'm going to say like dog of the day and for this disease i'm going to say lambda just to make sure that i know it's coming from lambda here and actually i need to deploy that again and test okay um unable to import module lambda function no module named requests so we got an issue with um the requests library and so why is that well this is when one of the tricky things with aws lambda is that because we're like packaging up our code and then like sending that to a server and it's executing like everything that's in that package the server by default you know the server has been set up with python in all python's default libraries but it doesn't have any external like new library so requests we normally have to like pip install requests to get it to run and our aws lambda server that's ultimately taking our package code and running things doesn't have requests on it and if you look around here in the interface um for aws land there's like no terminal window to there's no terminal window to like easily pip install packages and this is something that gets really frustrating and one of the things that does make it a little bit trickier to like run routine tasks on lambda so kind of bear with me here we know we want to run this lambda function so i'm going to real quick just copy everything that's in this file and i'm going to make a new file locally called lambda function paste in all this code okay so that's everything that's in lambda right now what we need to do if we want to actually use the request library here is that we need to get the request library somehow into this folder and we can't just like you know maybe you could upload it or something but it doesn't seem like there's a good upload um folder option so we actually need to do is create a zip file of this lambda function and ultimately upload that with the requests library to aws lambda so we can do that as follows i'm going to create a new folder called package and i'm going to pip install to the package folder the requests library and i do that with the following command i just typed in okay looks like we've got it successfully if i go into package we see that we have all these different things and that's ultimately the requests library all the source code right there in this folder that's what we did with that pip install command i go back so here is my windows console what we can do is we can actually create a like zip folder so i'm going to like move this lambda function into that package folder and then i'm going to create this as a and if we look at that you know we have everything that's in the request library and this lambda function what i'm going to do is i'm going to make this a zip so add to package.zip and now we have all that stuff zipped up and we can go ahead and upload that zip folder that has our lambda function and all of the resources in it uh to aws landa so package.zip save and one thing that's annoying is that we uploaded the folder so let's actually do this again instead of zipping this let's go inside a package copy all this and zip all this so i'm going to actually upload this new package.zip right here because that should not be like nested in a zip folder look at that okay so we have everything there now and we see that we have our lambda function we can edit our lambda function more which is nice but the big thing is that we have all the requests library i can go ahead and test this and it looks like it was successful let's check our email dog of the day lambda cool oh that's a cute puppy awesome so we just showed how we can zip in a folder by using a combination of pip install with the dash t flag and then uh you know making our lambda function and like zipping all the together um we could also do the same exact thing um using like command line methods and i'll show that when we actually package up our analytics report how you can do it like zip using the zip command all right now let's see how we can generate and schedule to send an analytics report so as a reminder in the last video we saw how we could create an analytics report in python and the code for that can be found on my github repo keith galley generate analytics report i'll make sure to link this in the description but if you didn't watch that video what we ultimately created in that was a report that looked like this so it's a report on covet 19 data and you know it has some u.s statistics it breaks down some state stuff it also looks at some global stuff and we learned how we could create that so to get started with this if you haven't already you'll want to get the code so you can clone the repo here or you could also download the zip one thing to note is that i made the changes for this video in the send email branch i'm not sure if i'm going to merge that to master or not so some of the files will be there that i'm going to be mentioning but once you have that locally let's open that up and start writing some code there so i'm going to open the folder generate analytics report i'm already in it and the the first file you're going to want to see how to run is this send report email function so you're going to want to change this path to be whatever you have as your path and you're going to want to in this automate report also change your path to be your your local paths but what we're going to do is let's run that send report email and basically it's just like our other emails we've done so far but now we load in some pdf data using this right here and we add an attachment to our message that is that pdf content and then we ultimately send it just like before so let's try running that all right we're getting an error it looks like we have a key error and the reason why i have a key error is because the way that this repo was set up it's loading from some local data that hasn't been updated with the most recent dates but if we want to get the most up-to-date data we can load directly from the github source of it which is updated daily so let's rerun send report email all right we see it finished in 35.6 seconds we go to our inbox we see that we have this covid19 analytics report i can click on that and we can see that we have the data for yesterday because that's the most recent data we can access and it's pretty interesting to look at like massachusetts here we've shot up in cases 3 000 plus this past day but yeah it has all sorts of information that is pretty cool and it's in an email so if we wanted to schedule that to run automatically we can just repeat the steps we've done previously in this video so we could use the task scheduler uh in windows create a new task i'll call it like send analytics report i can have that trigger and there's like there's some different options here like we could have it trigger weekly um like weekly on let's say monday at you know maybe 11 a.m and all i would have to do is recur every one weeks on let's say monday and you'd think that that would it and so that will then set this up to run weekly on mondays okay and then i would just pick that script that we just run just chose which was send report email okay okay and now we have a task that is scheduled to send at 11 am every monday and similarly we could do a cron job just like we have in the past to do the same exact thing and one thing i will note in the send report email function like you'll have to change up this path uh to be whatever your local directory is where that the code is hosted or you want these files saved and also automate report just does the same thing because ultimately when we're running either the cron job or the when we're running the cron job or the task scheduler we don't run from this folder so we have to specify those full paths so if i wanted to do this in chron the cron job like my linux side of things i would like take this path instead or i guess i want to be in generate analytics report so i take this path as kind of my full path and i would pass that in here and i'd pass that also in the send report email here and then that should be able to run from linux uh i was having some issues with the windows subsystem for linux when i was scheduling this with the cron tab so like i would edit the cron job like this instead of doing dog email i'd maybe have it send you know maybe at 11 a.m every day i would have the send report email function rescheduled but for some reason in the window subsystem for linux i was having some issues with the collido library um but that's kind of a collider specific thing so one thing that i did include and this is important for the lambda version that i'm about to demo is that i also created this simple version of the report that you can utilize uh in case the uh automated report with all the graphs and whatnot doesn't work for you i'm getting into the weeds of things a little bit so don't worry if you're getting a bit lost with what i'm saying really as long as you understand the core of what we're doing and how we're scheduling these tasks and these cron jobs that's what's key alrighty so the last thing i want to do in this video is show how we can turn this into a lambda function so let's take like a previous lambda function that we had and just copy this into our new folder so i'm going to just save another file called lambda function paste all this in and now instead of setting this code we're going to copy and paste what was in our send report email code so i'm going to paste all of this in and now because we're going to be running this in lambda i'm going to actually just delete the path and i want to let things actually save in the temp folder that's uh the only writable space on lambda i think is in slash temp or it's at least one writable place a lot of the things because we're not owning that server we can't actually write in those directories but we can write in the temp directory so this would really i'm making this an empty string so it's really just like that but i'll leave the path in not to change too many things up um okay and now we just also need to make sure that we have the libraries that we need cool okay that's our lambda function next what we're going to need to do is zip all this up so we can load it to lambda all right so how we're going to do this well last time we had to install with pip things locally so i'm going to do that same thing i'm going to make a package or a directory called package and i'm going to do pip install dash r requirements.txt so everything in the requirements folder i'm going to re install it in the package directory and this is going to take a little bit so i'll skip ahead in the video so it finished after like i don't know almost 10 minutes and if we go into the package folder now we can see all the different packages we need and i just want to check something real quick i think we're good to go in our lambda function looks good so now what we need to do is zip all this up and we're going to do it through the command line this time so what we can do is do a zip dash r and then i'm going to create a file one directory up called lambda.zip and we're going to take everything in this current directory in this package directory and zip it into there this also might take a few seconds all right so that finished now we need to go back and we see that we have our lambda.zip now what we're going to do is we're going to add the resources so i'm going to do zip to that same lambda.zip and then i'm going to take the i'll do a recursive here too just to ensure make sure we get all the files uh and i'm going to take that resources folder and zip everything in there to lambda.zip i'm just basically packaging it all up so it can run on lambda then i'm going to want to zip to that same lambda.zip all of the python files so i can do that quickly by doing star.py that's any file that ends in py will be loaded up there i might have some extra files by doing that but this is just an easy way to get things there okay we added that so now that we've added all of those things we should be able to upload our lambda function uh to lam like to aws so actions upload a zip file upload and we're going to take we're now in the generate analytics report folder and we're going to take lambda.zip okay and and this is a big issue that we run into and we're ultimately going to have to uh work around this and un unfortunately lose some functionality um because of this limit so there's a 50 megabyte limit when we upload zip folders and obviously our file our zip folder is way bigger than that i there are some workarounds where you can get above that 50 megabyte limit but there still is limits that we hit with our 144 megabyte zip package i'm going to just share an article real quick so here's an article on hacker noon that gives you some advice that you can try to upload bigger zip folders i'll link this in the description but ultimately i tried to walk through this advice and unfortunately our folder was too big still so what we're going to do for a lambda here is just reduce we're going to just reduce down to size ultimately the thing that's really big is that collido package that works with plotly to save png images so we're just going to get rid of our charts that use collido so in that folder we have a generate report simple file and in that one basically we just get rid of the maps even though the maps look really cool uh they use kaleido which was just too big to fit in with all the matplotlib and numpy and everything so uh in our send in our lambda function we're now going to not take automate report we're going to do generate report simple a couple other small changes i know we're going to have to make we're going to have to import a time zone to use in this okay we're going to import the us eastern time zone this is not like critically important but based on when we run this script it will get a different dates like right now i know that lambda defaults to utc which is like five hours ahead of eastern standard time here in the us so it would be the next day and we'd try to get the 20 seconds data and that probably hasn't been uploaded yet so just a small thing we're doing okay so we have our lambda function packaged up i'm just kind of making some changes just so i make sure that things are being handled properly before we run this also note that in this generate report simple paths are just slash temp because that's where we can write in lambda they're not any sort of special path so let's now package this new code up i'm going to use the command line again so this time we're going to [Music] i'll make another directory i'll just call this like simple this is just the simpler version of our script and in that one we want to install pip install dash t of simple we want to install the libraries numpy matplotlib pandas fpdf let's check what else libraries we use i think it has pi tz and date time by default so i don't think we need those on lambda so i think that should be good and it has os and smt lib and all these by default okay so install those now into the target i'm looking at this actually i think that it might not have pi tz on lambda but it's part it comes installed when we download some of these other libraries so i guess that's what ultimately gives us no issues just thinking ahead date time is a standard library okay cool so now if we go into simple we see we have all these different repo or like source files and now we will create a zip we'll say we'll do go back into that simple folder and we're going to do zip dash r i'm going to call this one simple.zip and we want everything in this simple folder now i'm going to go back and now we just need to zip up that same stuff we zipped up before so i'm going to zip up all the into simple.zip all the python files i'm going to zip up now into the simple.zip all the resources so let's do this because we really only need letterhead cropped but okay and now we should be able to let's look at what our zip file looks like simple.zip we see we have all this stuff in here um that's good and then we see that we have like our lambda function here all of our other python files it's kind of messy but ultimately as long as it runs and can access everything it needs to that's what's important and we see we have our resources here that we can grab to get that letterhead so now let's upload this to lambda upload okay and so one thing that's very frustrating um now that we've uploaded this massive zip file is that we can't even like edit our function but we can test it so we can run our test and if it works then great so we're crossing our fingers that it works that we didn't screw up anything oh timed out okay so you can also set different configurations for your functions so if i go to basic settings maybe we allocate a bit more memory to it and instead of timing out after three seconds let's give it two minutes that should be enough to run that script so we can save that let's try testing it again yay it's exceeded that's huge okay so now we can schedule it in lambda let's just see if we got an email look at that uh we can also schedule it now in lambda to generate and this is just a simpler version of the analytics report so note it doesn't have the graphs but at least we can send something in lambda so now if we wanted to schedule that to run like every morning at 11 a.m we could edit the cloud watch rule to be a different crown expression so we could maybe say okay we want it to run at 11 a.m and this is gmt so if we wanted it eastern standard time you just have to do whatever conversion i believe it's five hours ahead i could check real quick time in gmt so it's 4 45 a.m it's 11 45 a.m here so yeah it's five hours ahead of where i am so if i wanted to account for that in my cloud watch console i would do um you know 6 a.m gmt or i guess no i want to do it ahead i want 16 uh o'clock p gmt and that would be 11 am where i am and so that would see as it says it would run every day at 11 a.m i could save that and now if i activated this it would be scheduled to run every day at 11 am eastern or 16 gmt so that's pretty cool that's a little bit trickier and you could have got a little bit lost there but the whole goal of this video is to give you inspirations for different ways you can schedule this code to execute one thing i want to just quickly mention because this will cause you a huge headache if i don't mention to this to you and you're installing these libraries on a windows machine i'm using a windows machine but i'm using windows subsystem for linux so when we installed if i go way back up i think i might be able to maybe i can't go back into my history far enough but when we installed like numpy and matplotlib and and pandas we installed the linux versions of them because we're using windows subsystem for linux but if you're using windows there's like versions that are slightly optimized for windows and those will actually run on lambda and when i first went around and tried doing something like this i couldn't figure out why i couldn't build matplotlib numpy lambda or matplotlib numpy pandas but it was because i was using the windows versions of those libraries so i'm going to also share a blog post i found on medium that can help you just kind of understand what's happening and how to resolve them easily so shout out to uh ruslan kourniksuk man i bought that so bad uh but shout out to this guy who wrote a great article on getting around using windows but installing linux libraries for lambda so i'll link this in the description all right that's all we're going to cover in this video hopefully you enjoyed this one hopefully it gave you some inspiration for different ways you can leverage these techniques with your own python scripts to recap what we covered in this video we looked at how we can schedule our code to run automatically locally using task scheduler for windows and cron jobs for mac and linux if you did want to run that code and schedule it in the cloud using a cloud server like aws ec2 you can do that using the same cron jobs because you're probably going to be running a linux machine i want to schedule your code to run in the cloud but you don't want to pay for a dedicated server we also looked at aws lambda a server-less architecture to help us do this all these techniques are good in different scenarios so it's really kind of a personal preference what you're working on what your budget is etc to figure out which one is right for you if you have any questions on the content we covered in this video feel free to let me know in the comments also feel free to give me any suggestions for future video topic ideas i also want to say thank you again to skillshare for sponsoring this video as i said at the start of the video the first 1000 people to click on the link in my just in the description will get a free trial of skillshare's premium membership thanks again everyone for watching if you enjoyed this video would mean a lot if you throw it a thumbs up and also subscribe if you haven't already also feel free to check me out on the other socials instagram and twitter i post fairly frequently in both of those places so i'd love for you all to join me there that's all we got for this video thanks again everyone for watching peace out you

Channel: Keith Galli

Views: 31,750

Rating: 4.976789 out of 5

Keywords: Keith Galli, python, programming, python 3, data science, data analysis, python programming, crontab, cron job, cron tab, cron, analytics report, schedule code, schedule scripts, automate code, automate scripts, schedule code python, automating code, aws lambda, cloudwatch, cronjobs, scheduling code, mac, linux, windows, task scheduler, windows task scheduler, coding projects, data science projects, wsl, python 3.8, serverless, run code in cloud, cloud, server, microsoft azure, aws

Id: aqnJvXOIr6g

Length: 80min 22sec (4822 seconds)

Published: Fri Nov 27 2020