30 Days of Python - Day 13 - Using a REST API Service - Python TUTORIAL

hey there welcome to day 13 and this one we're gonna be talking all about api's or more specifically public restful api services but for us it's just api's the idea here is that we want our Python project to be able to talk to other services for reasons I mean there's a lot of services that have api's so as it pertains to 30 days of Python and day 12 what we did was we did some web scraping so we can get specific data that we are looking for now what we're gonna do is something very similar but instead of web scraping we're gonna be using an API to actually get very similar kinds of data or really complementary data that what we've already done so just to remind you in box office mojo com this is what we did in day 12 we actually you know figured out the structure of this data and then grabbed that data and then kind of parsed it and did a bunch of like simple processes for extracting that data to make it in a useful form for us we've turned it into a CSV file now that's really cool that we are able to do that now but what we really want is to use a service that already does all those things for you and that's what an API is all about so we're gonna be using the movie DB org so this is the movie DB org they have a REST API service that will allow us to grab a lot of movie like data in a similar fashion that we did with Box Office Mojo but in this way it's it's so much easier so if you went to the movie DB org and you create an account right so I'm in my account I go ahead and look at my account I can go into the settings section and I'm gonna scroll down to where this API is okay so you actually want to get your API key here right so these keys all three of these are things that we'll want to reference going forward now when it comes to API keys think of it as a user login right so whenever you go to a service that stores information about you you have to log into that service a program like Python also has to log into that service and that's what these API keys do sometimes it's just one key sometimes it's a combination of keys right sometimes it's like they often call it key value pairs or a secret key and a public key just in general keep in mind that these keys are like passwords so you definitely don't want to share them I'm probably gonna have to do something about the one I'm sharing right now but in general we're gonna want to keep this key so let's go ahead and copy this and I'm gonna open up into vs code and I'm gonna make a new directory in here and call it day 13 and I'll make a new file and we'll just call this connect PI and the first one is my API underscore key is equal to that string okay now in general I want to make sure that I have Python requests installed so let's go ahead and open up the terminal and Python 3 - M pip install requests now what request is gonna allow us to do is open up a connection between our Python program and their API service now this is not any different than any webpage itself it's an HTTP request right so we're doing what's called HTTP requests that essentially means an internet connection request it's a standard type of request it's what you do every time you open a web browser and go to any web page and do anything there it's it's typically done with an HTTP request so to do this what we want to do is first of all decide what is going to be our endpoint so what's our endpoint or you know another way to think of this or a URL right so like where is it that we want to request data from and then you know what is the HTTP method that we need okay so those two things I'll definitely explain as we go forward but let's first off try and decide what our endpoint is now the reason I'm posing these questions to you has to do with rest api's in general right so it's not just specific to this service that we're using but in general rest ap is typically have some sort of endpoint or a URL that you will call to get something or do something right and that's very similar to what you do as a human the program does it very similar right hopefully that's fairly obvious okay so back into the movie DB org we want to actually go to developers.google.com three right now version three just makes it a little easier to do connections and do all sorts of things like that so if for some reason the versioning changes the concepts will still remain the same or very similar okay so let's go ahead and take a look at the documentation here this is like well what's going on with this what what are these things mean exactly well I always look at this documentation for what specifics I want from it right in this case I'm looking for movies that's probably obvious the movie DV but if I was doing let's say for instance an API call to a transaction service well like stripe like if I want to run credit card transactions then I would be like okay well how do I actually run a charge so I look for charges or how do do billing or subscriptions right so you want to kind of think of what is it that you're trying to accomplish with this this API which which is encapsulated in what's our endpoint it's like what is it that I'm looking to do with this API you know with the box office one what we did was we wanted to get the box office sales numbers for an even year that's what I was trying to do right in this case I'm just wanting to find at least some list or be able to find movies themselves so I go into this movies one here and the end point or the seeming end point is this right here so I'm gonna go ahead and copy this whole thing here and we'll parse and talk about what this is so I'm gonna make a little comment and say alright I paste that in and I'll just say end point and let's go ahead and close down the Explorer here so we get a little bit more room so I got endpoint git and then a somewhat formatted path right hopefully we are familiar with paths at this point but that looks like a path because it is just like a web page would be so let's go ahead and talk about this what is get referring to especially capitalized g-e-t well with HTTP requests you have a few different methods that you'll probably see a lot of but the main two are get post those are the main ones you'll also probably see patch put delete okay so get and post typically mean grab data post means add or update data it's not always update if the Dune incorrectly they would use put or patch but by and large you'll see get in post as the main thing so it's like getting data think of that as like hey I want to grab whatever it is that I'm looking for so I'll use this method of get now programs are not rocket science in the sense that we can change what all of these methods do on the program itself right so if you were actually creating the API you could have your get method add or update data you could have a post method grab data so you can absolutely do that but rest api is in general will tell you what they are now you could also look up these methods they are HTTP request methods or HTTP methods you can look these up in more detail on wikipedia to better understand them even more but just in general if you ever see get that usually means that it's gonna grab data if you ever see post that means you're gonna send it data their way okay so first and foremost we now know that that's what that HTTP method is you will see this all the time all the time okay and the next thing is movie slash movie obviously or maybe not so obviously this is actually the endpoint itself it's no longer the method that is the actual endpoint but this is not including some sort of domain so we need the full endpoint for that so that might mean that I have to go back to getting started and find what their actual endpoint is right I could do a lot of searching now luckily for me I know that inside of movie DB org slash settings they actually gave me an example API request which is this right here okay so I'm going to copy that example and paste this in here and notice that I have this slash movie and then all of this data right here right so what I actually have now is I see that my API key is in here which we can also just select it if you double click it and scroll up a little bit you'll also see that that actual same string is highlighted up there so that verifies to me that it's the exact same key I mean intuitively I already know that it's the same key because we need to use our API key the next thing is we've got this movie slash and what do you know movie slash right there so what that also means then finally is I can actually start to set some parameters here I can say the API base URL is equal to well it's equal to this this part right here perhaps we leave that trailing slash out and then in this case I've got my endpoint path and I'm gonna go ahead and use the example that they had but that example of course matches the endpoint that they gave us originally which is what was in the documentation so going back into movies alright so this is the original one so let's go ahead and leave that back in there so the same point path I could leave it to being a little bit more dynamic by using movie ID and of course with Python 3 I can do this string substitution so all I really need to do is set a movie ID which again I'll use the example they have is 500 right I'm not positive what movie that is but another thing I noticed is this number 3 here now intuitively speaking I look at the documentation and I see there's a 3 up here and if I see select different version obviously there's a underline under this 3 this probably means that I'm on version 3 if I go to the 4 it shows me four right so again I'm gonna go ahead and change that version this is also a common thing is having an API version so API version is 3 so one more thing that I can string substitute there you go so now let's do the fun part let's actually request this data there is one piece that I missed out which was my API key right I didn't actually add that yet which we'll see why in just a moment but what I'll go ahead and do is say R equals 2 requests that get and I actually want the full path so we'll go ahead and say endpoints equals to another string substitution API base URL and then the in point path alright so we've got the combination of these two things which also combines those variables there we do get and I'll go ahead and print out our that status code okay I'm not sure why this is open I'm gonna go ahead and close that one out and we'll change into day 13 here and then Python 3 I connect PI oops I didn't import request so let's make sure we import that at the top import requests okay let's exit out of here and run that again and I get a 401 ok so for one we can also print out our text see what that is it's Ron again invalid API key you must be granted a valid key right so when I created my account I did get an API key but if we look at my request the actual thing that I'm trying to do didn't include it so what I can try to do is say data equals to API key and API key now I am familiar with this API only a little bit so I'm guessing that that's what the API key is now you can go into a lot more detail the first thing is you see their example API request here it shows you the API key there the next thing is is actually in the documentation if you go into the section called authentication the vast majority of API keys or api's will have an authentication section because the vast majority of them wants you to be logged in to use their API so as we see here I've got my application authentication and it's showing me the parameter a single query parameter or if we're using version 4 we would have to use a bearer token I actually want to show you both versions so we'll see that in just a moment but this gives you an example of what it needs to look like but notice that I only pressed I didn't pass it in into the same point so if I actually print out this endpoint we'll see what happens okay so let's go ahead and exit out again of course make sure everything's saved run it again and in this case I get some data back for all three okay so 403 is not what we wanted we want 200 so let's try to skin with the format that they suggest which is this right here so all that means is in this endpoint I need to add question mark API underscore key equals to that API key okay so this is now what's called a URL parameter passing in as data like that will work sometimes some services do it that way some don't perhaps if I pass it in as JSON data it will actually make a difference so let's go ahead and try that first okay let's try again same same problem same error same 403 okay that's good to see we want to see these errors this is how we get better okay so now all I'm doing is calling this specific endpoints with the key listed there now let's go ahead and try and run that again I'll exit out run it and sure enough there we go there is our actual data and it's a dictionary it's just a big dictionary full of really cool data so there you go you just did your first API request so how do I actually do this with version 4 okay so I'm actually going to copy all of these pieces here and we'll go into API version 4 I probably don't need this API key down here anymore but what I need to get is my token specifically for that version so this is the read access token version 4 off I'm gonna go ahead and say this so API key V 4 equals to that string ok back into the documentation we see that something called a bearer token okay it even gives us an example request curl is another way to actually run a request it's very similar to Python requests except where you call it is in the terminal so if I write out curl here you can actually call those API requests right in there but we're not using curl we're using Python and we're using a requests so let's go ahead and do something just like the which you will see over and over again inside of our project okay so if I take a look at this let's go ahead and scroll down here and I'm going to go ahead and say using the for there's one more parameter that I need to pass in here and that is called headers so headers is just an argument that you pass in to the request that you're doing and those headers are going to be set based off of what they tell you right so here is one header here is another header okay so if I copy this and paste it in here does this look like a valid Python dictionary to you currently of course not so we just add a quote there a quote there and this is saying access token that access token is our API key v4 or at least I think it is I haven't actually even tested that let's try it out I'm gonna go ahead and use this string substitution okay so now I've got my authorization header in there next is the content type itself and yet again I will go ahead and put that in exactly like this okay no change is needed to that or at least I don't think so but now I have the headers actually being passed I'm no longer than call the request that's up here so go ahead and comment to south command slash or control slash will say that we'll exit out of here I'm gonna change the movie Aidid just to make sure that it's different than the one from before I'll just go ahead and do 501 let's hope that's an actual movie okay I try to run it and it gives me a syntax error and of course this is not a valid dictionary I need a comma here okay simple enough exit out run it again and there we go look at that I got some data pretty cool okay so that's version four I like I mean this isn't telling me a whole lot it's definitely not telling me nearly as much as Version three of their documentation or rather their API so yet again I'll just have to go into their API here and take a look at their endpoints so if we go into list this API doesn't actually have a whole lot right now it has list off an account so we definitely don't want to use Version three yet but I would imagine at some point version three will be out and it will show you some of these arguments that you may need to use on a case-by-case basis right so not every API works exactly like this documentation here right in other words the way that the movie DB org set out how they want to tell you about their their API that's what the documentation is for sometimes they'll give really nice descriptions of how things work sometimes they will not that's where you'll just have to try it out on your own using Python requests based off of the things that we just did I just showed you headers query strings the query string is going back up here these are the query arguments so if you needed another argument say like page one right so API key and then you just use an ampersand page or that argument and then what it is not to worry we will do one of those in a little bit that's how you go about doing it all right that's what query strings are okay so let's go back to Python are the version three of this API and I'm gonna go back into my code - I'm gonna comment out version fours stuff and I'm gonna go ahead and copy some of the things related to version three as well so copy just these few requests items here and paste them down here okay so let's go ahead and try a different end point here so now what I want to do is actually search movies of a specific name so how would I go about doing this my intuition would be I want to go into movies and I want to look for something that says search or list okay all I'm seeing here is well and not that I'm seeing a lot of gifts get image get keywords I mean those things are all great but you know going back to this this ID thing I don't actually know where that ID comes from I don't know what that ID is so I need a way to actually find out what those IDs are and in my opinion doing something like search will allow me to find out what any given movie ID is now luckily for us is there's a search argument here and there we go we can actually search things and we can search all different kinds of things so if I look at search movies there we go we now have a new endpoint that we can use they use the get method and this is the endpoint I now want to have right so in the same point path I'm going to change this to being search slash movie notice I don't have any other arguments here I just have the endpoint path and I also don't necessarily need to put page in here yet so I'll save that let's go ahead and see what is required all right so we look at our query string here we've got the API key that is inside of our query or the actual URL itself or the endpoint itself language you could pass different languages it appears that's great query ah this is probably what we'll mean to use to search and sure enough there it is okay so now what I'll do is say and/or am pressed am query equals two and I'm just going to go ahead and say end game okay so we save that and let's go ahead and run it again I'll exit out eight oops let's exit out of the terminal here and run it again and so now I've got printed endpoints Oh obviously I need to make that actual request but here's the endpoint itself I'm actually gonna go ahead and copy this that URL and paste it into my browser I'll hit enter hey what do you know it's actually the same kind of request so it's a little bit easier to see inside of Python in my so we'll go ahead and again say R equals two requests that get and is gonna be the end point and then I'll go ahead and print out r dot JSON exit out of here run that and there's our data so typically with API is they'll send back what's called JSON or JavaScript object notation that JSON data is what this is when it comes back it will actually give you the appropriate data but if I did a URL that didn't actually exist and tried to run this what I'll get is an error code and a message that's related to it right so I could print out the other status code the requested status code not the one that shows up here but it shows me that hey this that this message is I requested a resource that's not found so resource being one of these endpoints right so the search I was able to actually find one so if I exit out and run it again I can see that it's actually yeah it's sending back some data for me now if I want to see this in a little bit better format because this is just like a lot going on here I can use something called pretty print which is import P print and then we come down here and just do P print dot P print and this is definitely for Python three we could go ahead and run this again let's try that well that was the endpoint I need the response let's go all the way down there we go P print dot P print and let's try that again now gives me a better formatted looking content here right it's giving me some indents but it's showing me all of this data that's coming back from that query that I just did which is pretty sweet makes it nice and easy for me now what if I wanted to do something different like a different movie let's go ahead and use a query I'm going to go ahead and say the search query equals to the matrix okay so now I'm going to use that as my search query here again using string substitution but still having that let's go ahead and exit out of here and run it there we go we get the matrix so this is the response data that's coming back now something else that I've kind of noticed is that it's giving me these items here so the response itself is a dictionary but inside of that dictionary it also includes results so another way to think about this is if I said let's see our JSON should still be available and sure enough it is the reason I knew that is because we're using an interactive version of Python so I'll go ahead and say data equals to our JSON and then if I do data back keys it will show me that keys that are associated to the data that came back in this case it's paged total results total pages and results right so if I did data page I'll see page 1 data results this most likely will give me a list and sure enough it does and we can go ahead and say type data results and it will actually tell me what type it is or what class it is so it's an easy way to see whether or not this is a list and I could iterate through these things and find them but this also assumes that our JSON is going to be actual an actual dictionary that comes back for us that's essentially what would end up happening so what do we learn so far what we have is we need some sort of base URL like the base to the API itself that might include a version it might not next we need a resource path in my case I called it the endpoint path which might have a variable argument you might not variable argument being an ID of something and then finally we have arguments that we might pass into the URL but generally we make the endpoint itself and then we call the requests with our authentication and we showed you two ways right here or using a authorization header and a bearer or token this is something you will see a lot with various api's so that's pretty cool and that's pretty much the basics of it but like I said I wanted to actually find out this movie ID thing so I still want to do that let's go ahead and go back into this terminal oops okay so I'm still in here and I have my results here so I'll go ahead and say results equals to the data results now not to worry if you didn't actually do that I'll go ahead and write it in the code itself meaning you didn't set these arguments and all that so data equals to our JSON and we should probably say something like if our status code in range 202 99 basically if it's in the 200 number that means it's probably a good response that we want to see okay so in these results I'm gonna get the first item in here so we'll say results 0 and keys super easy way to know what different options you have inside of any given Dictionary dot keys that's the key value pairs that's an easy way to do it hey what do you know here is an ID so we are in the search for the resource search movie I think it stands to reason that this ID corresponds to this resource okay so what I'll do then is say movie IDs equals to a set if you haven't seen this this is a really cool thing so go ahead and say four results in results then I'm going to go ahead and get the ID so I'll go ahead and say underscore ID the reason I'm using underscore is typically ID is reserved for various other types of things in a program so using underscores just a way to prevent any syntax errors that may occur much like if you want to use the or of or for for some reason as your variables you just put an underscore in there and you're good to go so ID equals two results or rather result and the argument of ID or the key value pair for ID and then I can go ahead and say movies ID that add and this new ID so now what I should have is a list of movie IDs at the very end of that or rather this is a set of movie IDs so I can turn it into a list by just wrapping it around a list now what a set does is it ensures that everything that's in there is unique in other words if I put one two three three three three three four five it's only going to show one three in there right so let's go ahead and see that I'll go ahead and exit out of this let's run it again and now gives me a bunch of movie IDs that match the query of the matrix okay so I can get titles too so lets if I scroll back up let's see let's actually get rid of I'm gonna get rid of the print statement from the end point and also the response so I'm gonna go ahead and bring this back and print out results zero that keys and of course this would also assume that there are results so if results is greater than the length of results is greater than zero then we'll go ahead and run the rest of it okay so again I want to get those keys so it's going to run that again and here's all the keys that I have I've got ID and I also have titled cool so let's go ahead and just print out the result title and saw print result and title and then that ID that I'm grabbing say that it's good to get rid of these other print statements because we don't need them currently and I said out of here run that again and there go so we've got all of these different titles that are related to the matrix no surprise there they had sequels they had documentaries about it they have all sorts of random stuff about this so that's pretty cool I mean the matrix the first one is probably the only good movie hopefully majors four is good we'll see but anyways we now have a way to search for a specific movie based off of its title right that's this right here I didn't do much more than that right I could probably add other parameters in there but as of now I can search off of its title and then I can grab its results or its actual ID so those movie IDs now what I can do then is say for movie ID in movie IDs I could probably quite literally copy and paste what I had up here and what we did at the very first part let's go ahead and do that there we go and our equals two requests get end points and I don't actually need a page in here I just need to make sure that I've got the movie ID and then the API in API keys all good save that and now I'm going to go ahead and print our JSON and yet again I'm gonna go ahead and make sure that all the other print statements are gone maybe even the title and ID of the results this time it's actually give me in-depth detail of each one of these or at least that's what it should do so it's go ahead and exit out of here I run that again and sure enough the does all rights giving me a lot of this detail pretty cool so here you go right so we're close we're close to having something useful here not quite there one more thing I want to do is turn all of this into a CSV file so I'm gonna go ahead and import pandas as PD and what pandas is gonna allow me to do is take a list of objects or something like this and actually store it to a CSV file okay so I'm gonna just go ahead and say output equals two movies dot CSV no big deal there and then I'll go ahead and say movie data equals to an empty list and that data I'm gonna actually set to what this resulting JSON data is so again I'll say if our status code in range 200 299 then I'll go ahead and say data equals to our JSON and now movies data dot append this time a pin because the list will move on that data okay cool so now that we have that data I'm going to make a panda's data frame will say DF equals to PD as in pandas dot data frame the data frame class will pass in that movie data keep in mind that it's a key value pair so they're all dictionaries in there with their own values we can print out what's in there with DF head and then we can also do DF dot too CSV and then I said the output file is movies CSV and I'm going to go ahead and just say index equals the false that's just something that makes it easier on us in general of pandas I realized we're not going to do too much with pandas here but hopefully all of this makes sense at this point okay so I'm running that it's going to take a little time and it prints out the head of this object here this isn't telling me a whole lot so let's go back into the Explorer and look into these movies here and I want to reveal this in finder and then I'll go ahead and open this up this naturally for me is going to take me into numbers you know or Excel depend on what you're what you're actually working on okay so cool things here it's telling me whether or not it's an adult movie which is nice to know all of these are false it's giving me budgets or what it thinks as a budget original title original language of you poster path I'm not positive how this poster path works but I'm gonna go ahead and try some intuition here and say it's movie DB org so let's go and say the movie DB org slash that item there nope it's not that let's try API dot also not that okay so I don't know what that URL is probably I mean it's certainly somewhere somewhere in the documentation I'm not going to get into that right now it has it stores lists of production companies it stores revenue or seeming revenue perhaps this is right number perhaps it's not it's hard to say status like when it if it's released was it released it also who shows you know what the vote is I think this is related to the votes that are done on the movie database but now we have data about this particular movie so the big question here is we have a way to grab all of this data or a good amount of data are related to a movie from the movie database but it's not giving us all of the same data that comes from Box Office Mojo and this is where you can take your web scraping skills and now your REST API skills combine them and really enrich this data to have all of the data that you're looking for in relation to any given movie and of course that's a challenge i'ma leave to you I think there's enough here that you probably make that happen maybe not maybe it's still too challenging but I think it's a really good challenge as a beginner to try and actually make sense of doing all that and you can save it in to CSV files much like what I've already done so the other thing that we might want to do is try and figure out not just the title of movies of course but rather the actors in that those movies like what are their names and can we add them there may be other staff members as well the movie database itself may or may not have that information I mean it does have people in here so maybe there's a way to find these people Oh get movie credits well that's true of a person ID that's not what I'm looking for let's go back into movies maybe there's something in here related to the people in it all right so we've got details that's what we've been doing here's one called get credits movie credits get the cast and crew for a movie hey what do you know exactly what I was mentioning so the movie database API is really really nice for all of these things I just don't think that they have the sales data that Box Office Mojo has so again combining the two might be something really interesting and of course it isn't straightforward because if I click on here this ID may or may not match the ID that's in the movie API database and really if we're doing it manually it's not going to be that fun anyway but that is one of those challenges of doing this kind of work is combining this data from different sources is a very interesting challenge and yet can be very profitable as well of making a cool new service that's never been done before and that's really the point here is like to figure out how to actually use diff rest api's in conjunction with web scraping to actually build a service that other people can use and hopefully love so thanks so much for watching day 13 hopefully you got a lot out of this I will mention that there is a lot more that can be covered in rest api's this is meant to scrape the surface but also to get you something practical pretty quickly so you can actually make something happen as soon as possible and of course if you want to go a little bit further something like stripes API is amazing but it adds a whole new element of complexity and that is how to actually collect the credit card information in a web application that Emma just makes a lot harder so that is a cool API it's a very good API a lot of the concepts are the same as what we did except for that credit card grabbing process didn't doing that in a secure manner that takes a lot more so now the idea is like leaving you with combining these two things let's see if you can actually do that if you can let me know I'd love to take a look semia CSV file if you like or a link to one on github either way the other thing is how do we actually turn this into a web page how do I actually do everything that I just did with scraping as well as using this API and using it as an actual web page that anyone can go to and use that is an interesting challenge that we'll certainly have to cover thanks so much for watching we'll see you next time

Channel: CodingEntrepreneurs

Views: 48,077

Rating: 4.9151516 out of 5

Keywords: djangourlshortcfe2018, install django with pip, Django Web Framework (Software), Mac OS (Operating System), Python (Software), web application development, installing django on mac, pip, django, beginners tutorial, trydjango2017, install python, python3.8, django3.0, python django, web frameworks, install python windows, windows python, mac python, install python mac, install python linux, pipenv, virtual environments, 30daysofpython, beginner python, python tutorial, rest-api

Id: Sg5VTTBIhqo

Length: 42min 48sec (2568 seconds)

Published: Wed Apr 01 2020