Requests Library in Python - Beginner Crash Course

what is going on guys welcome back in today's video we're going to do a crash course on one of the most downloaded python libraries out there called requests we're going to learn about basic HTTP requests the different types there are what they do and we're going to learn about some more intermediate or Advanced features of the requests module we're going to learn how to work with proxies we're going to learn how to change the user agent we're going to learn how to download content bite content like images for example we're going to also learn how to work with different status codes and with timeouts so we're going to cover some of the more intermediate or Advanced features however we're not going to cover all of the features because this library is quite comprehensive but I'm going to try to simplify and explain the concept that can be interesting to a beginner or intermediate python programmer so let us get right into it [Music] foreign so the request package is an external python package it's not part of the core python stack which means that we need to install it before we can use it and for that we open up a command line and we type pip install requests now change this how you will already have it on your system because it's one of the most downloaded and one of the most used packages out there but if you don't have it you can just install it using pip and then we can import it by saying import requests and we can do some basic stuff like saying request dot get then a URL and this sends a get request or we can do a request.post and this sends a post request to the URL with some data that we can provide as well that's the basic idea and we're going to use a specific web service for our request today we're going to use httpbin.org this is a simple HTTP to request and response service it offers multiple different endpoints and those endpoints um return different responses so for example we can send a simple get request to the get endpoint and we're going to get a simple response with some data we can also send a get request to status slash 404 to get a not found status code so we can force the status codes or 500 to get an internal server error which is not so simple if you're working with an actual website because then you have to figure out how do I cause a 500 status code here you can just say I want to have a 500 status code and then you can handle this in the python code for example so this is a very nice comprehensive service to learn about the request module because we can decide what the content of the response will be we also have stuff like delaying the response so we can work with timeouts we also have cookies and we have images and stuff like that so this is this makes it very easy for us to work with requests now before we get into the code let's cover these five basic HTTP methods we're not going to work with all of them in today's video we're only going to look at get and post but the basic idea is that you want to use a get request whenever you want to get information about a resource you want to use a post request whenever you create or upload a new resource you want to use a put request when you want to completely change an existing resource so you want to update it completely you want to use patch when you only change certain aspect specific attributes for example of a resource and you want to use delete in when you want to delete a resource those are these five HTTP methods here and now let's just get started with a very basic get request we're going to just say response equals requests dot get and we're going to send the request to https colon slash slash HTTP bin dot org slash get and um this will then give us the response and with the response first of all we can just print the status code so response.stat status code to see if we get something valid 200 if I enter something that doesn't exist probably we're going to get four or four there you go not found so this gives us the status code but we can also print the content now you can do that while watching this video I recommend you code along you can do that now you're going to see that your IP is being displayed of course due to privacy reasons I don't want my IP to be um displayed here so you don't have to do what I'm going to do now I'm just going to turn this into a Json object and remove the IP field so the origin of the request is dip I'm going to remove that field just so that I can show you the output without the IP but you don't have to do that I'm just doing this for privacy reasons so if you want to print the full response you just type response.txt this is going to give us this is going to give you the full content of the response you can also say respond Json this is going to give you a dictionary because the response of this API is a Json object which is equivalent to a python dictionary so you can just turn this into a dictionary I'm going to now call this rest Json and I'm going to delete the rest Json origin field so this is the IP address and then I'm going to print rest Json and again I just do this so that you don't see my IP address displayed because by default you also have here an additional field origin and then the IP address that this request was sent from I'm going to display it when we work with proxies because then we're going to only see the proxy IP address but yeah you can see we have some responses here nothing too fancy you can see the user agent you can see the host that we're sending the request to this is the header basically and you know we can now add some more stuff using the request module so oftentimes you will have the option to have query parameters in the URL so you could say something like get question mark and then name equals mic and H equals 25 for example if you're querying something now we can do that without doing it manually with a question mark in the equal and the end operator we can just say that we want to pass parameters so we can say params equal and then we can provide a dictionary and we can say that the name is Mike and the H is 25 and then we can just add this here as param so params equals params and the first thing that I want to display here is I want to display the URL so we can see when I say response.url we can see the resulting URL and you can see it's now HTTP bin.org get question mark name equals mic and H equals 25 and you can also see that we now have in the arguments key value pair we also have now a dictionary with H25 and name mic this is what um the response is so this is what the API receives when we send this here so that's quite simple now let's do a simple post request with a post request it works differently we don't send it using parameters if we do a post request what we do is we say data equals and then we usually call this call this payload this is the best practice way so we say payload equals and then name mic H25 and that's basically what we do we call a post request also we don't do it on get we do it on slash post and here you can see now that we have no arguments but we have um there you go we have the form data the form H25 name mic so this is how you send a post request you can do the same thing for put and for patch and stuff like that um those are the very Basics now let us uh see how we can handle different status codes because oftentimes when you do something like web scraping or you want to explore some API you might want to um have a loop that does everything in an automated way but sometimes you might do something that causes an error sometimes you might get specific status codes that will produce problems for you so what you can do in this case is you can just send a response so in this case we're going to send a response to the status endpoint so we're going to say slash status slash um 200 which is Success this is the default we can send a get request to this this will not be a problem we can just print response dot status code you will see okay what's the problem here oh I think I cannot in this case we don't need to we don't get the IP address so we cannot remove it but this is the status code that we get here now if I change this to something else like 404 this is going to produce 404 so you can see the status code of the response is influenced by what we enter into the URL and what we would usually do is we would not cause a 404 a 404 would happen so we don't find something and it would just return this error code or status code so what we want to do is we want to say okay if the response status code is equal to and you don't even have to memorize the number you can just use response dot code uh or actually what was it response dot codes plural and then you can just specify not underscore found and you can print not found for example and else you can just print response dot status code and what's the problem here oh not response sorry requests requests.codes so this is from the requests module we have a codes module or for the request package we have a codes module and here we have these constants not found which basically is 404 but you don't have to memorize the number you can just check for it if I change this now to something else like 500 internal server error it will just display the code because it's not this one so this is how we can do that we can just do if the status code is not found do something if we have an internal server error wait try again if we have um 200 just continue you can handle the different status codes using um using the status code attribute of the response and using the codes provided by the request module but of course you can also specify the number directly that's not a problem all right so another thing that you might want to do is sometimes you want to change the user agent so the user agent is essentially an identification that tells the web service what software you're using so by default I think we can see that actually if we uh what was the end point for this we have a special endpoint user agent but we can also just you know see it in the header but if I say user agent here is the endpoint and I print the response dot text we get the user agent from the header so you can see the user agent right now is python requests and then uh some version number but oftentimes websites will block you because they see okay you're using python requests you're not an actual web browser you're not an iPhone with a Safari browser you're not a MacBook with a Google Chrome browser you are python requests I don't want to serve you or I will serve you differently so oftentimes the website the web service will respond differently because of the way this site is structured based on your user agent so what you can do is you can just specify a different user agent manually while sending the request so how you would do that is you would say headers is a dictionary and there is this field user Dash agent like this and here you can just specify the user agent so I can do now hello world 1.1 or something like that um and in this case it didn't work because we didn't pass the headers so we need to also when we do request.get say headers equals headers um and then you can see the user agent is now hello world in this case doesn't make a lot of sense but what we can also do here and I have to copy this from my prepared code because I don't want to type this out you can just Google it we can also set this to a customized user agent that is for example in this case iPhone um apple webkit and Safari so you can see this is now the user agent for an iPhone I can just send this and then the web service sees that I'm browsing from this user agent and it will serve me like an iPhone that's using Safari that's the basic idea um we can also provide different stuff in the header um so for example we can also specify what kind of content we accept and this can be quite useful when we download images for example so if I go down here we have an endpoint image which returns a simple image and the type of the image unless we go for a specific endpoint the type of this image here is going to be suggested by the accept header so this is something that we can also specify in the header we just have to say accept and then I can specify the file type of the image that I want to download so for example image slash PNG or image slash jpeg and then I would just send a request to image and then headers equals headers and in this case I would get a response but the response would be bytes so we don't want to work with text we we want to actually save this into a binary file we want to save this into an image and this can be it can be done quite easily by just saying with open myimage.png in writing bytes mode SF and then just F right response dot content so not text but content because that leaves it in the byte format and then I can just open this you can see it's a pick I downloaded the image with the request module from the API um if I now go ahead and change this to jpeg first of all it will give me a different image because the API is programmed in that way but also I would have to save this as a JPEG file so that I actually see the image but then you can see we have what is this a wolf deer no I don't know um but yeah this is how you do that you sent the request you specify the file type that you want to accept I think it should also support web P I'm not sure about that though webp let's see if that works but I think we do have to then open this can I open this in the browser it doesn't matter uh you can check this on your own if you want to but you can specify just what font type you want to accept those are the supported ones jpeg PNG SVG webp so this should actually work um and then you get the image and you can do that by specifying this in the header now another thing I want to show you here that's quite useful is oftentimes when you want to for example you have a list of proxies this would be a use case you have a list of proxies of free public proxies you want to see uh which of those proxies work and you want to just do it by using the proxy sending a request to a simple website and seeing if that works sometimes you will not get a exception an error immediately you will have to wait for I don't know two minutes until you get the response which basically means it doesn't work or it's very very slow but it's technically speaking not giving you an error right away so you have to wait what you can do in this case is you can cost uh or not you can cost you can um specify a certain timeout so if the script if the response doesn't come in that timeout or in that defined um tolerance time you're going to just uh quit you're going to raise an exception and you're going to continue for example so what we could do here is we could in this case we want to Target the uh what was it the delay endpoint we just do slash delay and then seconds so slash delay and we can delay here for three seconds for example um and let's do it without the timeout first let's just do it like this I'm not sure if we're going to get the same response there you go so we get the response like from the basic get but we delayed for three seconds so I can also delay it for five seconds it will take longer um so you can see nothing happens nothing happens after five seconds it's going to respond now let's say we don't want to tolerate this let's say we want to have a certain timeout what we can do is we can say timeout equals and then we can say um I'm allowing for three seconds but if it's longer than three seconds just raise an exception and break so you can see now we had a read timeout we have this exception and what we could do here is we could just say four uh whatever in some collection one two three I don't know so you have basically URLs that you're trying or proxy service that you're trying you're doing something and you would do something like try oh it's lagging now uh you would do something like try to send a request but if there is a timeout so if it doesn't work just continue right so you would do it like this um this would not break the script then but it would basically tell you um it would basically just skip that particular instance and of course three is maybe a little bit uh too small but you can specify a timeout of maybe 30 seconds if you don't get a response for 30 seconds you just skip um and you can see here if I delay for only two seconds this will actually work but I would have to let me just delete the structure again here if I delay for only two seconds with a timeout for three seconds this would actually work as you can see it's in the tolerance all right so um then last but not least what I want to show you is how to use proxy servers I have done it in um some videos already I even have a video on rotating proxies uh the basic idea is you want to specify a proxy server you can either have a paid one or you can find one for free online that works and you want to send a request through that proxy service you don't send a request to http bin you send a request to the proxy server and the proxy server sends the request to hdp bin how would we do that we would just say proxies equal and then HTTP you would specify the IP address of the proxy I have found one here one three nine nine nine two three seven uh six two Port 80. this is just a very uh this is just a public free server that seems to work at the moment and then you would also specify one for https now this one doesn't work for https um I'm just entering it here so that we have something uh in order for this to work in this particular case because this is just an HTTP proxy we would have to Target the HTTP API this is not a problem in this case but if you want to do it professionally of course you want to use an actual um you want to use an actual professional proxy that also allows for https now we're going to do this on the get endpoint we're not going to have a timeout and we're going to also not delete the origin because this time we want to see the IP address because the IP address will not be my IP address if I provide proxies equal to proxies this is what we need to do we need to say proxies equals proxies then it's going to use this and it's going to give me the response so print response dot text we can run this and you're going to see now the origin is this IP address this is not my IP address this is the IP address of the proxy server so this is how you can do that so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you next video and bye foreign [Music]

Channel: NeuralNine

Views: 45,200

Keywords: python requests, requests, library, python http requests, python http, python requests crash course, crash course, networking

Id: Xi1F2ZMAZ7Q

Length: 20min 32sec (1232 seconds)

Published: Sun Dec 11 2022