Automate Google Search API in Python

what is going on guys welcome back in this video today we're going to learn how to automate Google search and python using Google's custom search API so let us get right into it [Music] alright so in order to automate Google search and python we're going to start by creating a new project in the Google developer console you will find a link to it in the description down below and what we want to do here is we want to click on this drop down menu and we want to click on new project so we click here and then we're going to provide a project name for example YouTube search Project we're not going to specify a location we're just going to click on Create and then once the project is created we're going to select the project so that we can adjust some settings so that we can enable the custom search API so you can either click now on select project here or you can use the drop down menu to select a YouTube search project that we just created and now what we want to do is want to go to apis and services so either here from the quick access area or we can also use the navigation menu here we have apis and services and we want to go to enabled apis and services and once we are here we want to look for we want to search for the custom search API so just custom search and then enter and then we should find the custom search API we click on it and then we have to enable it for this project so with Google Cloud projects we always have to specify uh which functionalities you want to enable for the project so we click here on enable this enables the custom search API for this particular project but to use it of course we also need to have an authentication mechanism so we need to generate an API key so that we can authenticate ourselves from python with this project so that we can actually use the API so what we want to do is want to go to credentials here in the apis and services section and we want to create new credentials so we click on create credentials and then we want to create an API key so we click on API key this is going to generate an API key and this is something that you want to keep secret you don't want to show it to anyone you don't want to share it with anyone because everyone that has this API key can just use your API so basically you're giving up control over your account if you're sharing that key so copy it save it somewhere preferably if you're doing this in production in an actual project you want to save it as an environment variable in a DOT end file for example I have two videos on this channel explaining how to do that um and then you want to load it into python for the sake of this video today I'm just going to load it from a text file so I would want to do here is we want to go into our working directory I'm going to create a new file API underscore key I'm going to paste the key here I'm going to save it I'm going to close the file and then I'm going to load it from there now the second thing that we need to do is we need to create an actual search engine so what we're going to do is we're going to go uh to Google and we're going to type programmable search engine and we're going to click on the first link here so programmable search engine google.com uh we're going to click on get started and in my case I already have two search engines here that I played around with I'm going to add a new one by clicking on add up here and we're going to call this YouTube tutorial engine or something like that we're going to search the entire web and we're going to also allow for image search I'm going to confirm that I'm a human I'm going to create and the important thing now is that we want to go back to all engines want to click on our engine here and we now need to copy the search engine ID so we want to copy this search engine ID here uh and I'm gonna store this one in the python file directly or actually I'm going to load it from a file as well just why not search engine ID paste save close and then we're going to open up a new python file main.py now all we're going to do here or we're going to only use um the requests python package which is an external python package we're not going to use any Google specific python package so we're going to open up the command line pip to reinstall requests if you don't don't have it yet and um then once you have that we're going to say import requests I'm going to zoom in a little bit here and we're going to say API underscore key equals and then open API key read and the same thing goes for the search engine ID search engine ID read so we have these two values now in our code and what we want to do now is we want to specify a search query so we're going to say search underscore query is going to be equal to something you want to look for for example neural 9 books and one simple use case of this automation is to just get the first link that we find and this can be useful for example if you have a bunch of company names and you want to find their website or something like that or anything similar to that so you just want to find the website the most uh the most matching website to a given search term so what we can do here is we can say URL equals and we want to say https colon slash slash Google apis.com slash custom search V1 that is the URL that we're going to send the request to and the request itself will have some parameters so we're going to say here params equals then curly brackets and we're going to define a dictionary of parameters of URL parameters I'm going to say the query so Q is going to be the search query we're going to say the key is going to be the API key and we're going to say CX is going to be the search engine ID and then we can send a request we can say response equals and then requests dot get so we're sending a get request to the URL with the parameters that we defined so params equals params uh and then we're going to say print results so results like this um and what we can also do or actually sorry what did I do results I need to do one more step in between results is equal to just the Json version of the response so if we have a dictionary um and we're going to what's the problem here Json decode error let me just see what the response says 404 did I mistype something Google apis.com slash custom search V1 that should not be the issue what's the problem here custom search was not found on the server why is that maybe I need to add www in front now it works okay so we're going to say results equals response dot Json and then print the results and then we're going to look specifically for the first URL so you can see we get some um some some data here as a response and what we want to do now is we want to say uh if we have items so if items is part of the response so of the results we're not going to print them here anymore what we're going to get is we want to get the first result so the first item here so results and then items and then the first one and specifically want to get the link of the first one so in this case this was this would lead us to the neural 9 book page here um all right so this is one basic use case another thing that we can do is we can also search for images so we can say Okay given a certain search query provide me with some images so for example I could say something like top 10 Linux distros or something like that or maybe let's go with something else let's go with cats because then we'll find a couple of images here so I leave everything basically in the same way so I basically just have to change one parameter or add one parameter here which is the search type search type is going to be equal to image and this will give us images now to the given search query or for the given search query cats by the way one thing that I forgot to mention is that per day you have a hundred free requests so you can use this API for free if you limit yourself to 100 searches per day I think you can also expand this if you add a credit card you can do up to what was it a thousand or ten thousand or maybe one hundred thousand requests but you have to pay for it then but 100 per day are free so what we're going to do now is we're not going to get the links uh or actually we're going to get the links but we're going to get different links so um or actually we are going to get the links but we're going to do it in a different way because we don't want to get just the first one we're going to say results is equal to the Json object but we're going to get the items right away and then we're going to say four item in results we're going to get from the item the link actually not from results from the item and now we got a bunch of images here you can see we get a cat here we got a cat here and we get a cat here which is actually the same one kind of okay so this works all right so this is for example image search you can also specify different parameters like for example um we could say something maybe like news and I could search for images of news and I would get very current events uh okay no this is maybe I was wrong about this so okay I get a lot of logos so maybe we can look for current events or something like that or maybe I have an idea if this doesn't work I have an idea maybe we can look for U.S elections or not elections election Maybe uh and we should get maybe something that's up to date I'm not sure no this is just explaining okay this is just very generic again um I mean yeah okay this is some data that's probably somewhat up to date I guess I'm not American so I don't know but the idea is that we can now go ahead and limit the search to a certain um time frame so I can say date restrict which is another parameter that I can pass here and I can say that I want to restrict this to 2021 uh first of January and then I can say colon 2021 uh 12 31. or actually let's go ahead and do this 2016 for example and let's run this and then I will get older stuff so at least we get different stuff I don't know uh yeah it seems like we have not sure is this current this is not current again I'm not American so I don't really know uh but you can this is a thing you can definitely restrict the date um I don't know if this is a good example now but you can specify that you're only interested in results that lie in this time frame here so that's a very useful feature if you're looking for some old stuff or only for current events only for current uh results for the latest results so to say um what else can we do we can also specify a file type so I can say for example Python tutorial and I don't want to get images I don't want to get um just links I want to get a specific file type so I can say here file type and I can say that I want to have PDF files so uh what I would do in this case here again is I would just run this code and I would get a bunch of PDF files so for example here whatever that is I'm going to save it we have some Python tutorial here as a PDF file and you can just go through them you can combine them with other restrictions maybe you want to have certain a certain time frame maybe you're looking for a specific search query here a different one so not just Python tutorial but Python tutorial python 3.10 or something like that that could be an IDM and another thing that we can do here the final thing that I want to show you and you can get creative with this of course this is just those are just some generic examples now you can build this into a project you can use this to autofill data you can use this to build data sets by looking for images for example of course take care of the legal aspects of this but you can use this in very creative ways but one thing that I want to show you here now is that if I look for the term football for example in Europe football means soccer so kicking with your lacks basically and in America football means American football so NFL and what you will see here is that if I say the search type or actually let's let's not look for images uh what I can do here is I can specify the location so I can say LR and I can say that I'm looking for language underscore English so e n and then I can say GL and I can specify that I'm looking for us so United States America and then you will see something interesting if I do that you can see NFL is the first result here and I'm sure we're also going to find some some uh some soccer probably but you can see that the first result was NFL if I change this now to UK instead of us you will see that the first thing I get is football so soccer and football again and also uh football manager which is soccer I guess and then we have um Liverpool which is also a football team a UK football team uh and probably if I change this to I don't know what the correct uh symbol is probably d e for Germany I think so yeah we get first of all the German Wikipedia article for football uh we get German website so you get the idea you can and I think here the interesting thing is if I say football in German because in German Football is not called football but there's a German word for it and if you specifically look for football you're most likely going to get American football because that's what we associate um in the German space with football so you can specify here the language and the region for the search and you will get different results based on that so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you next video and bye

Channel: NeuralNine

Views: 31,800

Keywords: google search api, custom search api, google cloud, google developer console, python automate google search, python google search api, python custom search api

Id: TddYMNVV14g

Length: 16min 34sec (994 seconds)

Published: Fri Aug 18 2023