How to use the Twitter API v2 in Python using Tweepy

hi everyone my name is sahim parak and in this video i'll teach you how to work with the twitter api v2 in python using the tweepy package so tweep is actually a popular package in python that's used by students researchers and developers for interacting with the twitter api and uh with the twitter api v2 we actually support new exciting features for uh students developers and academic researchers and tweepy actually updated their library to actually support this new twitter api v2 and so i wanted to do a quick video tutorial that walks you through the different functionality that's available in the twitter api v2 and how you can leverage those using tweepy so before we get started i want to quickly point you to the uh the the documentation page for tweet so if you want to learn more about this uh the tp package you can go to github.com tweepy and you'll find uh the source code there and then if you wanna look at the documentation um you can click on this link here that shows you how to use the um the 2b package in python or the different authentication methods and so on and um yeah so that's the documentation and there's this section here on github on the on releases so you'll see here any new updates that they make to the library uh to support the twitter api v2 or any new um new endpoints that we launched you'll be able to see those here so you can see here um since 4.0 is when they start supporting the twitter api v2 so in order to first get started with the twitter api obviously you need to have a developer account right and based on your access levels you can use the different functionality that's available let's quickly take a look at the access levels that are available today in the twitter api v2 so for that we can go to developer.twitter.com and under docs if i go to documentation in the twitter api section there is this um this section on api access levels and versions if i go here so you'll see a few different access levels available the first one is this essential access right so that's available for like anybody who signs up um for it and you don't have to wait for a um use case review what i mean by that is you sign up and then you get instant access to certain functionality on the twitter api v2 right so obviously you were able to get up to 500 000 tweets per month so 500 tweets per month that basically means that anytime you make a request to the twitter api and that retrieves back uh tweets for you um you're able to get up to 500 000 tweets per month as part of this essential access right so this is actually a good access level for people that want to explore the twitter api or want to use it for like a small class project or some something basic like that once you've used the twitter api the essential access and you're uh interested in getting more twitter data then obviously you can apply for a use case review and depending on your use case you're either able to get access to the elevated uh package so in elevated you get up to two million tweets per month right which is more than the 500k that's available for essential access and if you're an academic researcher meaning you're a master's student phd student post doc or associated with a university you can actually apply for a use case review for academic research and once you're approved you get up to 10 million tweets per month which is a um which is obviously a lot of data compared to the other levels of access and also you get access to the full archive search right so if you have access to essential uh essential and elevated you're able to get tweets from the last seven days right if you wanted to search to a twit twitter data you're able to go last seven days um only but with academic research access you're able to get tweets from the full archive meaning the very first tweet from 2006 all the way up until now you're able to go back and get data from that time frame so definitely check out our access levels and apply accordingly so once you've applied for access then obviously you can go to your developer portal once you're obviously once your application is approved so in my developer portal here i have let's zoom in a little bit i have two projects available one's the academic research one and this is the elevated one which gives me up to 200 or 2 million tweets right so in order to start using 2p and to get data in python using um the twitter api v2 obviously using the tweetpiece package i need to first create an app because the app acts as a placeholder for my keys and tokens right so if i want to work with the twitter api i need to authenticate my request with different either like better token or api keys and so on right and um the way that you do that is you go in here and you click um add an app right and you select which environment you want to deploy it in so i'm going to select the development one and then let's give this a name so let's call it pp demo oops twitter api v2 okay that's the app name and now if you see here it gives me my api key api secret and better token and these are um the the keys that i'll use to connect to the twitter api to get twitter data back right um but this is useful to use functionality like searching for tweets getting a user's timeline getting tweet by tweet id and all those functionality if you want to further um create tweets or let's say you want to retweet a tweet or like a tweet programmatically then for that you obviously have to um you have to give special permissions to your app and also get your authentication tokens or access tokens um so what we'll do right now is i have this ide pycharm right and in pycharm i'll go ahead and create a config file um so pycharm by the way is a id that you can use to build projects in in python so here i'll create a file called config.pi and i'll start adding all of my tokens here so that i don't have to um repeat um like constantly copy paste those right so i'll create api key and i'll copy these here then api secret obviously you'll have to use your own ones because i'm going to delete these after this tutorial um so better token right and so these are part of my elevated project which means that if i want to use full archive search these these credentials won't work cause for larka search is only available for academic researchers and we'll um see how academic researchers can use um the full archive search in later on in this video but for now i've copied the the tokens next i'll go ahead and go into app settings so in here i'll go ahead and click on setup and here i'll enable oauth 1.0 and the permissions that i'll set to is read and write and then in the callback i'll put just for example um.org and i'll go ahead and save this next i'll go ahead and create the access tokens for this if i uh go here and create access token i'll copy these as well because i'm i'll be needing these when i create uh tweets and retweet um the tweet so i'll call it um access token and then i'll call it access spoken secret okay so now i have um i'll save these so now i have my uh code set up uh for uh starting to use tweepy right now before you use sweepy you need to make sure that it's installed for you right so the way that you can install it is a couple of ways if you've never installed tweepy before you can just do pip install 3p right this will install for you but if you already had let's say an older version and you want to get the latest updates so we saw that the 4.0 onwards is when it supports the twitter api v2 you will just do upgrade next to that um and okay so now you'll have tweetp ready on your machine right so i'll go ahead and start showing you some use cases so let's start with a basic use case around search right so let's say you want to get tweets from the last seven days so we'll create a um a file called search.pi i'll import this preview package because i need to i need to use that in order to get the tweets and then i also import this config file and this config file has my tokens right so um what i first need to do is initialize a client right so that client is the the thing that has the different methods um that you can use to connect to this twitter api so i'll do uh 3p dot clan and i'll pass it my better token so better token is config dot better token so now um i have my client set up now let's quickly take a look at the client and see what different things are available so you can get like tweets that's a function another function is get users get user followers and so on right so let's see if we have anything for search okay so we have uh search recent tweets so we'll call that right and then this method requires um for example a query right so you'll pass it a query i'll actually create a different variable here for a query um let's call it a query yeah and then um query is basically uh where you specify what you're looking for so if let's say you're looking for tweets about a certain keyword let's say you want tweets about code uh you can type that and then you can have or condition if you want r if you want an and meaning you want both the terms then you can do code with uh space code 19. this will give you tweets that contain both of these obviously if you want um an or you add those and then there are other operators that you can use as well so if you want to get all tweets that are um retweets you'll do is retweet if you don't want all tweets that are that have retweet you simply add a negation to it right similarly if you want tweets that have a media so then you do has media if you don't want that condition then you do negation before that so this will give you all tweets that don't have media right so for now let's just get all tweets that contain the word code and that are not retweets meaning their original tweets so i'll do not is retweet by the way if you want to learn more about how to write search queries right you can actually go to um developer.twitter.com again i'll show it from here oops yeah developer.twitter.com and in this there is actually if you go to the documentation section this actually has um explanation of how to build search query likes if i go under tweets and search tweets you'll see there's an integrate section that has everything about building queries like what are the different operators that are available examples of writing search queries right like here uh give me tweets that contain the word apple or iphone and um ipad and things like that so definitely check this as one resource and the other one is if you go to uh github.com twitter dev you'll see that we actually have this course on getting started with twitter api for academic research and in this there's actually a entire module that teaches you how to write search queries if i go into this one five it gives you different use cases that if you want tweets from a certain account if you want um and r conditions how do you negate something so like things that are not retweet you add a negation so check out this resource obviously it has a cheat sheet as well that shows you how um how you can use um the different operators available to write search queries so let's go back to our code now so we have our query ready uh we want all tweets that contain the word code and that are not retweets so we'll pass this and then by default per request you can get a hundred tweets back right um so if you oh sorry by default you get 10 tweets back if you want more than 10 back then you need to use this parameter called max results right and you'll say that max results is 100. so now instead of getting you 10 at a time will get you 100 tweets right so let's quickly just go over this um and see what response we get back so i'll do response is this and then uh print response i won't actually print the response i'll add a debugger just to show you what's um what kind of response we get back so if i go ahead and debug this you see that the response has a few different things it has a data object it has includes it has errors and meta meta actually has things like the result convenient there are 100 tweets that we got back and then um data if you if i go into this you see how they're like different tweet objects the response that i got back and by default like if i just run it as is you see it's giving me only the tweet id and um tweet text right um now if i want to go ahead and get additional fields i'll have to do something um extra but before we get into that let's go ahead and like just print this for now right so if i um if i do for tweet and oops uh resp response dot data because data is where the tweet is right so then i can do uh print tweet dot id so let's just quickly run this and this should yeah this prints all the tweet ids for me right now um so we saw that by uh from this we only get the tweet id and tweet text back now let's say you want additional fields back right so let's say you also want what languages this tweet uh uh these tweets were in so you can actually use the tweet fields uh parameter so you can do tweet fields right and then we can get let's say we want what time um the tweets were created so we'll use created ad and we'll also do um let's do length which gives us the language right now if i go ahead and here i'll just do oops let's do tweet dot id and along with this let's also print now the tweet dot link so which language the tweets are in if i run this this should um give us the tweet id and the language as well yeah so it's giving you the language course like what language the tweets were in so so far so good we're able to get additional feeds that fields that are related to tweets but now what if you want um let's say user information right because you want to also know let's say the profile image url for each of these users so the twitter api v2 actually supports a different way of getting this information and the way that you um get this data is using what we call expansions and uh fields right so if i go here go back to this uh the docs um if i go to fundamentals it actually has this section on expansions right so here it's telling me which different expansions to use for different things so let's say if i want a user i'd have to use this expansion author id if i want to use uh if i want to get places right or geo-tagged places then i'll have to use this expansion so let's do those two examples at a time so i'll copy this one because i want user information um and um i also want additional fields available for that user right so but before that let's go ahead and use that so what i'll do is here um i'll add this expansion spare meter which expansions and i'll pass it um authority um yeah actually let's debug this real quick um to make sure that it's uh includes yeah so perfect so if you see here in the includes now it has user information associated with it and by default you can see that there is the user id username and uh the actual name so let's print the username for each of these right so the way that we can do this is let's create a dictionary with this user's information right so we can do users and then for you in this is in response.includes right what we can do is we can create a id and the user object pair so we'll do u um id oops uh u for okay so now i'll have a users all the user information in this available so let's debug and make sure that we have that um oops oh that's because i didn't include the the users object itself so let's do users and debug this okay perfect so now you see users has the user id followed by the the user object grid now we want to uh make sure that for each of these uh tweets we're printing the tweet id and also the user let's do user um let's do just the name or username let's do username so uh the way that you can do this is that you'll check if users has this obviously this user id so the way that you do that is tweet dot author id all right so meaning that if obviously this author id is present in users so we'll print the tweet id let's remove this language and then let's get the user information out as well so user is um users of um tweet dot author id okay so now i'll have the user and then i just want to print the username so let's also print user dot user name let's go ahead and run this this should hopefully give us perfect so it's giving us the user id and the username as well similarly let's say if you wanted additional fields so what you can do is uh above expansions here you'll see there's a field section and it tells you what fields you can request for users right so for user by default i get id name and username but let's say i also want to get the um let's do profile image url right so the way that you can do that is similar to how you use tweet fields let's add one more here and let's call it user fields right and in this user feels let's add oops this image uh profile image url and it's the printing the username let's print the url right so now um i'll go ahead and run this so it printed that so that's how you get uh the user information as well now let's before we move on to other things let's also quickly take a look at how we can get geo information right again um for getting geo information you need to use um a better token from um your academic project right so i'll go ahead and replace that real quick because uh these certain operators like has jio place country and things like that are not available to are under like uh the elevated uh package it's only available for um the academic package right now right so if i wanted to get let's say i want tweets that contain the word covered that are not retweet and i also want um uh the place country as us right so if i do place country uh us right this should give me an error right so let's try and um actually and let's remove all of the the user information um so i don't want the user fields let's first just run um as as is in c so i'll remove all of this um i'll just print the user id but before that it should give me an error because i'm using a better token that's not from the academic project so if i go ahead and run this okay uh so it says that this is an invalid operator place country is not available in this so now what i'll do is i'll go here oops um i'll go to my developer portal obviously this will only work for academics if you're not an um or if you don't have access to the academic pro product track you can go ahead and use a you can go ahead and use a different um uh you can you you can use uh you can forward ahead so i'll go in here um i already have a academic project and i have a project associated with it or an app associated with it so i'll simply click on uh keys and tokens right i'll just do regenerate so this will give me the better token to use here i'll copy this and i'll call it i'll go back to my config file let's just call it academic better token right academic bearer token and then i added that here now if i go and replace this here instead of doing better token if i do academic oops better token this should hopefully um work okay perfect so this worked because now i'm using the academic better token but now um i want let's say i want to get um some uh place information uh for each of these tweets where in addition to the tweet id tweet text i also want to know uh the place associated with these and so for that again you need to use the appropriate expansion so let's go back and um on our docs and let's see what expansion to use for getting jio information right so i'll go back into my docs uh twitter api fundamentals if i go into expansion now it shows me here geo.place id to get metadata about a location so go ahead here and add that expansion here so i'm removing this author expansion and this so let's look at what fields are available by default so if i go back here fields and i go to place let's see what fields are available by default the full name is available by default and the id okay so let's just use the full name for now um and i'll go in here so again uh what i need to do is uh let's pull out actually before that let's let's make sure that we're getting the places object here right so if i go ahead and debug this uh the response has an include section that has places great so let's pull that again similarly into a dictionary so let's do oops uh places right and then uh for p in um so it was response dot includes right and places and then let's use the id for it so p id um p okay so perfect so now i'll have this places um dictionary so let's quickly take a look at that uh perfect so i have those id followed by uh the actual object so it has a full name so let's print the full name for now right so this same way uh what we do is we check if um places contains the id for it and in the in the the response data it's actually part of the g object so the way that we'll do this is tweet dot geo and then we've got the place id right so if it's um available then we'll make a place object and we'll i'll just copy paste this thing we'll get that information and in addition to the uh tweet id we'll also go ahead and print the place um let's do full name right so if i run this hopefully this should uh work perfect so you see it's giving me um the tweet id and then it's also giving me the corresponding um well placeful name so so far um we've taken a look at how to uh search for uh tweets uh using recent search now if you notice that we've been getting tweets that are um like only 100 tweets what if you want more than 100 tweets right so we'll use uh what's called the paginator and the way that works is actually before that let me just remove the expansions for now let's just work with the default uh the default data that we get get back and also remove this for now let's just keep it too and also switch back my better token to my regular better token um remove all of this right so now we're getting 100 tweets back but let's say i want 1 000 tweets so the way that um you that works is that you essentially um i'm going to copy this or actually i'll start over so what we can do is um uh for uh tweet in right and then you can do client.paginator um or sorry quickly.paginator right and this page generator takes in um but before that let's specify how many um uh tweets we want to limit so we let's say we i'll say we want 1000 tweets um for uh for this uh search query right and so uh in terms of the actual method we'll pass it uh which method we want to use so we want to use client dot um search recent tweets right then we'll also pass it the query right and then we want per request 100 tweets back so our max results was 100. so now for each of these i'll simply print the tweet id so if i if i print this this should hopefully uh get me one thousand um oh i didn't specified query this query okay this should work now oh i i missed the the the brackets perfect so now this this is giving me um those one thousand uh today so that's actually how you get uh data that's more than um uh more than a hundred right um and then i i'm actually going through the list of um which uh methods to uh showcase next okay so before we get to full archive search let's take a look at how to um how to write these tweet ids to a text file right so pretty straight forward what i'll do is uh for each of these um actually i can do it here uh with uh actually let's give it a file name so let's call it file name is tweets dot txt right um and then let's do open that's oops yeah file name and then we want to append to the same file so i'll do a plus as file handler right and then for each of these we'll simply append it to this um this file so we'll do file name dot uh right let's write to it our file handler dot right file handler dot right and then we'll i want to write one line at a time so i'll do mod s uh backslash n for each of these and then i'll do tweet dot id let's remove this print or let's just comment it out for now so this should create a tweets.txt file with one tweet id per line if i go ahead and run this okay created this tweet dot text file let's go ahead and open this okay perfect so now let's look at how to work with the full archive search functionality again the full archive search is only available to academic researchers at this point which is why um i've replaced my token with the academic better token right now um let's uh let's actually write the code for this so the same way we'll use the client and then we'll use instead of search recent we'll do search all tweets right um and then let's assign the value back to a response right i'm going to remove this file name because we're not writing to a file now um this method again you need to pass it a query so we'll we're going to pass it the the query that we have uh we want max results um as um 100 um for this um example and then uh you can actually specify the start time and end time right so let's actually um mention the start time and end time here so i have an example here start time and then let's replace this let's call it end time and here we'll say we want tweets from 2000 uh 2021 to i know let's do five right and if i um actually yeah so this is getting me tweets that are older than the last seven days right now i'll just pass it the start time and end time here so start time is start time oops and then end time is end time here um and then let's go through the response again so for tweet in again response.data let's print the tweet ids again and so this should get me 100 tweets from uh this time period that have the words coin oh i forgot to print it so let's print it and this should print the tweet id so that's how you use a full archive search again there are certain advanced operators that are available to you such as place country um has geo and things like that so if i do place country um us or so on it'll uh further refine those tweets um as such so definitely check out the search all functionality that's available to academic researchers okay next let's take a look at uh how to get the tweet volume for a certain uh search query right so so far we've seen how to get the tweet id tweet text but what if you want to see how many tweets about a certain topic over a time period right so for that we have the um uh the tweet counts functionality that's available in the twitter api obviously there are two kinds one is recent counts and one is um all counts so all counts are like full archive counts full archive is obviously available to academics only and then recent which gives them like you can go back in time and get um the tweet volume for any topic um whereas for recent tweet counts uh you can only get tweet counts for the last seven days right so that's the one that we'll use in this example so what i'll do is i'll do plan dot um let's do counts and it has two options recent counts and then obviously tweet um the um all counts so i'll do recent counts for now and again same things i have to pass it a query so let's call let's use our old query um where we're using code not is retweet right and then uh in addition to so actually let's print this first and see uh what response um uh what what response we get from for this right so let's do counts is this right and then for count in counts.data let's just print the count um and if i go ahead and run this so it's giving me the time period um by default at like 15 minute windows right but what if i want the granularity to be one day meaning i want for the last seven days so i can simply use the granularity parameter gray a new literary and pass it the granularity that i want so if i pass it day go ahead and run this so for the last seven days it's giving me the count of tweets that uh contain the word code that are not retweets so that's how you use the um recent account functionality obviously if you want um the academic research equivalent of that then you'll use the um you'll use your academic better token and instead of using recent counts you use um all counts and then you also pass it the time window for which you want the tweets for next let's look at how to get a user's timeline right in order to use get a user's timeline you need the user's user id so let's first get the user id so to get the user id we'll do get users right and then you pass in the user names and then let's call in this case let's use um suheim actually let's do twitter dev so i'll do twitter dev right and then i can just call it users um and then for users or yeah for user in users i'll just print it and let's see what user id we get okay so this is the user id that i got for it i'll go ahead and i'll store it in my config file so that i don't have to use it over and over again um user id is um yeah this is my user id perfect so now let's see how we can get the user's timeline right so simply replace this with get or let's do um user um yeah so get users tweets is uh what the eq one for like getting timelines is and then instead of this i'll pass it the id right so i'll do id is config dot user id right and then again this is getting me all the tweets and again by default as we saw earlier by default it's only going to give me the tweet id tweet text if you want additional uh fields you basically specify those as tweet fields and and so on as well um let's just call uh lang right and now for tweet in tweets dot data i'll just print the tweet idn tweet dot lane again so if i run this this should get me all the recent tweets um for um for this user id um and it's also printing the the link associated um with it so that's how you get a user's um users tweets um if i put a debugger here you can actually see uh that in the response if i go back here uh in the meta you see by default it got me a hundred or ten so i can obviously include more by specifying the max results and getting more data similarly if you want to get um users mentions so you can simply do get users uh mentions and same thing you pass it the user id and you get the get the get the corresponding tweets for the users mentions um next let's take a look at how to get a user's um followers right so the way that you do it again you specify the client and then after that i'll just go ahead and do client dot um we want followers so i'll just type followers so there are two options it gives me get users followers right so again i pass it the id and the id is config dot new oops user id and then i'll assign the response as users um and then i'll just uh go through the response to see what we get by default so then i'll do for user in users.data and i'll just print the user right actually before that i put a brick breakpoint here and if i see if i debug by default for each user it again same thing it gives me the id name and the username if i wanted additional things so if i want the user's um profile image url as well so i'll have to do similar things i'll have to pass user fields um user fields and then pass it the the different fields that i want available so in this stage i want i want uh profile image url so i'll do profile image url right and then here i'll simply print instead of user i'll just print the url here so if i go ahead and run this this should work yeah so if you see here it got me the the image url so this is how you get the followers but what if you want to get the users that this user is following right a small change so you just do uh follow a get users following and if i go ahead and run this this is getting me the users that this user is following all right next let's take a look at um a list of users that like a certain tweet right so let's get a tweet id real quick uh if i go back to um twitter here let's get this um tweet id here um i'll copy this i'll add it to oops i'll add it to my config file as well so let's just call it tweet id right now in um in uh my code here what i can do is client dot um and i want um users that are liking a certain tweets right so i'll just do like so here liking users this will give me users that like a certain tweet and then i'll just pass it the id and the id is config.tweet id and this will give me all the users so let's go ahead and do users and then just print the user username so for user in users.data i'll go ahead and print user.username right and if i run this this is giving me the user names of people that um that liked this tweet now if you want to get the retweeters for a certain tweet meaning that uh who retweeted a certain tweet similarly you'll just do users and then you do client dot i think it's called retweeters get retweeters yeah and then you pass it the id id was config dot tweet id right and then again just go through the response so for user in users.data i'll just print the user id user.id go ahead and run this so it's printing the user ids for people that like or that retweeted that tweet now if you want to get tweet information about a tweet using the tweet id you can do client dot um i think it's get tweets oh yeah get tweet right and then you can pass it the id um so i'll do config dot um uh the tweet id um and so yeah tweet id and then i'll just do my response here oops and then i'll just do a print for the response and i'll add a debugger here so if you look at the response again the response by default has data right and data and the idn tweet text as well if you want additional fields you'll pass the tweet fields if i do tweet fields and then i'll say i want when this tweet was created at as well right and then if i do um uh tweet is equals to response dot data right and then i'll just print the tweet dot creator that this will print the time the tweet was created right so similarly you can get additional information about the tweet so so far we've seen examples of searching for tweets looking up tweets with tweet ids uh uh getting a user's timeline but now let's look at examples around uh creating a tweet retweeting it liking and so on right in this case we'll uh initialize the client slightly differently and we'll actually be using the access token and access token secret that we had initially um set up uh in the beginning of this video so the way that we initialize the client for this is we'll call a client and then we'll use qp.client and instead of passing the very better token this time we're going to use uh the consumer key right and so from config we got the api key then consumer secret which is apa secret then con or um the next thing is access token so config.access token and finally the access token secret so config access token secret right now um we have our client ready to be able to create tweet retweeter and so on so forth so let's start with the create use case so we'll do client dot create tweet and then we'll pass it a text let's call it hello world right and then let's get the response from this as well um and then simply let's just print the response so if i go and go ahead and print the response and i'll go ahead and run this this should create a tweet for um for us if i do go ahead and run you see it created this tweet with this tweet id right so if i go in my um twitter app i should be able to see that tweet that says um hello world so right there that's the tweet right what if you want to create a poll as well right so um we can um use the poll uh duration so this is like 60 minutes um that i want to create a poll for 60 minutes and then there's poll options and so the options can be yes or no right and if i go ahead and do this should create a poll with the title hello world um yes and no or yeah i'm gonna go ahead and run this so it did create a tweet let's see um if it created the poll here and so it did create um this poll for me right so now let's look at um uh this other or this old tweet and let's see how we can retweet it right so if i go back here i can remove all of this and i can just do retweet and then i can pass it the tweet id and set it to the street id right if i go ahead and run this you see it says retweeted true if i see it has that one retweet now from me now let's go back what if you want to reply to this tweet right so you can simply do a create tweet right and then you can do in reply to tweet id and you can say that i want to reply to this tweet with the street id and then we can use some text and just say hello so i go ahead and run this this should give me a response to this tweet right there so i hope this tutorial was helpful to you um if you found it beneficial please share it with other students researchers and developers who are interested in using the twitter api in python also don't forget to check out the documentation for tweepy on docs.pp.org and finally don't forget to check out our documentation on developer.twitter.com if you have any questions you can always reach out to me on twitter my twitter handle is sahim parak thanks again for watching this video

Channel: Suhem Parack

Views: 76,352

Keywords:

Id: 0EekpQBEP_8

Length: 43min 28sec (2608 seconds)

Published: Thu Dec 16 2021