Python and Redis Tutorial - Caching API Responses

hey everyone welcome back to another tutorial in this video i'm going to show you how to use redis now what is redis i have the web page pulled up right here at redis dot io it says redis is an open source in-memory data structure store used as a database cache and message broker now in this video i'm going to focus on how to use redis in order to cache your data now why is this useful well the example i'm going to use here is if you remember in the previous tutorial i showed you how to build a dashboard using the iex cloud api now each time you hit one of these web pages in the web application we turned around and made an api request to iex cloud and the way iex cloud works is it actually charges you a certain number of allocated credits so you only get a certain number of api requests per month and many different apis will throttle you or limit the number of api requests you make per hour right and so if some if you have a web application that's used by you know thousands or millions of users a lot of people are going to be hitting this web page and getting the exact same data so there's no reason to really turn around and hit the ix cloud api over and over again in order to you know get these fundamental ratios or quarterly earnings because that data doesn't change very often it only changes you know once a quarter or maybe you know maybe there's a revision and it changes a couple more times but there's no reason to keep hitting the api in order to get the same data over and over again so for this reason it's helpful to just cache this data locally and be able to return it directly back to the user that requested it that way you don't have to make this round trip to the ix cloud api and get charged for credits over and over again the second reason is that it's much faster so retrieving information from this in-memory data structure store like this it's super fast so you can serve up thousands of requests like really fast uh right back to the user without hitting a database or without hitting another api and making the round-trip network request so it's very fast to serve a data from redis and so those are two reasons why you might want to cache your data and so how do you use redis how do we get started with it the first step obviously is to install it i have os x so i have my own installation instructions so uh for os x you can use homebrew and most people that develop on osx already have homebrew and look up how to install homebrew if you don't have it but you can just do brew install redis and that'll set up everything for you download the dependencies and get it installed and after it is installed just like that i can just type redis dash cli and i have a command line interface for redis just like that now if you're on windows you want to google that and there's a number of people that maintain redis installations i believe microsoft uh maintains one installation and then there's this link here and you'll want to look up how to do an installation on windows and there are some redis binaries here alternatively if you've watched my channel in the past you know that a couple months ago i created this docker container this docker image called a trade kit at github.com hacking the markets and i included a variety of dependencies that i use including you know databases technical analysis libraries brokerage api packages and so forth and one of the packages i included was redis because i thought it would be very useful and so if you're using this already and you have the docker container you can use that and so i have this installed in my projects directory already so in project tradekit i have this cloned from github and i just do docker compose up and i bring my container up and you see i already have a timescale db a web application and so forth so that starts my docker container and then i just do a docker exec it if i want to jump on this container you can see let me see the name of it a trade kit web one so on my system i have trade kit web one and then i just start a bash prompt there and you can see i can type red cli looks like it's not started and so i do s in a d redis server to start a redis server so i'll quit that and let's see x in it d redis server start and it'll start a redis server here and then now if i do red cli i have command line access on trade kit here and i can start using it so the first thing i'm going to show you how to do is how to use redis from the command line and so that we understand it at a more low level and secondly we'll use the redis package in python in order to interact with redis from python and then third we'll modify the iex cloud dashboard application that we created previously and show you how to cache api responses in redis and how to serve a data up from redis to the website um and if you are using another api besides ix cloud and you didn't follow that example you know this is still going to be very useful and i recommend you still learn how to use redis because it's a very handy tool to have as part of your toolkit so let's go ahead and get rolling with this so i have the redis cli running i'm gonna go ahead and clear it and run it again so here i am on the redis cli and i want to see what's currently stored in redis right now so i'm going to type keys star and so this is saying list all of the keys that are inside of redis right now and you see it's completely empty and so this is a key value store it's the first thing to understand and so a key is just some arbitrary unique name that i thought of and then in that key i'm going to store some value and that's it's as simple as that and so let's say i want to set a key equal to some value so let's say i want to cast the user's username and instead of looking it up in the database or in some other place i just want to cache it in redis and serve it up from there and so i just need to set some key equal to some value so i'm going to set user name equal to part-time larry and let's see how that works and so now if i type keys star you see there's a key called username and so let's set some other thing let's set a password which he would probably never do but i'm going to set my password and my password is obviously a q-w-e-r-t-y right and i'm going to set a password and i'm going to do keys star and so my username and password are cached in the database right and so if i want to retrieve those values i just get it by key so i do get username and that'll return part-time larry and if i get password i'll return qwerty123 and so i'm just setting keys and then getting them and so you see here i'm just setting a key equal to some string value and that's good enough for the tutorial that we're going to be doing here so that that concept right there is very powerful on its own just so you're aware though there are a variety of other data types available in data structures so if i search for h set here you can see you can set the string value of a hash field and so h set here you can set my hash and then some add some attribute equal to some value so if i did h set user username part-time larry and then i did h-set user password query123 you can see then if i do h get user or h get user username right there's actually a more complex data type here so we actually have is a hash called user and it has a username of part-time larry and a password of qwerty123 uh in addition uh look you could do h getall and you can get it get all of the keys and values for this one data structure here so user has a username a part time larry and a password of qwerty123 so a little more complex data structure there likewise there's a list functions here and so you could actually manipulate some lists so you could store like an array inside of redis as well so for now i'm just going to use a string data type because what we're going to be doing is taking a json response which is really just a string in a way and we can just store that string under some key in redis and so if i let's say i have a json response and it represents fundamental data right and i just want to set fundamental data so set a key called fundamental data equal to some long json string right so i could do let's say it has a profit earnings ratio of 100 in here and some profit to earnings to growth ratio so a peg ratio of 1.3 or something like that right so that's a key in value as well and so this is just storing a json structure inside of a key called fundamental data right and then i can get this json back out and convert that string back into a python data structure that i can serve back up to the user and i can process this json string here and maybe i want to render it as a dashboard widget so you can see how i could get an api response as json like this and store it inside of a key now there are a couple other considerations to make here because let's say we have lots of different stocks in which we do right and so if i looked up the fundamental data for microsoft and i stored it in a key called fundamental data and then pulled it out that would work really well but what happens when i change symbols to apple and i retrieve the fundamental data for apple and i store it in a key called fundamental data i would actually overwrite the fundamental data from microsoft and lose that cache so you want to use keys that are unique so if you're storing microsoft's fundamental data you might do a set microsoft fundamental data right and then that would be some microsoft's fundamental data would be in here and then you might want to set apple fundamental data so you want these unique keys right with unique values so that they don't overwrite each other right and so i have apple fundamental data right and so what we'll be doing is dynamically generating these keys so that we can find them in the future and storing the values that are unique to each particular stock and then so now let's look at the keys here and let's talk about how would you get rid of now that i've decided these keys are what we're going to use i don't need this fundamental data key so i can type del fundamental data to remove that data from the cache so it's great that you can cache microsoft and apple's fundamental data inside of redis and serve it right back up to the browser but if you keep fetching this data from the cache eventually it's going to be outdated right i could cache someone's username but what if they change their username even though they don't do this very frequently eventually you want to update the cache and refresh it and so in order to handle this you should pay attention to cash expiration so if i type expire here you can see that you can expire the cash which means the data will go away after a certain amount of time right and so if i look at expire i can set a key but then i can expire that key after a certain number of seconds so if i do expire microsoft fundamental data and i put 10 that's going to expire microsoft fundamental data in 10 seconds so if i wait 10 seconds here and then look at the keys again you'll see that the microsoft fundamental data key here is going to get deleted right so you can see now it's already gone and let me do this for uh apple as well right so i'll do expire apple fundamental data in 10 seconds i'll type keystar you'll see apple's still there i'm going to fetch it from the cache i type it again i can type apple fundamental data i'm going to get it and eventually that's going to return nil right and so now i look and you see it's no longer there so we're going to do in our web application is we're going to fetch data from that key and if that data is available we're going to serve it back up but if it's expired right if it's expired and returns nil then we know to hit the api again right fetch new copy of the data and then store it back in that key right and so we can use this cache expiration to control how long this data gets cached so maybe we'll cache it for 24 hours so we'll multiply yeah there's 86 400 seconds in a day i believe 24 times 60 times 60 and we can just expire things after 86 400 seconds and then you know that's the equivalent of caching data for 24 hours and then the next day rolls around right we go to fetch that data again it's no longer in the cache so we say hit ix cloud and get a new copy of the data and then cache it again and we're good for another 24 hours so let's go over to the python side and we'll see how to do that in the python code and add it into our web application and we'll be done so the first thing i'll do here is i'm going to create just a simple script standalone just to show a very simple example of using the redis package from python so if you type python redis there's a python package as usual there's always a python package for everything and so you just do your pip install redis so i'm just going to do a pip3 install redis on my system here and then make sure you have the redis package installed and then once you're inside of a script where that has access to your environment i'm going to import redis right that imports the package and then all you need to do is create a new redis client here and so this has r equals redis dot redis okay so i'm going to call this client though because i sometimes i use r when i'm working with requests i use that variable name a lot and so i'm going to just call it a client or you can call it redis client or whatever you want so i'm going to let's do one called redis client red is under short client equals redis and then we create this redis object and we're connecting the local host since uh redis this redis server is running on our local system and the default port is 6379 right and so that should give me a connection already and if i look on my local system right we have some keys already right and then you see we have one called username and password let's see if we can connect to it so i'm going to do redis client dot get okay so that this redis package here has uh basically python methods for accessing uh redis right so the same way i typed get and set from the cli you can do restclient.get to get some key and then you can set some key equal to some value and it exposes all of these functions so if you look at h set these hash maps and all that there's there's functions for each of those right and so you can go through the documentation to see how to use all the different types here but all i'm really using is get and set so if i do redisclient.getusername here and then i'll store this in a value so username equals redis.restclient.getusername let's see if we can print it out right so we're going to get it and so you see i printed it out i ran the program and it printed a part-time layer there so you can see my python script is connected to the reddit server and i was able to retrieve a value and so now i'm going to do redis client dot set and i'll set uh microsoft fundamental data so i'll just make something up fundamental data and then i am going to give it a python dictionary and i'm going to do a json.dump so i'm going to dump a python dictionary right in and so i will also need to import json to do this and so i'll say the profit to earnings ratio is 33 right and so i'm going to dump a json string so i'm going to get a python dictionary convert it to a json string and i'm going to set the key microsoft fundamental data equal to that string and let's see how it goes so i'm going to run this didn't get any errors now if i go to my command line just to check and i do keys star you see i have a key called microsoft fundamental data and then i'm going to get microsoft fundamental data mental data and i should see the value that i set right so you have i have this a json string called pe33 and so you see that's very easy you can see the interface for using redis is very simple and it's very simple to use it from python it's basically the same commands but there's a python object here where you call some methods so now all i need to do is go over to my dashboard here and so i am using streamlit so i'm going to use streamlinkrun dashboard.pi get this guy up and running again and so you can see currently if i go to my dashboard and i go to my overview it's hitting the ix cloud api each time right so each time i go to the overview screen it calls a stock.get logo and this is making a request to the ix cloud api and using my api token which is charging me for credits and it's more slow because i'm making a web request and returning that back and then returning it to the user so i want to do what i want to do here is take this reddit redis client and put it inside of my web application so at the top here let's just put it at the top i'm going to add a connection to my redis client okay so i'll do redis client equals res.redis localhost 6379 and i'll import redis up here at the top and then inside of my overview here this part is calling iex cloud right and so what i'm going to do is after i get the logo i'm going to do redis client.set and i'll call the key logo and i'll just set uh that key equal to the response from iex cloud let's see how that works so i'm going to just use my rest client and set the logo next time i refresh this page okay so i'm going to do i'll type apple just to make it load a new one right so this gives me an error so it looks like i have a dictionary getting returned from my api this little wrapper class that i wrote previously so it looks like this is actually a dictionary coming back for logo and so i'm going to do a json.dumps here so i'm going to convert this to a json string and it looks like i'm importing json already and so now let me refresh this one more time so i'll do let me change symbols right okay so now it actually ran successfully and i'm trying to set a key called logo and yeah let's see if that works i'm going to go to my cli and i'm going to do keys star and you see now that i have a key called logo down here and so if i do get logo let's see what's in there i have a json string in there just like that uh where i have a cached copy of this microsoft logo right and so if i put that in the browser you see i have this cached uh instance there which is great right so i have the logo for microsoft uh right there and so now the next time i refresh the page let's think about what happens here so if i refresh the page here and what will happen is it's going to call the ix cloud api again so even though i'm setting the cache value here i still have this call here so what i want to do is instead of calling ix cloud what i'm going to first do is try to do redis client get logo right rest client get logo so i'm going to say logo equals redis client get logo and i'm going to comment out this part that calls ix cloud right so i'm going to do redis client get logo and that should be cached right and so now let's rerun it again so when i run that i try to get that json string from redis and reload it and he says it says byte indices must be integers uh it says it doesn't want this string here and so remember what i said previously um my existing functions here returned a dictionary right this returned a dictionary and so i'm expecting this to be a dictionary right i do logo and then access the url key when we wrote the original dashboard right now i just have this json string that i stored in redis and so what i want to do is do json.load so i can do json.loads res client get logo and that will convert it back into the right format let's see if that works okay and so now it pulled uh this that json string here i did json.loads it converted it back to a python dictionary and then it was usable uh when i accessed that those dictionary keys uh in the dashboard so you can see i was successfully able to cast the logo and then load it back up okay and so now that i can get the logo like that so i'm gonna do logo equals rest.clientgetlogo okay and so there's going to be situations where the logo is already there in cache and situations where we don't have the logo in cache so what i want to do whenever i load the dashboard is say try to get the logo from the cache and store it in logo and then i'm going to say if logo is none right if it couldn't find the logo in cache only at that point am i going to fetch it from iex cloud so if we didn't find the cache let's get it from iex cloud and once we get it from ix cloud i'll do redis client dot set redisclient.set logo to json.dumps logo right so we try to get from the cache if it's not in the cache get it from ix cloud and cache it right otherwise we're going to say logo equals json.loads logo right so try to get from the cache if it's not there get it and set it otherwise that means it is in the cache so that means logo now has that json string and we can load it up and use it right and so let's go ahead and print what happens here so logo has none i'm going to say print could not find logo in cache retrieving from iex cloud api okay and then otherwise i'm going to say print found logo in cash serving from redis okay and let's see how that goes so now i'm going to run it so i'll run it here okay looks like it already happened and it says found logo and cache serving from reddit so you see my console here says it found it in cash serving it from redis okay i'm gonna rerun it again right so basically every time i load it now it's gonna say it found it in cash so if i go back to the overview here it's gonna print again found logo and cache so it's just going to keep serving it up from cache and so now if i go to my command line here and delete the logo right that means it's not going to be in cache and then i'm going to load this guy back up again so let's say or let's say i load a different symbol apple right it's not in the cache could not find logo and cache retrieving from iex cloud api so it's hitting the api only when it can't find it in cache and so now if i go to get logo here in the command line you can see it has the apple logo in cache and remember i mentioned earlier that you want to use unique names for your keys and so i don't want to just call this logo right i want to call it i want to make a naming convention for the key so that i can store all the different logos that i request separately and so instead of just doing client.getlogo let's make a variable for the key so i'm going to say logo key equals and i'll do an f string so i'll do an f string and i'll use the symbol like whatever symbol they type in i'm going to use that in the key so i'm going to do symbol dynamically like that underscore logo and then i try to get the logo key from the cache so i'm going to try to get the logo key from the cache if i couldn't find it i'm going to set logo key equal to json.logo otherwise i'm going to load it from the cache okay so yeah let's see how that works right so now i'm going to do this i'm going to first of all i'm going to set the default to load on the overview so i'm going to do index 0 there so that when i rerun this it'll stay on this overview okay so now you can see it could not find the logo in cache that's because i created a new key right so it retrieved the data from ix cloud and this should have set a new key inside of the redis client so if i do keys star now you'll see i have a key called apple logo right because i made that naming convention right and so if i run this and refresh the page for apple or i refresh the page it looks like my default symbol is microsoft it's going to say it couldn't find the logo in cache because it's looking for microsoft underscore logo right so now if i do keys star you see i have one for apple logo and microsoft logo okay so there's that microsoft logo and here's the apple logo right and then now no matter how many times i refresh the page right now it's going to find the logo in the cache and serve it from redis and if i switch to apple it's going to find the logo in the cache and serve it from redis and then now if i do a new one like netflix right it's going to say could not find logo in cache so it's going to retrieve the logo it doesn't know about and it's going to make a new cache key and you see what i have one called netflix logo right so i know it's a very simple concept and i hit it a number of ways but you get the idea that right you try to cache some data you try to retrieve it back from the cache if it's not there set the new cache key and then the next time around it'll be there in the cache right okay and so that's the simplest example and that's pretty much all we need to know and the other thing i wanted to cover was the expiration time and so let's go ahead and set an expiration as well for one of the other keys and and you can do this for fundamentals as well i'm just going to keep on the overview it's the exact same concept so we also fetched a little information about the company and so you see here i have stock dot get company info right and so i'm going to use this exact same pattern let's say i have a company key and then i'm going to set that equal to an f string and i'm going to use a symbol and then underscore a company right so i'm gonna create another uh key just to store this uh description and the ceo and let's say i want this cache to expire um once a day or let's make it a shorter time so that we can see it happen so i'm going to say i want this cache to expire in 30 seconds right so i'm going to use this exact same pattern that i did here so i have a company key i'm going to try to find the company info so i'm going to try to find it in cash i'm going to do redis client dot get company key and i'm going to say if company is none then i'm going to get the company info from ix cloud so i'll do print getting info from ix cloud and then i'll set it in the cache so i'll do redis client set so at this point i'll set the company key equal to company and then otherwise that means it is in the cache and i can do a company equals json.loads company right and that will load the string from the cache and then i'll print that out as well i'll just say print getting info from cash okay and i can do that exact same thing and so now i'm going to rerun and you see invalid input of type dict so here oh i didn't set i'm going to do json dot dumps there so i'm dumping the string let me try that one more time okay cool and so now if i go to my command line i'm going to do keys star and you see i have netflix company as a separate key so if i do get netflix company get netflix company you'll see i have all the information so this full json response about netflix with its description is stored in cache and so you can cache these very large strings and lots of data inside of redis okay and now let's say i want to expire that data so after i set the data so what i'm going to do is delete netflix company i'm going to delete that key so i'm going to do now is after i set it i'm going to do redis client dot expire okay and then i need to give the key that i'm expiring and then some time delta so i'm going to do a company key so i'm going to expire a certain company key this company key that i generate at a certain time and so to do that i need i'm going to use a function called a time delta so i'm going to import time delta as well okay and this lets me set a certain number of time in seconds or days or whatever amount of time i need and so i'm going to say redisclient.expire i'm going to do time delta and i can do hours or seconds or days and so i'm just going to say seconds equals 30. okay so i'm going to set a 30 second expiration time and let's see how that goes so i'm going to rerun this and it says no module name time delta so i think this is part of date time so i'll do uh from date time import time delta let's see if that works so i'm going to rerun that okay i think that's right so i do that and i have microsoft and i do keys star and you see i have microsoft company and then if i also refresh the page you'll see hey my console it says getting info from cash so it's in cash i'm gonna refresh it says getting info from cash refresh again getting info from cache refresh again getting info from cache and then eventually this should expire and then it'll hit ix cloud again to get the data again so do it again so eventually i refresh it and it says getting info from iex cloud so the cache expires that data is no longer available so i do a company company equals redisclient.getcompanykey and eventually you get none because it's expired and then i just refetch the data from ix cloud and then i set it and cache it again for 30 seconds or one day or however much time you need and just like that that's really all i wanted to cover today and so you can cache all the different types of data like news fundamentals and overview and cache anything you want whether it's an api response like this you can technically cache you know the entire rendering like the way a whole web page looks you can cache just certain numbers and you can even cache some time series data or you can just cache an api response let's say you build an api that just returns json and you have your own api service you don't want to hit your database over and over again you can just return the json response directly from cash to some api client and that's it that's the way you cache data in redis both to save in terms of iex cloud credits but also just to save time and serve up data much quicker to the end user or whoever is consuming your data service so thanks a lot for watching and stay tuned for the next video thanks

Channel: Part Time Larry

Views: 12,227

Rating: 4.9796953 out of 5

Keywords: redis tutorial, python, iex cloud, caching, data, cache, key value store, stocks, stock trading, algotrading

Id: _8lJ5lp8P0U

Length: 32min 25sec (1945 seconds)

Published: Sun Mar 21 2021