Normalizing Data from Multiple APIs in GraphQL

so I wanted to give some context on what you're about to be watching I was on a call with one of my patreon supporters it's for one of the reward tiers where I do a coding session where I answer any questions and kind of go over code and whatnot so in this video it is a screen share where I'm sharing my screen and I'm on a call with one of my supporters and in this session we go over how to handle multiple api's in graph QL and how you would go about normalizing that data and then how you might go about actually cashing the data so you're not fetching from the api's every single time so you're gonna be watching a recording of that and so without further ado I'm going to just run that right now so all right so I guess if you want to just start off with what you kind of want to go over and I kind of prepared a little something that goes over kind of like how graph QL you can fetch different from multiple different api's but if you wanna just talk through what you're trying to do I think it's exactly that so it's really how do you kind of hit two different API so an example might be where you're building a recipe app and you want to pull recipes from two different api's but again those api's are going to have different name properties so how do you kind of resolve those different types of things together so they might handle dates differently or they might call things differently so how do you kind of hit these two api's and kind of merge them together and then potentially then kind of integrate that with a calls to Prisma as well yeah that makes sense yeah that definitely makes sense and the nice thing about graph QL is it's actually relatively easy to do that sort of thing / it'll lets you be fine-grained about where your data is getting from different fields so it's very easy to say these fields I want to get from this API and these fields I want to get from another API and do you are you a JavaScript or a typescript guy JavaScript okay cool that's I didn't know so I went ahead and just wrote this little example in JavaScript to be safe great and this is basically just the Apollo server example that they have in there started and I just added a simple field on to it to show you like this is the most basic level of fetching data from two different api's so what I have right here is I here's my schema so all it does is it has a single query called advice and advice is a type and in my advice type I have to I either get advice from Kanye or I get advice from Chuck Norris and those are just strings but they could be their own data types if you want to - and then if we look at the resolvers for this so notice like in my advice one I'm actually not even resolving any fields I'm just returning an object that's empty and then what I do is I actually resolve the individual Kanye and individual Chuck Norris fields so inside of the advice object type I guess we have Kanye and we have Chuck Norris so these are just resolver functions so in this case I didn't really need any parameters from the resolver functions and they're asynchronous you can so that allows you to fetch from the API and so here I'm just saying fetch and here I'm hitting the Kanye API and getting the response getting the JSON response from it and I'm just saying data not quote and then say Chuck Norris I'm doing the exact same thing fetching from the Chuck Norris API grabbing the JSON for it and I'm returning the value of it so if we like take a look at what it looks like to actually run this guy it looks something like this so we have curly braces advice I'm getting from Kanye and being from Chuck Norris and if I run that I'm going to get a quote from each one of them and again there was the fields are resolved individually so if I only wanted a Kanye quote we are only hitting the Kanye API with that and if I only want to Chuck Norris quote get rid of that we can do that and then we're only going to hit the Chuck Norris API or we can do both and we'll hit both api's so does the basics of that makes sense basically that does make sense cool so that's basically where you're gonna go is you're just gonna expand off this concept so you're gonna take the individual fields that you want to resolve and obviously right now this is this is basic cuz I just have strings but they could be their own types so for example if I was fetching for Prisma I might have a hole Prisma type for a recipe for example and maybe I'm returning the recipe from Prisma and then I just return it from here so you create a schema type for it and then you resolve the fields like that and so for obviously in theory the Chuck Norris API and the Kanye API those might have you know 20 30 properties each and would you then if you want to do that would you then just create one function from each one of those that you want to hit so we can we can take a look and we can like we can we can take as many fields from the API as we want or as little fields as we want from the API right so for the Kanye one it's only giving me a quote an ID let's see how what we get from the Chuck Norris one for the Chuck Morris one so I have more IDs here or more values here right that I have access to so it really depends on how you see using the data so I could normalize this somehow if I wanted to or I could just create two different types so like I could create a Kanye response right so Kanye responds and it has an ID which is a string and I believe it called it quote and then we could do that here and then I can just return that data down in my resolver so if I were to restart this now I can fetch an ID and a quote from Kanye now and we have to refresh this too and now we get both of those so you can pretty much grab as little or as much as you want from the response the idea is you grab the data from the API and then you can normalize it in some way that you like you don't have to normalize at all like right here we just exactly matched the response ID quote but you could just pick off the fields that you want if you want to or you could let the user pick off the fields because it's graph QL and I could just say I only care about getting the quote I don't care about the ID so you can just grab it like that and then we're just getting the quote here so you can kind of choose where you want to do your normalization and do you do the normalization afterwards so again an example maybe where I didn't exactly catch the two api's there but you might have for a Kanye you might have ID and you might have quote and for Chuck Norris you might have ID and quotation and then you actually want to use the same guys and use the same type for both how do you kind of just hit those individually or what would you be your process for handling that okay so we know yeah that definitely makes sense we can go over example I would say you could approach that in like two different ways so here's way number one way number one is if you want them to have the exact same response you can normalize them to the same type so for example I'm just going to call this a quote response and all it's going to return is a quote in here because I don't know if Chuck Norris had an ID yep Chuck Norris has an ID so we could do an ID and a quote for that so quote response in Chuck Norris so they're both gonna return quote responses and so Chuck Norris doesn't have the same names right so he uses value instead of quote right so what I can do here is I can return it and I can say the ID is dated ID and the quote is data value and so this is say our normalization so we fetched from the Chuck Norris API and then we normalized it like that and the other thing you could do is yeah I think that's the best way to do it actually I was gonna say you could actually create a resolver for the quote response but you don't know whether it came from Chuck Norris or like Kanye I think the best ways to do it probably right here after we fetch from the API normalize it like that now you could also tell me I don't want to normalize it I want two different types of responses and then you could do like a union in that case I don't remember the syntax for unions off the top of my head but you could do something like go back to our Kanye response he has these values and then you have a Chuck Norris response called Chuck response and he has those two fields and then we may want to capture more like let's say the URL and the icon URL so he has way more fields than he does and he has different fields so we could return a Union from this so we could either like make this a Union I want to say the syntax is something like Union I guess I call this quote Union response and that's either a Kanye response or a Chuck Norris response I don't think that syntax is right well let's just see if this there is an air at us if not we can google it real quick type is missing a resolve type resolver oh yes so I think my syntax may be right but I didn't add so whenever I don't have you dealt with the union's before I have not so well I guess we can we can either go down this rabbit hole or we can we cuz there's a one other thing step we have to do for a unions are you interested in the Union rabbit hole before we go down that rabbit hole well I think that I think it really depends upon what the benefit of it is so the example you're showing before it's where okay you hit the Kanye PR you hit the truck Norris API it kind of those go back to a single type close quotation I think it was and then the user could then then go and just kind of hit that quotation query and then it would hit golf peak and I think that makes perfect sense is there what is the value of doing a union on top of that so I guess so our example here is kind of probably probably like right here is not where I would use a union let me put this back I think the quote response is probably the best thing for this but you could tell you could have a field here called Kanye or Chuck quote right right and then that returns a union and so I think that makes a lot of sense basically the idea is if I get a Kanye response back I want to grab these fields if I get a Chuck Norris response back I want to grab these fields so you'll use them in kind of two different ways right if you want to normalize and have one type of recipe then you would you do the first technique if you wanted to say I want to have like three different types of recipes you may get back you´d one drives a union and I think what I'm trying to do and you can tell me which one to make senses is I want to really kind of just hit multiple api's and say I want to show me recipes that use mozzarella cheese and like and then two different recipes site api's and bring back whatever recipes they have for mozzarella cheese yeah and how to and then bring those back together into a single response where I would return the ten from one side the 20 from another site and then you'd get all that back in your graph QL your query okay yeah definitely I think then maybe so like you don't even have to do something like this I think so for that case where it sounds like is I would probably say normalization is probably the one I would go for mainly just because I don't think it let's say you're getting it from ten different sites I don't think it makes sense to have ten different response types or something like that I think you probably best just to normalize the ten api's into a single recipe that you generically use across I think that makes the most sense and so like in the case that we wanted to do something like that I think it looks something like this so let's say we want to say like search recipes is what we're gonna call it and that takes like of query and it's gonna return back in a ray of recipes right so I'm guessing like is this kind of like the type signature that you're looking for exactly okay so for something like that I would even say I would just have a single resolver down here called a search recipes we could close this and so this would be like we don't care about the parent which is the first parameter and the arguments is gonna be query and then I would just call the five different api's or whatever so I would say like and again if you want to make them all.i sounds like also I'm guessing that you want them all to run in parallel when you're hitting the different end points I think ideally that would be great no reason to wait for one to finish the next load next one okay so let's say these two are two recipe endpoints I would say instead of putting a weight there I would say promise one promised two and then you could just do this for every single API you have and if you need to like you can pass in as a parameter here the query because the API probably takes a query or something like keyword equal query and then at the end I would say you'd have basically array of data and you can await say promise at all and you have an array of data because after you fact each one of those you would add it to that single data array exactly so now yes so this is an array so you could you could even d-structure this if you want to and you could say recipe site response one recipe site response two and so on because you may need to normalize from this point right cuz now you have like five different responses from five different recipe sites and you want to normalize them into a single type of recipe but just like the example used above you might still need to do again like where the Chuck Norris site all that value and the Connie as I called it yeah well you would still need to do when I do that initial fetch for the first API in the second API that would be the place where I would then map that before or after so I would I would literally do this sequence right here so I would fetch them all so I basically have like five different fetch queries or whatever there and then I would wait for them all the respond and get back and then I might have name and this is called recipe name whereas from this from the second site right it's called just name right and so I would do that for every single one so this one might call the servings yield whereas this one calls it just servings and so I would actually this is what the the stuff would look like I would do and you may again this may be a for loop if you're getting back you more than one recipe from the response but this would be the fetcher data result like get get it back from all the api's and then normalize it fantastic that makes perfect sense cool and in that case that from the destructor standpoint the recipe site response one is the results from p1 and the recipe site responds to as the results from p2 exactly yeah we could and we could just rename this like P do data write whatever you want to and then so like yeah I don't think you even need to get into for this particular use case like in this case I have it advise resolver where it's going like a level like I'm resolving the advice type fields I don't even think you need to resolve the recipe fields I think you can put all your logic in the search recipes resolver right and so and again last part is that within that return loop for each one if I'm returning multiple recipes from each site I would just have a where I have the name p1 data recipe name and surveying p1 that would be a loop that would go through all the data in p1 data exactly so it may look something like this so let's say we have what we can call it normalized p1 data is equal to p1 data recipes map and then for each recipe we are doing this right and we're doing X dot recipe name so now we have we're looping through the array of recipes that we get from the first sight and we're saying let's get that recipe name let's get that yield and we're setting the values to name and servings and now I have array of those alright so I could just say dot I don't know if you're merely with the dot dot dot syntax yep so you could just do that and so now you're just put it passing in all the recipes you normalize there and to the normal ice array so depending on the site if you're getting an array you map through it if you're just getting a single recipe back from the site you could just do a response like that so make sure you have this for here so yep so you went you said normal f21 data it's going into p1 data mapping it not returned in the name of the servings then how is that then being merged with the name aim and the servings in sight for the second time so you have normalized p1 data there is that being returned and integrated with the name of the servings for p2 data well we could just do the same thing for each one and then have that so yeah this is going to integrate together and let's say p2 data has is an array 2 we could say normalize p2 data and we could put the logic we have down here move it up here use p2 data and then we could spread it in there too so let's say your own let's assume you're only getting a raise back for each array of data you're getting you're just merging them all together at the end like that oh and this should be X because yep after we're mapping to the recipes each one's that okay well that makes perfect sense thank you yeah is there anything else you wouldn't wanted to go over you have any other questions well I think one extension of this little time is and you kind of brought this up previously is caching of this the data that's being returned yes and any do you have any videos that kind of touch upon caching or in this regard about Tesla you wouldn't off you keep making the same it's the API so what I would suggest in this case is I do I do have I think I have like one video that may be relevant to you that has to do with Redis caching but we can basically cover that real quick with I'll give you like two different ways you can do it real quick so there's there's this thing I mentioned to you called a loader but I don't think data loader will be relevant for what you're doing right now because right now really what you want to do and correct me if I'm wrong let's say someone searches pizza or I guess mozzarella so someone searches mozzarella you want to cache that so the next person that searches mozzarella you already returned the response to them and you're not hitting the API five times is that the idea so I think you're right about the data loader that might come into play in some level but I think that is the question is that something worth doing and is that something that can be done I definitely think it's worth doing and it definitely I like you can start very simple so the simplest thing I would I would suggest to you is I would create an object called cache like this and this is how I would recommend starting and then so this object is basically going to store previous data so let's say we have our query this is mozzarella we hit the API we normalize the data and this is like what we're gonna you know totally normalized so this is what we're gonna return right totally normalized so this is what we'd normally return to the user who ever searches for mozzarella we're just going to say cash is equal to the query is equal - totally normalized so basically what all we're doing is we're storing the data that we would return inside of our cache and the key is that query the user searched so then we can just add at the very top here if cache contains the query return it and now this cache object it's just an object sitting on our server so it's not gonna like it I don't know how big of like how many requests you expect to like get hit but that's where something like you could replace this with say Redis and you'll be able to handle some more data instead of just having it in memory but this will definitely hold you over at least to start and gives you a good idea of how it will work and you can just say like check the rest cache see if it exists if it does return the data in the cache and then here so here we're filling the cache here we're checking the cache and here cache hit and we're returning data from the cache so the idea is once we someone searches mozzarella it goes through here we're gonna fill the cache with mozzarella next time someone searches mozzarella it's gonna be in the cache we just returned it and the fetches never happen great and and typically then do you have those the cache cannot expire after a certain period of time because I'd imagine what a certain point you want to go back to make sure that there are any new you've got to be a sort of mozzarella yep that's another good thing to do so like with Redis there's a way to say like for example I only want this to be like like after you store it so fill the cache and it'll say expire after X seconds so you can choose how long it expires so if you want to expire in a minute or in an hour you could pick whatever time you want with an object I don't know the simplest way to do it maybe you just say you keep like another object called like a cache key or like like for example I want store like the date that the key was stored so maybe we have like date stored and you say dates stored is equal to new date so basically we're just storing the date that the query was first filled in the cache and then when we go to hit the cache we can say and oops and we can like basically here just check the date and we can be like if the date is no longer than one hour you're good to go so I guess that check would be like something like that but for for a long run I would suggest something like Redis or memcache would be your best but if you wanted to start with something simple like that you could yes the date stored thing feels a little weird I don't know if that's the best way to do that particular thing but something to that effect okay well thank you very much Ben and thank you for all the videos they're all very helpful and this definitely takes it to another level no problem is there anything else I can answer for you or nope I look forward to the next time we talk alright see you then okay thanks a lot bye

Channel: Ben Awad

Views: 5,293

Keywords: GraphQL, Normalizing Data, Multiple APIs, Multiple APIs in GraphQL

Id: hLcAlln-D_s

Length: 25min 11sec (1511 seconds)

Published: Wed Mar 20 2019