AWS Amplify with Amazon DynamoDB Single Table and Direct AWS Lambda Resolvers with Rick Houlihan

oh my gosh okay i i apologize uh folks we were kind of drifting through i'm walking through things nobody's hearing what i'm saying and i'm talking and talking uh so now hopefully people actually hear what i'm saying what i'm saying so we can go back through all that stuff okay i apologize folks i i'm actually for the first time i am running in a a dual mode where i'm actually got a a a co-presenter here nodder david is going to be joining me in a few minutes and i guess when i went into my uh streaming presentation i must have muted the wrong microphone so apologies for that everybody so let's talk again i'm sorry i had a really good ramble there where i was talking about aligning you know single table design uh with the fundamental concepts of amplify and and and and uh graphql and so you know i'll summarize this discussion again since none of you heard it that's okay nobody will be hearing it twice again when you look at the at the at the amplify you know community and graphql uh you know uh implementations within that community uh there's a lot of debate around you know single table design and whether or not you know we should go with multi-table for single table and there's a lot of advantages to both cases right i mean if i've got an application where i've got a green field environment and i don't know what my application's access patterns are going to be i don't know what it is i'm trying to actually build yet i have some idea but not maybe a fully clear idea of what this model is going to look like then being able to break things up into entities right and being able to kind of create a table per entity is actually not a a bad approach it's a pretty good approach when we when we are you know when we are well understood with the access patterns we're trying to build when we have uh you know user stories that are well defined and i know exactly what it is i'm trying to build here or i have a pretty good circle around these access patterns then when we start to look at the efficiency that we can be get can be gained in a single table design it makes a lot more sense so again pros and cons to both approaches but you know fundamentally the idea of single table design kind of is woven through the concepts of graphql so if we look at you know a graphql implementation what does that really mean let's go ahead and go back to what graphql really is let's you know the bottom line is it's about this type system right we're going to find entities we're going to create queries that are you know uh combinations of these entities to retrieve the bits and pieces that are interesting to our application right and this is exactly what we're trying to do when we build uh efficient applications is work with bits and pieces of data when we start to talk about uh nosql databases a lot of people out there believe that there's a that we should model nosql databases to to retrieve big hierarchies of data right large document structures that are going to give us the you know full view of an entity but the reality is applications don't work with data that way right applications work with data in bits and pieces and and when we start to access data that we don't need and start to retrieve data that we don't need when we make queries then we start to burn a lot of you know iops and a lot of cost at the data layer right so so oftentimes what we start to do as you guys know i'm an advocate of single table design which is about creating hierarchies of items within partitions that represent groups of data that we're going to be interested in and then being able to selectively query the bits and pieces of these hierarchies that are relevant to the application so let's take a look at graphql and really how is graphql align to this idea and so if you look at this this the fundamental concept is a construct within a graphql framework is this idea we have a type system we're going to find entities we're going to define how these entities are related to each other and what are the bits and pieces of data that belong to these entities right we're going to describe these things in a big hierarchy but we're not necessarily going to populate all of the data and store it all together right as a matter of fact graphql is an excellent system to look at as a data layer abstraction right it's really what it is it's it's a way to be able to take your developers and remove them from how the data is stored how the data is accessed and instead let them focus on that higher level application layer logic and this is cons this is really something that i've always advocated in my development teams right you create a data layer abstraction and you know we coded the data layer and then we don't have to worry so much about distributing the knowledge of how things are stored across the entire development team right there's usually some subset of people who really understand that and it's kind of like the same thing with graphql we're going to build these type you know definitions we're gonna build hierarchies of how these things are related and then we're gonna ask for exactly what we need and this again well aligned to the idea of a single table design uh what we have is a um a a mechanism to be able to structure bits and pieces of data across many items associate them to a single partition which represents an entity that entity may be constructed on the index it might be constructed on the table regardless it's a group of items within a certain partition that really represents an entity or collection of entities related to an object and then we're going to query those bits and pieces out of that partition again well aligned here what we're looking for in our query structure is well aligned to the data structure that we're trying to build you know with a single table and then the last thing we want to do and this is fundamentally it right it's get many resources in a single request and you know that's the that's the crux of the efficiency model of a single table design when you make a single query you get multiple types of objects we're going to use those multiple types of objects to populate this hierarchical data structure so i mean fundamentally the graphql constructs are well aligned to single table design so if you have a pretty good idea of what these patterns going to look like then it's going to be a really good idea to put this thing into you know a single table and one of the things that i think is important to realize here is that this is not a new idea right understanding how to use the data how the data is supposed to be structured before we actually start writing code is a pretty common place requirement for most applications and and if you do any kind of search on bad uh database design right all these things that come out and they're all about relational databases right because that's how people model data well what is it poor pre-planning unable to understand the purpose or failure to understand the purpose of the data inadequate normalization you know these are the types of things that are that are fundamental to you know anti-patterns within database and you can doesn't matter who you go you go google bad you know practices for database design it's always the same thing poor design and planning ignoring normalization you know not understanding how the data is being used ignoring the purpose of the data you know i mean we should understand how we're using our data no matter what kind of database we're using and in that particular you know vein then starting to look at things from a single table design is probably you know really should be more considered a best practice which it is in in dynamodb however there are often times when developers don't understand the velocity of their data they don't understand you know how the data is going to be related i mean a really good example you know from my own ex from my own background is a time back in the day was building a system that ended up getting acquired by ibm for a little company called meiosis and you know we're building our own rbac uh you know management system for users and initially we kind of started out with you know hey users are going to be admins are they going to be users so let's just set a flag on the user record right well eventually you got well users have profiles oh well those profiles tend to be com up across users let's define those profiles as roles and and so you kind of see what's happening here you start to expand you know the entities that you're building and and you don't know how these things are actually going to be related when you start and and this was a real you know green field startup right and so then i think when you there's kind of a different class of application right there's this well understood problem that we're trying to solve and then there's this green field area and when we're in that green field area maybe doing things a little bit more simplifying things and simplifying assumptions you know can be can be can be a better way to go right so i mean again i'm not going to come down and say absolutely single table is the way we go all the time every time uh versus multi-table as a matter of fact even in my single table designs with customers as we've discussed over the last couple weeks there's often times where i'm going to recommend multiple tables and you know it's always what's best for you at the time what we're going to talk about today is how do we actually use single table design in aws amplify and we've got a couple mechanisms to do that and a really cool one that was just recently released was direct lambda resolver which allows us to now operate across multiple data sources uh and you know retrieve uh you know data from you know multiple systems uh in complex ways that aren't necessarily supported by the the uh response mapping templates that are provided with the default amplify so nodder's going to show us a lot of that stuff but just so that we kind of kind of go through what customers have done we've got a couple examples of some amplify users that actually gone down the single table path with the existing technologies rich buggy's blog learn to build serverless apps has a really nice uh you know breakdown of how he went about single table implementations and it's interesting because his focus was how do i reduce the complexity of managing multiple tables in production and it was done actually before we had on-demand tables i think if he went down the on-demand path he might make some different choices but rich's approach was to more or less put the entities on the same table uh you know execute query per entity and more or less uh his access patterns mirrored multi-table so he didn't really get a lot of efficiency in the access patterns because again he's querying you know each entity by type and id but he did a man he did satisfy his top level requirement which was merge all the tables together create one table and be able to you know drive some efficiency and how he managed those tables probably the most graceful implementation that i've seen in the single table is to use those custom response mapping templates right and so here's a really good example that maxwell russell did he put a blog post up actually a couple years ago where he described uh his problem which was you know basically these three questions right if i need to search based on value in table a how do i return information for table b with graphql right i mean you know that's going to require multiple queries can require iterating across results if he needs to search a single query based on two fields that are shared across tables why not use a single table you know if table a has a title field and table b has a title field why shouldn't these things be joined together on a single table and is there a way to create return different aspects of a single table in the same graphql queries and so when he went down this he started looking at this thing his model was recipes right recipes have elements elements have ingredients and the schema definition was pretty straightforward right he's got a recipe type and recipes again have elements and you know those elements have ingredients and so he basically created this nice little mapping template template to produce those query results but what was interesting in all of this when he basically when he went down the path of what he's supposed to return right he's returning let's say for this example returning a recipe for breadsticks breadsticks has some elements the garlic butter element has two ingredients it has butter and has garlic now if we were on a single table in dynamodb i could have a recipe id right as the partition key i could have an element id hash ingredient id as the sort key and and that's what that's what you know uh was realized here in this exercise was that a single query from dynamodb can retrieve the results that include um you know the entire set uh you know the the the recipe the top level recipe and all the properties of the recipe uh the all the elements all the ingredients and i can make one query and that maps very well to that graphql construct of you know make one query get many results so you know what he basically built was a response mapping template which iterated through the array of items that was returned from his query right his query was basically saying okay i'm going to look by id for recipe and then i'm going to get a bunch of results and in the response as he mapped the results into the response it returned the structure he was looking for right so in this particular case he's got you know a structure packed with all the elements and the ingredients for those elements and he only had to make one query to the database so again this is a really good example you guys can go through but you know what we're here to do today is talk about an alternative method to this which was just recently released by the team it's a a really neat uh technology by the amplify team it's called direct lambda resolvers uh what i'm going to do is i'm going to go ahead and bring my buddy uh not uh not or dabit on he's going to take us through the implementation where we're going to take the schema from last week if you guys remember that let's go ahead and pop that up real quick you can take a quick look at it this is the schema we had last week this is the online shop schema we have many many types of objects on the table many types of objects within order partitions multiple queries against gsis with different patterns we went through and nodder actually took this model he implemented a direct lambda resolver to support the various access patterns that we had we went over last week including a few additional ones because the patterns we went through last week were all kind of get entity by id whereas the patterns that knotter is going to show us today include you know complex result sets right they can claim contain many entity types so let me go ahead and bring that nodder in real quick there's our and can you go ahead and share your screen nodder you're showing us you're showing up in the big screen hang on one second let me uh absolutely let me go ahead and share my screen there we go we got you so uh again thanks nodder for taking the time to actually go through this and actually implement this uh but certainly thanks for taking the time to come and show us um what's going on here so i want you to quickly just give us an intro introduction and tell us what you're going to talk about yeah absolutely um so i'm uh i'm actually a big fan of rick's i've learned a lot from his uh his talks and stuff so it's really cool to be on this show with him and i've learned a lot over the last couple of weeks actually working with him to kind of implement some of this stuff in graphql really excited to talk about it i'm actually really excited about the technology that we're going to talk about in particular because one of the big limitations in my opinion about appsync has been something that we're now addressing and when i say a limitation it's more a dx pattern that could be improved and that's kind of what we're doing so and that's that's kind of enabled us to now build um you know a little bit more complex more sophisticated things using appsync and um and do so in a more understandable way in a teachable way in a way that is actually going to be you know a lot easier for a lot of developers a lot of customers to start using because it's just a lot easier than it was before so um you know when we're talking about appsync and we're talking about amplify so you know let's kind of split those two apart well amplify is more of a an open source toolkit for um you know really full stack developers that can do a couple of things so the cli is an infrastructure is code provider or provisioner so you could think of the amplify cli as something like cdk or the serverless framework in the sense that you use it to kind of create you know infrastructure as code but a lot of customers are actually using other tools to provision their graphql app sync apis and either way you know whatever way you go about it there are some trade-offs so let's say that you are a cdk developer and what you would like to do would be to kind of provision an app sync api with a bunch of different types and you need everything to work together the challenge in the past was that the mapping template um which is essentially kind of where the operation happens between graphql and your data source uh that operation was written in velocity templing language so if you've ever used something like api gateway you might have seen vtl in the past and it's extremely powerful and you can do a lot with it but the problem is most developers just aren't comfortable with it because we're used to writing python or javascript or java or whatever so um you know what what we've done to now address that is we're going to be uh you know rolling out a you know a set of different features that are going to be giving alternatives to vtl so the first one that we've actually enabled is the direct lambda resolver support so in the past here let me actually share a slide deck that i've kind of put together so i'm going to go ahead and click on new share and it's going to be my entire desktop but um i'm going to go ahead and pop open this slide back here okay so um yeah so this is just going to be a really short intro to kind of appsync and amplify or really just app sync so um this is kind of what appsync you know looks like in a very short you know overview you have a an api that is an app sync graphql api it's kind of an api or a graphql as a service it's a back-end as a service type of offering and um in your absence api you're going to define your schema and that's going to have all the different types and also all of the different operations for updating whatever data sources you have now by default we support a few data sources first class so we support http as a data source so you can kind of map different um you know rest endpoints or microservice endpoints that you already have you can map a graphql request to elasticsearch aurora dynamodb and lambda now to do this you have this one area where you have to write a request and a response mapping template and this is what i was talking about that is dtl or velocity simply language um in in kind of in doing this you're essentially just doing like a dynamodb put operation or query or whatever you're used to doing with dynamodb for example if you're using dynamodb um and that would basically you know make the call to dynamodb and then in the response mapping template that would be the data that would be kind of the response so if you've used like the dynamodb document client for with javascript for example you might say you know do a put item or you might do a query you know that's kind of where this stuff is happening in these mapping templates but we overwhelmingly are hearing that people don't like to to use vtl people that understand it you know they like it a lot because it's very performant and stuff like that but we do have a lot of customers that don't like it so that's essentially what where we are today um so what we've now done and in the future we're actually going to be doing this across all of our data sources but for now lambda now has a direct resolver that maps the request directly into lambda so instead of actually having to have this vtl layer we're just now mapping the event object into the lambda function and in that event object we're going to kind of log that out and look what's there but you'll have everything that you need to kind of make the operation happen uh from lambda so you're going to have the arguments that are passed into the graphql operation you're going to have things like an info object that tell you the selection set that's been asked for you're going to have things like the parent a lot of good metadata is there in that info objects you also have things like the identity of the color so if there's an id token that's been passed in that will be decoded and you'll have that available to you as well so it's really it's a really nice experience and that's kind of what we're going to be you know just kind of focusing on how that works so so matter is it fair enough to say that the lambda direct lambda resolver more or less it just kind of replaces the response mapping template but it just puts it gives the developer a more familiar framework within which to work right they don't necessarily have to learn more than here's the context or the that's being passed in uh is that do we feel that that's a more familiar tool for developers and more or less that's why we're doing this so we've been supporting lambda as a data source with vtl but in the past there was actually a vtl request and response mob response mapping template and yeah into um into lambda um so for instance dynamite b if if we're going to be interacting directly with dynamic dynamodb from appsync we still have that layer there right um what we're going to be doing to replace that is instead of having that um two parts a request and a response in the future we're just going to have a single operation that is written in javascript it's almost you could consider it like a a lambda function that's going to be the resolver okay um so so i can take basically what we're going to show today with the direct lambda resolver i could almost i could convert this one to one to a response mapping template for example if i wanted to go down the vtl route right right exactly yeah everything that we're doing can be done and and those mapping templates that's cool because you know that's one of the things we want to make sure that people understand single table design is not something that's just recently recent to amplify it's actually something that has been always supported it's just required a little bit more knowledge about your access patterns one of the questions we just recently got i just just got on the on the chat line was are there any hard rules about when to use a single table versus multi-table and i think there is really the one rule that i would say is that if you understand your access patterns well then you should go down the single table path because it's going to generate more efficiency over time and it's going to allow you to scale without having to rebuild your data model right and nobody wants to rebuild the data model so if we understand what we're doing then let's go down that path if we don't understand what we're doing and we're not comfortable with you know structuring our data in the single table then the multi-table model gives you an ability to kind of say okay i'm going to just take the entity per table model and at least i know worst case scenario performance will be x cool um that that's that's helpful yeah i'm going to keep an eye on the chat as well and see if there's anything i can jump in and jump in with but um so so i guess let's take a look at the table because you kind of already have shown the access patterns but how would you actually kind of take that and move it into graphql well um essentially what we're going to be kind of doing is looking at the different items or the types really that are available in our database so if we um just do kind of like a scan and just get like the first hundred items and maybe we'll uh organize these by type we'll see that we have like a type of customer uh the type of customer doesn't really have any other metadata that i can tell um the type of invoice you know it has a let's see here an amount and a date and then we have a couple of ways that we're going to be accessing this with a couple of different keys so we have the primary key and the sort key we also have gsi1 and gsi2 but what we would like to do would be like okay how do we kind of map this into a graphql api well because graphql like you mentioned is strongly typed or it's a typed you know has a type system we would basically create an invoice type and we would say okay what do we need from this invoice what we what would we potentially need and then we would just create those fields so if we go over to our appsync api and we look at the type of order you can see that i've defined a couple of fields here so we have like an id we have an amount and we have a date and we can add additional types here whatever we would like to return you know can just be some type of field on this type and then all we would need to do would be to kind of say okay now that we have an understanding of the data in our in our database we have the actual type defined in the types of data that we would like to have for that type we need to now define a query to query for this data and if it was a mutate if it was a create operation we would create a mutation but we're only focusing on queries today so we would say okay we have this type of order now how are we going to access this order um and then we would look at you know our database again and kind of understand those access patterns we would say okay we might want to operate on gsi1 we might want to operate on gsi2 we might want to be using the primary key and the sort key and then we would create essentially the same names of the access patterns that you talked about earlier we just kind of define those here and then we're going to now after this kind of walk through how this actually happens you know through the absync api and lambda and dynamodb so um yeah how does that look do you have any questions rick no i mean it looks pretty straightforward so these map one to one to the access patterns that we had in the powerpoint last week it looks like there's a couple of new patterns down there at the bottom because i think as we discussed earlier in the week that we didn't have any patterns in that list that we're kind of beyond get entity by id right so we've got a couple patterns in here they're going to actually retrieve you know entities by date range uh they're going to give us you know order history or customer history and so those are going to actually be collections of objects that'll be the interesting aspect here because a lot of these get entity by id heck it doesn't really matter we go across multiple tables right since it's just kind of get empty by id there's no real efficiency there but those other ones those would be the real interesting ones so a good set of access patterns looks like awesome yeah absolutely i mean you can uh and with with uh with these graphql operations pass in arguments uh you can return different types of data and um all you need to do is kind of define how that looks by by the types so for instance uh get custom rob by id we're just going to be returning a single customer so this is going to be coming back in the form of an object um get inventory by warehouse i'm sorry get inventory by product id right here that's going to be returning an array and the array is going to be of type warehouse item so we would just kind of define it like this you could also return a string you could return an integer anything like that so the type doesn't have to map to an actual um type that we've declared it could be some you know data type like a string and determine and these types could actually be complex types that contain you know properties that are actually other types and so if we're querying the table and that all those types are present then we should be able to just kind of assemble those complex objects and hand them back to the application using these direct lambda resolvers right and and to do something like that you would probably just create some type that that mapped to all of that data and then you might have a type field on that on that type you know yeah yeah exactly a field on that type or property of that type that contains you know either one or more or list of other types exactly so it's really again as we kind of showed in the example i showed previously you're going to get an array of items back from the query and those items might be homogenous or some items might be heterogeneous but you know at some point you have to map those items into the construct that you queried for and and that construct might be a base entity that construct might be a collection of entities for example in order history which might contain not just you know uh the the products that were ordered but it could contain the invoices and shipments of the order a variety of of things that might happen here so you know these entities that we define in graphql don't necessarily correlate one to one to the actual application layer you know entities or the data model entities and a graphql entity and this is something that i think was key understanding for me is that a graphql entity really defines uh you know it could define a collection of entities right a hierarchy of relationships it's not it's not just a top level item right so and when i query for when i send that down that definition that graphql definition or query down to the system that's what we can do we can actually look at the patterns or the object that the hierarchy that's being requested and map that directly to a query that's going to efficiently retrieve those results so um you know how would we actually you know get this information into lambda so let's now take a look what we what we basically are going to have right here on the right are the resolvers so you can resolve um any query or any mutation into a resolver but you can also resolve a type field into a resolver so for instance if i wanted to resolve this id from an invoice i could actually create a resolver and that resolver would be a lambda function or some you know dynamodb operation or something like that but we're not really worried about resolving any of the fields because we're going to be returning the fields from dynamodb correct via lambda but what we are going to be resolving are all these queries so if i scroll down and i look at um the query type i'll see that we have all these different fields on the query and from here we can just decide you know where we want these operations to go so if i click on one of the resolvers um we see that we have kind of um you know this new view that we're checking out but i'm going to go ahead and delete this resolver and just show you how this works so we're going to go back now and we have all these resolvers defined i'm sorry we have all these queries over here defined and now i want to map this into that function so how would we do that i would just scroll i would scroll down here and i would find the function that i would like to attach so for instance we have git customer by id i want to map this to a lambda function so all it needs to do would be to click attach and then here we can um view all of the data sources that we've created and then now i'm going to switch over into one more panel and kind of show you how we created that data source um there's a data sources view and apps i think and all you need to do is click create data source and then you would choose the lambda function that you would like to be created as the data source and then you would be done um it would then have the arn connected and the permissions set for appsync to actually access this function so i've created this one data source and all we're doing is just mapping you know just a regular lambda function and uh and this is going to be the one that we're doing and that's it now we can just click save resolver in the past though we would have these requests and response mapping templates and since we're since we no longer need that those are kind of you know the old way of doing things i can just click save resolver and we're ready to go and i go back to my schema scroll down and i kind of want to match that up again i see that now that uh that query has a resolver attached and we're ready to take a look at our function so let's go ahead and take a look at the lambda um what you have going on here is in the uh handler you have that event and the event object is going to be you know having some metadata that is specific to app sync into graphql so let's go ahead and log let's go ahead and log that out by performing a query before we kind of walk through the rest of this just to kind of give you an idea of what's going on there so to log that out i'm going to go ahead into appsync and i'm going to go into uh we have a queries editor where we can actually perform queries live onto you know into our into our data source so i'm just going to perform my git history get order history about id um we're not going to walk through that just yet we just want to log out the event so i'm going to go to cloudwatch and we'll take a look at the event okay so here's the event um we have a couple of different properties that we're going to be focusing on so arguments if we look at the operation here we have an order id and you'll see that we now have that order id coming through here in the arguments so we can we can get that off of the the event um if we were signed in and we had an id token this identity would be populated with an object we have our request headers we don't really uh need those for this example so these are like test events not or is that what this is what we're looking at right here the user would put this together to kind of test the api yeah you could think of it that way but but the the event data itself is going to look the exact same exact same way right it's just going so some of these things will be will be populated when you i see okay yeah um and then the info object is is pretty pretty useful this has a lot of graphql type of information so the selection set list is the the selection set data that we asked to have back so in the query we said we want the amount we want the date and the id so that comes through in the event um so if you ever need that for any reason it's there so now real quick here so in this particular example of get order history we're simply getting the top level order item but the order history on the table contains you know invoices and shipments and you know order items and so theoretically we could simply just extend that request so to speak right now all we're asking for is the amount the date and the id but i could have an array of invoices i could have an array of shipments and so then and what's really cool about that is that you know once i add those things into this query then the result set that's going to come back when i query dynamodb by order id is going to include all those items already right so all we need to really do then is push those things into the object using the lambda code that you just showed us right so we go back to that code real quick and take a look at so the code that we got right now kind of doesn't really process multiple items but there's no reason why the query definition and the result set couldn't handle it right because if they think about the dynamodb structure on the table we've got a lot more items in that order partition than simply the top level order item and we could return those right absolutely yeah you can work with any any type of of data you know that you would that you could think of that i can think of yeah yeah so um and then and then we have the parent type name so this was a query and then we have the field name and the field name is really kind of what we're going to be focusing on because we want to be able to understand as quickly as possible so we can then take the operation and kind of map it into some type of dynamodb operation so now that is parsed by your lambda process to determine how to query the dynamic table is that right so you have basically one lambda function that can execute all these various access patterns and that's what you're when you say field name get order history by id we're checking that in the lambda function we're making basically a switch on the field name saying hey if it's get order history by id then i want to query the table with the order id as the partition key and i'm going to get those results uh what we're looking for is the amount date and id in this particular query so i could actually define a specific so i don't necessarily have to get all the items out out of that partition right in this example since i'm only looking for the amount the date and the id i could define my get order history by id to say get query the or by order id uh where um what i don't know which item holds that is that the item that i sorted on customer id what is the query that we're running when we go get order history by id so the query on dynamodb yeah so let's take a look at that yeah so um so to to to map that into a function and and then you know uh into dynamodb um we're basically gonna take the field name off of the event so i'm basically going to let me go ahead and remove this console.log for now just to kind of make a little more room um so i'm destructuring the field name off of the event so this is that that of field name we just looked at and then we're just doing like a switch statement and we're gonna say okay based on the field name we're gonna then perform this operation now of course if you are um the type of person that wants to have a separate function for each graphql operation you could do that right but i think for this example it made a lot of sense to kind of put them all together so we can view everything at once sure um so let's say the so the operation that we did was get order history by id so let's go ahead and take a look at that i have a function called get order history by id right here and i can then just go to the function and open that up and all we're doing here is really just a really basic um operation on dynamodb using the dynamodb document client so we have our params defined here we have um you know we're setting the primary key and the sort key we're then doing a filter expression on the order um there's only a couple of items that come back for this operation so it's not really filtering that much and then this is kind of where we're trying to make sure that the the data that we want looks like the data that we've defined in our graphql api right and we're only making one small modification um and this this was only done because i like to have an id for every item in in my graphql schema that is like a unique you know id but um we could have also used the primary key as the id or the sort key or some combination of the two sure but um yeah so so basically this would be your logic that loads the the the response basically at this point you're going through this thing and you're saying okay here's the well actually is this where where is it that we're actually generating the response to the for the request so anything that that's returned from the function is the response okay so now if i had a hierarchical data structure that would be maybe a complex instead right now we've got a simple you know okay here's the eye we found the item we wanted in the order partition but you know maybe there's going to be a complex response that's going to include invoices and shipments so i'm going to at that point would i return an array would i to return a map how would that data go back up to the to the client to be processed um you could return uh either one it just i guess depends on how you defined it in your schema i see um right okay so then on top of this now does this replace the response mapping template does the lambda right exactly yes okay so so there's a response object that would be repre that would that would contain the data structure that was defined in the in the graphql query and that's what we would populate in this particular exactly whatever data structure you return uh is the return type and there's nothing else like they're like in before in that in the vtl and the response mapping template there was some other stuff going on it made it kind of uh tough in my opinion yeah kind of understand what was going on but since we're just working with javascript or python or whatever in a lambda it's just like you're working with an api call from anything else you know sure okay so we got another question are there plans to make amplified directives work better with single table design they seem like they're forced to they seem to force an entity model um yeah yeah absolutely it is something that we're um it is on our um you know on the radar it's on the radar backlog at some point but it's not going to be something that's going to happen like like in the next few months but it will be something that that is going to happen and we're already actively investigating and working on it yeah well and again i think what we're kind of demonstrating right now is that the tools and technologies already exist right they existed in vtl they existed in direct lambda resolvers uh you know the entity model is not something that goes away i mean yes i believe that amplify absolutely does force an entity model but that's actually one of the primary fundamental tenets of data modeling and it's not something and if you listen to any of my sessions that's what do i talk about right we can't even start to talk about the data model until we understand the entity relationship diagram right so i i absolutely agree with this construct of we're gonna we have a type system we're gonna define our types we're gonna define the hierarchy of those types and then we're gonna query those hierarchies that's a that's extremely well aligned you know philosophy that graphql has with the whole single table design with normalized design in general even across relational databases right and and again when you start to look at you know graphql in a relational context i mean i could certainly go and query each individual entity table in a relational database and do that or i could you know define a complex query and map that to a graphql template and execute a parsing response mapping template or a direct lambda resolver so that i can execute a single query against the relational database to retrieve all the results that i need to hydrate my you know graphql or my graphql response right as a matter of fact we're going to go through the exact same process of determining when does it make sense if you almost look at it like you know in the relational context when do i map a uh uh a graphql query to a complex sql you know join statement uh or versus when do i query a table per entity in the relational model right i mean this is this is really what we're getting down to underneath the covers it's like this is the this is the crux of your data layer api right get order history by id that is exactly and so underneath what happens with get order history by id uh you know that's that's irrelevant to the developer all they know is that i'm going to get this information back when i make that that that request so giving the cut the developer a a you know decorative construct to define what it is that they're looking for from the data layer is really what graphql is about and mapping those queries underneath you know that's how we drive efficiency in the system it doesn't matter what database you're using whether it's relational database or nosql database so i i i agree that we need to make it easier for developers to embrace a single table design but i i seem i i totally think that this that the entity model is a foundation of any application and amplify apps uh you know are actually enforce a best practice by doing that yeah we're um we definitely um get a lot of a lot of great customer feedback and we're always working to prioritize stuff so um yeah anything else that anyone has that they'd like to share feel free to reach out to me on twitter at dabit3 or just go to our github repos if it has anything to do with open source projects and you know open up a conversation there yeah i mean i love this tooling i love the i love the you know honestly like you said we've talked about the tooling was there was in the vtl constructs which maybe was a little bit harder for developers to kind of get their heads around but it was always there the direct lambda resolver is really just a way for them to kind of get into an environment that they're very comfortable with right i get some context in i can process the elements that are stored inside of that context and and execute and return a response and they're just a little i think developers just a little more comfortable with this it'll be interesting i think we talked a little earlier about potentially doing a bake off right you know a shootout between multi-table single table using you know direct lambda a single table using vtl mappers yeah i mean we're going to do this in a couple weeks we're going to put some code together we're going to come back online and we're going to say okay let's run some high velocity workloads and actually show you the efficiency i would expect to see that the direct lander resolver and the vtl response mapper approaches are going to be very very close but we're going to see a significant advantage for both of those in a performance context over multi-table and i think this will really be i think what developers need to take home from this it's like and it's a lot of it has to do with how we kind of advertise all kinds of features across aws it's like you know we do things to make things easy for customers and oftentimes the easy button is all we need right it's never going to get painful using multi-table for 90 of the applications people are building multi-table is not going to feel any different than single table right for that 10 of apps that are scaling beyond you know what we would think of as like a single instance application or something like that things that are starting to get big that's when we see the benefits of doing these things right and so it's up to the customers right i mean sometimes an application you know is never going to get bigger than x right so just go ahead take the default resolvers use entity per table you know for table models it's just never going to be anything that's going to bother you and with on-demand pricing the complexity of that is is and the benefits the cost benefits are negligible the complexity difference you know is negligible so it doesn't really matter and one of the things about the entity per table model is it sure makes things easier for the service to make those assumptions right i mean it's not easy for the service to make you know assumptions about complex data models but it certainly is easy for us to build a generic service that can address the needs of you know a a a basic entity model and that's really what multi-table does right so you know you know i i can't say that one is you know has i mean we can always say one has advantages over the other but you know when do i use one versus the other i i don't know i mean a lot of this is up to the developer i think that um also with the direct lambda resolver support the mental model for people that have picked up something like api gateway in the past and they've been able to use the proxy to kind of proxy the event directly into a lambda function and then have access to over 48 aws services directly from lambda really easily oh yes it's really easy to kind of take that mental model and start applying it to graphql but here you're actually able to look at your schema and know everything that's going on in your application because it's almost like a menu you have all the different types that are available all the queries all the mutations the mutations work exactly the way we're looking at now so if you actually wanted to create a crud app or if you wanted to actually have a real time application based on all of these types so you could think of if someone created a new customer by id and you needed to subscribe to that event all that's actually already built into the service all you need to do would be to create a subscription definition here and the actual subscription real-time implementation has already been baked into the service so it automatically is going to work anything that you return from a mutation um in in your lambda function uh you know for the examples that we're looking at for example anything returned from that function gets sent as that that subscription in real time so it's the service has so much additional uh power and benefits to it um i think making this so much easier is is it's extremely exciting i i really personally i'm a big fan of graphql i really am excited about the stuff that i'm working on but now that we have this i can't really think of a lot of use cases where i wouldn't use this as my api layer um rest is uh is extremely you know still powerful and if you like rest of course continue using that but if you're into graphql this is kind of a really really nice deal well i i i'm becoming much more of a graphql fan honestly after this last week or so or two you know diving in uh you know a lot of the things that i looked at initially was kind of whoa you know because again i kind of got struck by this entity per table model you know executing complex you know patterns of multiple queries to retrieve you know with data that could be retrieved in a single result set but now understanding you know vtl mappers understanding direct lambda resolvers and how we can actually build this efficiency into the system it's just another application development framework right and and just like anything like so many of our systems it's like you know we make assumptions out of the box that aren't necessarily optimized for you they're optimized for they're not even optimized necessarily they're just generics right generic assumptions and this is the entity per table model is the generic assumption that we know okay look a new user comes in doesn't know what they want they just want an app that does x hey we can give him an app that does x you hit this button right is this going to be the end-all be-all of data models that you want to you know run your application at scale with no no and it's certainly not an exercise that's unique to you know this service right i mean when how many applications have we built over time that have required us to refactor the data model i i mean it's a common part of the application development experience right i mean we're going to be the in the worst in the best case you did a beautiful job you know architecting this you know sweet data model and it works perfectly and now i need to separate it into multiple api services okay what am i doing i'm you know i'm vertically partitioning tables i'm truncating you know things out of this catalog and creating and migrating data into new catalogs and creating new apis on top to to to scale my service right and so you know refactoring the data layer is not something that you know developers should be completely afraid of it's something that we should you know absolutely understand that as my application grows i might i might outgrow the entity per table model right it might become painful from a cost performance perspective to use that model at which point hey great let's refactor this thing and what's nice about amplifying appsync that it does for you it gives you that abstraction layer that makes it easy to do that right i mean i could say hey these queries are costing me i'm going to migrate this data into you know single table and and i can do this in stages i don't necessarily have to bite the whole bullet all at once i can as we demonstrated you know individual queries mapped to individual individual operations mapped to individual queries through the abstraction layer of of amplify which gives the developer the ability to throw a switch at any time right and say i'm going to use it from this source or that source right so you know that's one of the things that we as an enterprise at amazon struggled with and actually came up with solutions for was you know how do we deprecate one api in favor of another uh you know and do it in stages right as we started to migrate across from our services from relational databases to nosql um you know and and so the giving customers tools like this right where i can actually you know define these constructs and throw switches to make that happen you know it's actually really really nice and it's a really valuable feature for developers so you know you know i don't know i mean i i actually think it's a very powerful platform and going forward it's something i'm going to look more for um and someone asked for the code so i'm going to actually uh go ahead and link to this gist i'm going to i'm going to have to update this a little bit because it's not completely up to date but it does have everything that i use for this api it has the schema it has the queries it has the functions and all that i just maybe need to update a couple of things but i just pasted the link in the chat sure so we had another question about multiple resolvers in parallel limits and throttling become a concern as always uh and with multiple resolvers in parallel i'm not all that worried about it because often times those resolvers might be firing in parallel but they're going across different partitions right now get me this order id that order id get me these things from that get me these things from the gsi versus the table so you know access patterns are always something you need to kind of consider when you start talking about velocity of data that's when throttling becomes an issue so this is something that you know we're just going to have to understand up front i wouldn't say there's any more concern with throttling in a single table than there is in multi-table right i mean i've got the same issues 1000 wcus 3000 rcus and if i'm hitting them too hard i'm going to throttle so this is just something that as a as a developer we need to understand and address as far as how do you help with throttling dynamodb has a really really good tool now it's called cloudwatch contributor insights if you enable that on your table you're going to be able to see exactly which keys are hot not just which partitions but which individual logical key values are being requested and how much of your you know write capacity and read capacity they're consuming so you can identify you know which areas might need to be addressed inside of your implementation or find your hot users and so that's probably the best tool for dealing with that but again it's it's a it's just something that's always a concern so no matter what application you're building so cool just kind of going through a few more of these queries you know it's all it's it's pretty much the the the the values that we're getting back are of course different but the fundamental idea is the exact same across all of these we're taking that uh that argument we're mapping it into the the event and lambda we're then taking that event and we're going to kind of take a look at the event we're going to see the arguments we're going to see the operation we're going to decide at that point which function we're going to hit and then whatever uh we get back from that that that return value is going to populate these fields here that's it i mean it's honestly it's actually pretty simple if you've ever worked with dynamodb from lambda in the past um getting this to work is is not hard at all and someone mentioned the authorization um so yeah appsync actually has a bunch of different types of authorization modes right now i'm using more of like a public access which is an uh an api key meaning you can hit this api from any client application as long as you send an api key in the header you're good to go but you might want to do more um secure forms of access so we also support amazon cognita we have we support oidc and we support iam so if you'd like to hit this api from another aws service we support that as well so all that stuff is baked into the service and again that identity will be available in that event depending on how you hit the api very cool all right so there we are we're running up to the end of the hour here uh great session really appreciate you coming on board here for this nodder we're going to have you back in a couple weeks because we're going to do that shootout i want to i want to i want and i actually want to work these access patterns a little bit more i'd rather show some folks that have some you know some real complexity here how are we hydrating these complex objects that have you know relationships to arrays or lists or of other objects or properties that exist that are other objects right so we'll we'll show that but this is again folks please put your hands in this i think this is a really good framework for establishing a a very well-defined data layer api it's it's uh and that which is language agnostic agnostic right that's what graphql gives you it gives you that abstraction your developers don't need to understand how the data is being hydrated or where the data is stored or how it's being queried all they need to know is what do they need to ask for and in order to do that what do they need to find what types of objects they have right what are the properties of those objects okay this is what i want in my query statement and i don't have to understand sql i don't have to understand document api and dynamodb or anything like that that's for your data layer api engineers and that's really what amplify gives us is this kind of well-managed well-abstracted data layer api framework and i really really like it so again thanks so much nodder for coming on board i look forward to seeing you in a couple weeks when we do the shootout and if anyone else has any questions fire away i don't see anything up there give the moderators a minute to let us know if anything else is happening yeah absolutely i appreciate coming on um someone mentioned some cognito stuff it would be cool to possibly do some fine grain access control and authorization stuff in the future that's something to consider i think that would be pretty cool cool all right then that's it uh thanks folks again for the time and we'll see you again next week i don't know if i have a subject yet but we'll come up with something to let you know pretty soon and again thanks nodder for your time and your effort on this and i look forward to seeing you again in a couple weeks talk to you later bye [Music]

Channel: Serverless Land

Views: 6,255

Keywords: serverless, dynamodb, aws, nosql, database

Id: EOQqi6Yun7g

Length: 55min 6sec (3306 seconds)

Published: Tue Sep 15 2020