7 Common DynamoDB Patterns for Modeling and Building an App with Alex De Brie

hello world and welcome to another episode of what this week we are going to talk about dynamo and for that i have an amazing guest alex dibri he is the author of the book the dynamodb book and we are going to work together through an instagram clone application and apply the best practices of dynamodb so let's get to the video hello and welcome to another episode today i have a super special guest he is one of the dynamo db gurus uh he's an aws that a hero and the author of the most popular dynamo book out there i'm pretty sure you all know who he is i have with me alex debris so welcome alex how are you doing thank you thanks for having me i'm doing well great to be here excited excited to make my debut on on your channel been watching your stuff for a while so yeah i'm super excited to have you here because dynamo is one of my favorite services so i have rick last year and now i'm bringing you here to speak next time i should make a debate yeah exactly yeah yeah so what do we have been doing lately yeah i've just been doing um you know talks and promotion and things for for the book and then also consulting um you know with different clients either dynamodb specifically or or more aws and and serverless type stuff is sort of anything in that realm been working with clients that way so uh yeah it's been it's been fun getting exposure to a lot of different problems working with a lot of different clients and doing some teaching and all that so yeah yeah that's a nice thing of doing your own or your own thing you can find your own way and yeah exactly yeah but before being your own boss you also work for the serverless framework for serverless inc and and i was looking that you work in these particular years where basically everything was kind of very busy and and hype about serverless from 2017 to 19 or something like that you were there how it felt to work in a company that was basically in the middle of the serverless explosion because by then it was the biggest framework yeah um it was just a lot of fun um you know i loved working for a company where like i was a user myself it felt like i was building for me but i was also helping build for for all these other developers it was just cool to see like the way you could enable other people and the cool things they're building on top of that so just like seeing all that was super interesting and then also just seeing the community grow i mean that's how i sort of met and and found you and folks like jeremy daly and all the great you know da's at aws and it was just like a really fun time to to see the community grow and like it was so new so we were still figuring out all the patterns and and yeah it was just uh it was a really fun time for sure that's the the most interesting thing because basically at least when i started with serverless into 2016 nobody really knew what you could do i don't think even aws have a clue what like what was the power of this uh and everybody was like coming with new ideas and new ways of using it so i imagine as a service provider a framework provider it was kind of interesting to catch up with all the yeah absolutely yeah so yeah so much energy going on at that time so it was a lot of fun yeah so today what we are going to talk if you have not imagined we are going to be talking about dynamo [Laughter] aren't we yes and i think we are going to take it in a different way like it's not going to be a class on dynamo i want to take it to on good practices and try to pick your brain on on how to do things and and you suggested when we were preparing for this to do and build an application and and go from there and i think that's something the audience will love because they love to see code so so maybe we can start from there why why dynamo yeah why dynamo i think um you know you see a lot of folks especially in the in the service world in the last you know uh three or four years picking up dynamo and choosing dynamo for a number of different reasons and and i think probably the biggest one that was the people were choosing was was the connection model where people were trying to use relational database but when you have this this hyper-ephemeral compute with with lambda and you have lambda spinning up all over the place you know the vpc cold start it just like you know in connection limits to relational databases it just uh it was problematic to use relational databases in lambda so then you saw people reach for dynamo because of the way it has that http connection model and doesn't have connection limits and networking that's why we all started it truly was and then you know for a long time you know i myself and a lot of people were just like using dynamo poorly and like getting frustrated with it and things like that and then um you know i found rick huahan's stuff and and went deeper your mind exploded exactly yeah and then i just slowly piece it back together all the pieces i found all over my my room but uh yeah you know rick uh doing an awesome job teaching teaching the world how to dynamo and i think it's just been fun to see the community grow and and things like that so yeah but i think that's that's a good thing to say that we were doing dynamo wrong because i remember when i got started with dynamo i think i rewrited my databases like 20 times i was putting we were doing a migration from basically a postgres database to dynamo because we were moving to serverless and it was so hard to find how to put that data fit in there and how to use the access patterns i i was such a such a pain when you don't know what you were doing because i never got to study dynamo by that time i was like i can throw everything into it it's true and i think yeah just the content out there on nosql you know it used to be just kind of so bad and just yeah um i think it's gotten a lot better over the last uh couple years and rick and the aws team have been a huge part of that so yeah and i think all the community as well because like because several less people started using it more and more and it's not just the big organizations that were using this nosql for their things it was just also normal companies normal sizes that just wanted to take advantage of it and then they were like oh maybe we need to write better documents and more blog posts and all these type of things so yeah cool yeah absolutely it just sort of exploded from there so yeah it's been it's been fun to watch so where do you want to start with this application yeah so first of all i want to start off with just a little bit of background on dynamo and i'll share some slides for this i'm going to keep it fun keep it visual hopefully and two things i want to show i want to talk about number one is is a little more concepts around dynamo especially around why choosing dynamo you know we talked about the serverless aspect but there's another aspect around consistency of performance that i like to talk about that's interesting and then we'll just talk about some basic concepts in dynamo some of the terminology and and what this single table design stuff is just so we you know lay a little foundation because some of the stuff we'll do later i don't want you to to feel like it's way over your head so this is like the the crash course 100 200 level stuff and then we'll we'll go to the higher level stuff so uh starting off here with the slides uh one thing i like to show people is uh just kind of compare relational databases to dynamo in in terms of consistency of performance i like to start off with this little chart and on the x-axis there i have data size right and as you go further out on that x axis you have more and more data in your application so this is years down the road as your as your data grows you you keep moving to the right and then on the left side you have performance and you can think about this as as sort of latency right so lower is better you have blazing fast and then fast sluggish and and up to painful you know and if you're working with a relational database you might see a curve sort of like this right where in your test environment or the first day you launch you have a little bit of data everything fits in memory it's super fast you're you're working really well you're very happy with that but then you know six months a year two years down the road you get more data you get more users it starts to get sluggish users are complaining you have to do some investigation do i need to denormalize do i need to add more indexes do i need to increase my instance size stuff like that and then at some point you know it might get so sluggish and and beyond sluggish even to painful that you need to re-architect your application so you sort of see a performance curve like that often with relational databases and then in contrast with dynamodb you're going to see a performance curve more like this right where it's completely flat and two things i want to call out there number one on the far left side of that that curve you know a relational database might be faster at that very very low end of data but dynamo is generally fast enough that your users aren't going to really notice it so it's not a huge deal there i think the bigger difference is just how that it's going to be consistently fast throughout so like in your test environment or on day one it's going to perform the same as down the road you know five years ago and i think that that's super important because we are all building global applications and we are all collecting all the data in the world and like exactly it's nice to know that you know you can like develop this thing and then just sort of set it on autopilot forever and you don't have this long-term drag of like going back to that application you know figuring out what's going on performance debugging things like that you can sort of move on to new things so uh that's what i really love about dynamo i think that that aspect is over or underrated in terms of that consistency of performance so i think that's a big thing for users so that's another reason why to choose dynamo uh and then just moving on to some concepts around dynamo here's some data that you might see in dynamo this is all modeled with nosql workbench which you can pick up this is an aws tool basically i've got some some users in my table here in terms of terminology right if i look at an individual record there i've outlined it in red that's going to be an item so this one happens to be myself alex debris you can also see we got you marcia we've got rick huahan in there show a couple different users in our table so that's one item there i want to split that item into two parts you have the primary key there over on the left and that primary key is going to be declared on your table every single item in your table needs to have that primary key and that's how you're going to access your data so in this case you're going to access your data by the organization name that they work for and the username something like that and they need to be unique this combination and i think that sometimes is confusing good point so you need to uniquely identify each item with that primary key so that primary key in dynmo super important make sure you kind of understand the concepts there in addition to that primary key you also have these other attributes there on the right those are things like name email i put favorite aws region whatever you want to put there those are things you're not going to query by they're just sort of going to come along for the ride when you fetch that that user so in this case you know we'll fetch the user by their username and who they work for but but not by their first name or their email or their favorite region things like that so those other attributes you just think of those as other things those aren't indexed necessarily with dynamo so that's a basic example i also want to move into or talk a little bit about what we call single table design because that that really blows people's minds so um you know this has just users and a table but um what we're going to be building later today is sort of like an instagram clone right and we have users and we have photos and we have likes on photos we have comments on photos we have followers things like that so i'll just show a quick example of part of what that table looked like so here is a table with some single table design it looks a little more confusing than our last one but if you look at this you can see i have two user items in this table and i have a type attribute that's identifying hey this is a user item so again i have i have both of us i have myself and i have you marcia in addition to that i have a couple photo items those are outlined in blue there you can see the type is photo for those ones so they're all in the same table there so but if they're all in the same table it's kind of confusing how we use the primary key for that sort of thing my question is why you will put everything in one table yeah it's a it's a good point um a couple different reasons number one there are no joins in dynamodb so if you ever want to get sort of different types of entities in a single request like you would in a join operation or relational database you'll need to put those in the same table and in the same what's called item collection which has the same partition key and you can get those very efficiently in a single request to handle that sort of thing so that's one reason to put them together another reason um especially before dynamodb had on demand mode is it just makes it sort of easier to maintain operations around that table where you only need to sort of configure provision capacity for one table not for all your different separate tables you can just sort of do one and handle it that way only need to monitor alerts and alarms on one table as well now that dynamo has on-demand mode if you're using on-demand that's not quite as important um you can have these different tables on-demand mode it's still pretty easy to handle those um the the last thing i like about single table is it really forces you to think in dynamodb terms i think and it makes you think access patterns first it makes you it makes you think about these sort of generic primary keys i'm going to talk about in just a sec to where you're like okay i split my item into these two parts the primary key which is how i access my data that's those are my indexing attributes in dynamo and then i have my actual application attributes you know like the username name things like that that we talked about that i actually use in my application and you think of those separately one is for dynamo only and one is for your application and and it just helps you think about that data in a more generic way because if you see in this example the username you have it in your partition key and you again have it as an attribute so it's very clear that one is for dynamo and one is for your application yeah so it's not that you will get your partition key and then you will parse it in your application to extract the username no they're independent views okay so yep absolutely and that's that's a recommendation i do to have those separately some people like to parse out their partition key but i'm just like hey keep all that stuff separate think of this as one way and think of that the other way and um but it's up to you a couple other points i want to call out here so if we look at that primary key you know i used to have organization name and username for the partition key and my sort key but if you look now i have these very generic names for my primary key so the partition key element is just called pk that stands for partition key the sort key element is sk that stands for short key so again because i have these different items that aren't going to share the same attributes i'm going to use these generic attributes there we're not going to go deep into what partition key and sort key are but that's how that generic attributes work and then if you look at the values for these you know look at the value for that partition key it for the user item it's it starts with user so capitalize user hash and then the the username in there and that's gonna be the same pattern for for both sort of users that i'm having there but then if you look at you know the the photo item i have a different one i have you photo for user photo and then hash and then the the username which is gonna be that owning user that's gonna allow me to get all the photos for a particular user uh very easily so this is just like some very basic stuff um you don't have to understand all this right now we'll talk a little bit more about some of this later on and also just check out you know other material or the aws docs but that's a little bit of how sort of single table design works with dynamo and some of the concepts we'll see and then uh yeah we can move on to actually working with our application yeah so next i want to talk about the um how you go about modeling with dynamodb and sort of what the the uh patterns are to do this and and um i have some code on github you can check this out it's gonna be shared as part of this but yes um for basically what we're building is is this instagram clone right so here i have um our code in in github that you can look at this dynamodb instagram and when you're modeling your data with dynamo the first thing you need to do and this is true whether you have dynamo or something else is you really need to understand your application and understand what you're building right so i always recommend starting with some sort of design document saying what you're building and sort of walking through the different steps but one of the big things i always like to have in a design document is what i call terms and concepts and it just says what are you building and sort of what are the different terms and things we're talking about here so i have a very basic and sort of simple one here since instagram is what a lot of people know but you might have more details here but you know i'm saying here i'm saying hey a user is is something that is someone that signed up for our application they're uniquely identified by that username so you have that sort of constraint in there that unique identity talks about a photo right it's an image uploaded by a user photos can be liked or commented that we'll talk about below so we have likes as well you know user can like a specific photo a constraint that we have here is you know a specific photo can only be liked once by a particular user so you don't want you know a user be able to like a photo 15 or 20 times just so you get a bunch of likes on it things like that but then you have comments right where there's no limit to the number of comments on a photo by a given user so likes and comments similar but different so i think just like illustrate all this different stuff what a follow is you know a follow is not bidirectional it's a one-relational a one-way relationship so just think about you know just list some of those things out and really understand your application and what you're building and get that on the same page then someone reading the design doc says okay i can understand this stuff and and if you have a constraint in your terms that isn't reflected in your application it's easy for them to comment and say hey how are you how are you handling some of that stuff so that's what i always i think that's a very good point and i think we all have started to develop applications without really like having a clue in our head what we were doing but really not thinking about the real access patterns i think at least for me a common example in the past is that i always try to implement that search and when you realize it most of applications don't have search in all the things they might have searched for one parameter like you need to find something but the rest of the attributes and that's very hard to model in in dynamo for example full search in all the whole things like you can do with a squall uh so that's i was my common mistake at the beginning like oh i i cannot model this thing i need search but then i didn't understand my access better so yeah absolutely especially for like complex things like that like search is just a hard thing to do you know definitely in dynamo but you know anywhere you're doing it's a hard thing and a hard thing to scale so understanding your constraints around search or your constraints around all these different things i think is really going to help your your modeling and how you go about that so after i sort of have that and understand my turns and concepts the next thing i recommend doing is is thinking about your access patterns and in dynamo you definitely want to think access patterns first so you know once you've sort of listed out those turns and concepts you go into access patterns and i recommend listing out your access patterns in like a google sheet or something like this where you're very detailed and actually list out every one you have so you can see here you know i have basic user and photo crud so create and read and update delete for around user photos but then things like listing photos for a particular user liking a photo listing likes for a photo comments following a user listing followers of users all those different things list out every single access pattern you have and then what you'll do is you'll sort of go through and design your table to handle these different access patterns you'll design the primary keys and any secondary indexes you want to handle these access patterns so this is what the blank version of it looks like and then the filled out version is gonna be something like this you know if i want to get photos for a particular user i know that i want to go to my name main table i'm gonna need my username parameter and i'm gonna run the query operation or if i wanna like a photo right i'm again using that main table i need these three different parameters and i'm running a transaction and we'll look at this transaction in a little bit it has two different operations right where we're incrementing this light counter on a photo but we're also creating that like and ensuring uniqueness right because we don't want a single user to like a photo more than one time so all these different constraints you have these access patterns listed out and once you have this uh listed out pretty in a nice detailed way then it's very easy for you to to go out and actually implement this in your code and actually most of the work you're going to do with dynamo is going to be in charts like these rather than in your code the code part of it is pretty uh straightforward once you get that another thing you're going to have just as you work through this is what i call an entity chart so you know i had those those five different terms and concepts i had user photo like comment follower list each of those entities and as you're working through your your access patterns and designing it you're gonna say hey what's the pk pattern what's the sk pattern you'll end up with something like this where you say hey my user the pk pattern is going to be user hash and then the username and same for the can we go on what is the difference on a primary key and a sort key so people can understand what what this means yep sure so um so with dynamodb uh first of all when we're talking about a primary key again that's how you drive most of your access patterns in dynamodb and there are two types of primary keys the first is a simple primary key that has just a partition key we don't use that very often because it only allows just key value type access patterns so getting a single item at a at a time the second type of primary key is what's called a composite primary key and if you have more complex applications that's what you're going to be using and and the composite primary key has two elements it has a partition key which we have as as pk here and then it has a sort key and the way to think about this is partition key is sort of like a group by right so anything that has the same partition key is going to be grouped in and next to each other in the database on the same partition it's going to be very easy to to retrieve items that have the same partition key and then within items that have the same partition key they will be ordered by that sort key and that sort key is going to be a b tree going to be very efficient to get that you can sort of think of within a given partition key everything's ordered like a a phone book or like a physical dictionary to where it's very easy to get a particular word or say hey give me all the items between d and f in the in the dictionary right and it's very easy to sort of find that range of items and return them back to you in that that composite primary key pattern with that partition key and that sort key that's how dynamodb gives you that consistency of performance no matter how how big your your data gets yes and do you want to talk about index like the indexes that you have there or you want to talk about that later yeah sure i didn't um bring that up too much but yeah that's a good point so we talked a little bit about how you know the primary key is really going to drive your access patterns with dynamodb that's how you access it but you know what if you do have multiple access patterns on an individual type of item right you want to be able to access um maybe a user by the username but maybe also the user by that email address how do you enable those what you can do is create what's called secondary indexes and what these secondary indexes do you declare them on your table and you're basically giving you another primary key for your table that you can perform read-based operations on so when you write an item into your main table it's going to look at that item see if it has the attributes for your secondary index and if it does it'll copy that item into that secondary index using those new uh attributes as the the primary key there and then you can read it you can access it that way so that user you know if you access if you index it by its username as well in that secondary index you could access the user by its username even though it's not the primary key i think that's super powerful because maintaining multiple date tables with similar data it becomes very complex but this is like transactions that happen in the back like you don't need to care so because at the end i think they are maintainers like they have their own cost the indexes like if they were their own table so they behave like their own thing but everything gets copied automatically a synchronous in the back so whenever you update the main entity the main table all the indexes get populated everything is totally synchronized and i think that's that's one of my favorite features from the indexes because absolutely yeah i mean like you're saying like you don't have to maintain that you can rely on these aws engineers that are they're really good at what they do and they'll handle those copies of data for you so because copying data in distributed systems can lead to a lot of pain a lot of failure modes you need to think about that that now you no longer need to think about because aws is happening now yeah yeah so definitely use the indexes yeah um so that's what i like that's just sort of the process of of what you're gonna have right understand your application sort of detail that out once you've done that you list your access pattern you make that access pattern chart you make that entity chart and you sort of fill that out by designing your table and then once that's done you actually go and implement it in your code and i think that's what we'll walk through sort of the rest of the time look at some of these these patterns that you might see that are that are pretty common in your code and uh and yeah i think that'll be um yeah but i think this process is super important because we tend to not do it that often like we just go directly and oh let's code no exactly and it's fine to go do some exploratory coding if you want but at some point bring it back standardize that do all that stuff and it's also different than a relational database you know where relational is sort of abstract from your access patterns you normalize your data first and then you think about how am i going to write those sql queries to access that data and it's completely flipped with dyno right you need to think access patterns first and once you know those access patterns you design your table to handle those access patterns yeah and one common question i get when i talk about the single table design or this concept of really thinking about the access pattern is how do you keep up with applications that keep on growing and changing and migrating and how you do that yeah yeah it's a good question um and it really depends on the types of migrations you have i like to split them into three different buckets so the first bucket is like existing item types you're adding new attributes onto those items right like maybe for those users we had before we didn't use to store a birthday and now we do or something like that for something like that if it's not something you're going to access the data by you can just make that change in your code right and it's just like you have a default value if it doesn't exist or you allow it to be null and it's just a code change you don't have to change anything about existing items in dynamodb so that's the first one that's pretty easy and it's easy because dynamodb is schema-less other than that primary key second type you have is when you're adding a new type of entity into your application right so you know we have all these five entities in instagram um but maybe we have maybe want to add a new entity of something i don't know what a new type would be reels reels exactly i don't use instagram enough to know this stuff but yeah say we add reels in and now we have this new thing well we don't need to change any of our existing data right because reels didn't exist before we just need to model that in our application and going forward that sort of goes now the third part and this is the hardest part and this is the one i think you're talking about and a lot of people are asking about is like i have existing data in dynamo that i need to access in new ways right and that would be like the user if we weren't allowing them to be accessed by user or by email and now we want to add that pattern later on how do we do that and what you're going to need to do is basically do like an etl process on your data where you um you'll perform some sort of background job where you scan your entire table you identify the items that you need to update in this case the user items right and if you have users and photos and likes all on the same table you don't need to change photos and likes but you do need to change users so you're scanning through your table you find a user item and then you just perform an update in place to sort of um you know create a new attribute that can be indexed once that's all once all your items are decorated you know now you have your secondary index and you can handle that pattern and it feels hard because you sort of have to manually do it in a way that you didn't have to in relational databases but it's also like pretty formulaic once you've done it you know it's that three-part scan identify and update and i think in dynamo you get the benefit that you don't need to worry about that background process because i did that for uh really non-relational databases like what was the name now it's out of well react it was called and we needed to go through all the items but it was hosted so basically we needed to be careful not to put a lot of pain in the system because it could crash so we could not go that fast on the iteration but in dynamo you can basically launch as many like processes as you want and go as fast as you want because it'll be a problem yeah just put your permission up and go for it it's so true and like i love how like fully managed dynamo is where you basically have like one knob you need to turn like how many units am i am i having and other than that you're good to go fruition more and let it go and you don't need to worry about anything because i remember when we were doing those migrations in riyak it was like okay let's leave it running for 27 days i was working in the company that was making angry birds and our database was one of our databases what's that and we had updates on the database all the time for new features and new games and new things and and it was happening that now the whole week there is this react process growing don't touch so dynamo i like that yeah yeah so dynamo's got a lot of features makes that easier another thing that's cool about dynamo is they allow you to do what's called a parallel scan on your table so a scan is going to look at every item in your table you know if you have a big table like you do at rovio that's going to take a long time to do but you can actually do a parallel scan where you can put a thousand different workers at it and dynamo will do the work to sort of chop up your table into a thousand different sections they can all be working independently at that and make it happen a lot quicker so that's amazing yeah so it's not a problem anymore to add those things so if you need to migrate your data you mean it's an extra work but you will need extra work in any non-relational database yeah or in relational database as well so yes so do you want to show a little bit of the code now let's do it yeah so again you can check out this repo i'm sure it'll be linked but um you can do like a proper youtuber it's in the description box yep yeah um okay so here's my my vs code window um so this repo i'm all deploying this with sort of serverless technology so i'm doing serverless framework here so if you want to look there's a servles.yaml file that it's my infrastructures code that describes what i'm actually building here you can look through that and it's going to have um let's see it's going to have my functions and i just have a function for sort of each access pattern i have right so here's the create user function is here's the path that's going to be at slash users here's the method post and here's the handler that'll actually run when that happens got a bunch of different functions here maybe you know 10 or 15 that i'm running through there and then i also have a resources section that's where i can provision additional resources using cloud formation i'm provisioning my dynamodb table here and i have the different attributes key schema all that stuff uh secondary indexes that i need probably last thing i should say is you know i need my my lambda functions will need some permissions to access that table so i have i am role statements up here that give you know get item put item query update item all the stuff that i'm going to need on my dynamodb items so i'm giving that access to my lambda functions here we go from lines 102 down here has this so it's got my type which is going to be just an aws dynamodb table again you need to declare that primary key on your table in dynamodb and the way you do that is through this key schema attribute and i'm using a composite primary key that has two elements so the first element the attribute name is pk and the key type is hash which is another name for that partition key um and then that's always so confusing that one is hash and range and in dynamo is partitioned and sort i know that's that's my least favorite thing i wish we could we could change that but it'd break too many people's things at this point so uh and then yes and then i also have the sort key which is the range as part of that whenever you have an attribute in dynamodb you need to declare the type of that attribute as well and it could be a string it could be a number it could be binary it could also be complex things like lists or sets or maps so whenever you're sort of declaring an attribute in dynamo you need to have that so that's what we also have up here in our attribute definitions so you can see up here i have pk and sk for those two attributes and they're both attribute type of s which is for a string so i'm just saying my primary key is going to have a partition key of pk which is a string and a sort key named sk which is also a string so that's what those are right there and the good thing that you don't need to define any of the other attributes because it's a schema less so you only define the ones that you are kind of mentioning so like this exactly yep and then um and then i also needed a single secondary index on that so i have this global secondary indexes section i give it an index name of of gsi1 and then you just have to declare the key schema of that just exactly similar to the the key schema on my main table but this is going to be for your secondary index so here i'm using that same pattern of you know gsi1 pk gsi1 sk and then because i'm declaring those here i also need to clear them in my attribute definitions up there last thing here i'm using billing mode paper request so dynamodb has two different billing modes the the traditional one is provision capacity where you say um this is how many read units and this is how many right units i want they're going to be provisioned for you ahead of time they're going to be available per second every single second you can use consume that much from your table uh the second mode which is the newer mode it's it's on demand or pay per request where you don't need to provision anything upfront in advance you can you can just say hey pay me bill me for every single request i make against my table you know when a read unit when a read comes in they'll see how many units it consumes and charge you accordingly for that same thing with rights and you don't really need to worry about it at all which is which is really nice just in terms of like the time you're going to save on capacity planning but also so paper request is gonna be more than provision capacity on like a fully utilized basis but uh i find that people in myself included are really bad at fully utilizing their dynamodb table you know you have you have peaks and and flows during the day during the week during the month things like that unless you have something very stable then yeah i think once way i have seen it some some customers use it is that they do pay for requests for a while and they understand what is the usage and if they find these tables that are very predictable then they provision concurrency but if not they stay on the kind of paper requests but i think paper request is so much easier it is it's so nice and the advice i give to people is like use paper requests until your bill is is actually meaningful because you know if you're spending less than a hundred dollars a month on dynamo it's not worth the hours of engineering time to figure out how to right size that or scale up and down things like just send on paper requests and wait till that bill is meaningful and then see if you can optimize it yeah so um so yeah i think that's uh that's good so now we have this table so that's that's how we have our table and then you know if you're using serverless i already deployed this application but if you run sls deploy it'll deploy all this so uh what this is going to do with with serverless framework it's going to be similar to sam if you use that but it'll create all your different individual lambda functions it'll package up those zip files upload them to s3 then it'll you know register your lambda functions it'll register the endpoints it'll create my dynamodb table set up the im permissions all that stuff for you in a nice repeatable way so even though i already deployed this i can run it again and and deploy it and it'll only change what changed which which should be nothing since the last time but um so yeah it's pretty nice that way so definitely recommend using some sort of infrastructure as code tool to to handle this sort of thing and you can also see now that i've got this deployed here all my endpoints and and things like that um for my for my application that i have the different functions all that stuff so yeah they want to create a user to show us how that works yeah absolutely so so i'll pop over into insomnia here i'm going to and i've got just a bunch of endpoints to find i need to update my environment real quick yes so you can point right url done okay so now i have a bunch of different endpoints here that i can test this stuff i create user endpoint i have my base url uh slash users i'm going to send a post request to that so i have username of alex debris name of alex 3 i'll send that one up and hey it worked and it sends that user back um you know i got the username and name it also shows like my follower count and following count right which is which is none right now but as we start to follow users later on that count will change things like that uh let's also just we have two users to play with let's we can follow each other yeah exactly let's send that one up all right so now we have two users in there if i also want to get a particular user i have a get user one right and it's just gonna have user slash and then the username so i can send that back send that up and i got that user if i want to change it to marcia i can do that as well and there we go so we have our post and and get user endpoints working should we hop over and check out sort of the code for for that stuff and how that's working because i think a lot of the magic now is happening in the code yes this database that you created is very abstract it could hold anything yep absolutely so um a couple things i want to um talk about first just in terms of like how we model some of this stuff right so we talked earlier about how you list out your access patterns and then you design your table and as part of designing your table you have that entity chart that says hey for a user item here's the pk here's the sk for that photo item here's the pk here's the sk all that sort of stuff in terms of how i sort of set that up i go to source data and i have this base i'm using typescript here i have a base.ts file and i have this this abstract class that i call item which just defines a few things that that needed to get declared on every entity that's going to subclass this item so i have two abstract getters pk and sk right that return a string so each item or each specific entity that subclasses this item class needs to declare the pk and sk which defines that pk and sk pattern for that sort of thing i also have a keys property that's going to return the pk and sk because it's pretty common to use the keys in some of these different operations if you're doing a get item an update item a delete item you need to have those keys so it's nice to have a method defined on that and then i also have this additional method called to item where if i'm saving this this item to the database i want to have this method i could call this says hey just turn this into a dynamodb item so that i can actually uh save it to the database and i don't like you know in my application code i don't need to think about it it's just going to be contained in that um that actual ending yeah i think that's that makes life easier to to abstract everything out so you don't have dynamo jumping all around your code yeah exactly i like to keep um that's a great point of like you don't have dynamic all around your code like i like to keep some of the dynamo stuff like right this pk sk we talked about indexing attributes and application attributes earlier and like keep the indexing attributes totally away from the rest of your application because your application shouldn't think about that uh your application should be thinking about um you know the application attributes so keep that within your classes and not sort of leaking into your your business logic things like that and what about the operations that you're doing on these uh classes i imagine you also have them hidden like abstracted out yep yep so let's move into like one implementation so in my data folder i have like a user.ts file and first thing i do is i'll just like implement a class user that extends that item right and i say the different properties i'm going to have on it i have my constructor things like that but you can also see i have my pk and sk getters here and what i'm doing is just returning that string that i have from my entity chart so i go to my entity chart and i just say hey what do i need to make that string look like and i go implement it here and that's that's what i'm talking about with dynamo where like once you've done the modeling the implementation is very easy you're going to have this sk in pk for every single one pretty easy to do i also talked about that two item method here right and basically i'm saying hey put the keys in there but then also have the username the name the following count things like that some of this is is is funky dynamodb syntax so yeah just just the explanation of what's going on here um every attribute they put in dynamo has to be typed right that's the string number binary things like that you need to specify that type so what i'm doing here is for this username value i'm saying i'm pushing up a map that says hey this is a string type attribute type s and the value is going to be user name same thing with the the name here whereas the follower count and the following count that's actually a number attribute so then the the type there is n for number and i'm putting uh this value up there all the actual actual attribute values that you send up have to be a string so that's why i'm changing this to a string but i'm indicating that it's a number value yeah by sending up that dynamo needs to know that so that's what we have there but one last thing i want to talk about that i often implement on each of my methods is um like a from item method that uh it'll base it it's like a class method that'll you know you when you get an item back from dynamodb you can call this on it and it's going to return an instance of the class for you oh right so basically just like parses that out and and returns a new user item we'll see how that um you know because it what it's doing is parsing out the username string value right here or it's parsing out the follower count number value and converting it to a number so it's doing all that conversion for you so again you don't have to do that in one place so we'll have a class like that for sort of every single entity we have and then i i also just like implement my different patterns on top of those entities you can have it in a different file like a service file or something like that i just put it in the same one here but so here i have a create user method which is one of our our key methods we're going to have here where you pass in a user item and we'll actually just save it to dynamo so what it's going to do is call put item which is how you insert an item into dynamodb the item you want to put here you can just call that user.2 item so again it's just it's doing all that conversion for you that's all encapsulated in the class you don't really need to think about it and the last thing i have here that you're going to use quite often is a condition expression and what a condition expression is when you're using dynamodb anytime you do a write operation so whether you're inserting an item updating an item deleting an item if you're doing something that changes state and dynamo you can have a condition expression that's evaluated before that right and basically that if that evaluates to uh false then it'll cancel that right and it won't happen this can this can you know prevent you from having bad data in certain ways or doing things you don't want to so in this case what i'm doing is i'm saying um it's saying attribute not exists pk so an attribute doesn't exist that has that same pk value and this is just preventing me from overwriting an existing item so if i'm creating a user with a particular username you know someone says hey create a user alex debris i want to make sure there's not already an existing user with alex debris because otherwise it would just overwrite that user and delete their data so we don't want to do that so this is gonna be a very common thing that you do is have that condition expression um that makes sure but that's a great thing because it's not that you need to go to database and check if the user exists and then come back because if you do that in the meantime maybe if the first function it tells you that it doesn't exist maybe somebody else created it so in this way it's more atomic and everything happens within i know yeah absolutely so it's yeah two things i mean it's it's atomic you don't have the race condition stuff and also it's it's faster because you don't have to make one request come back look at it make another request like it's great and you can do this you know i'd say a common one is going to be around uniqueness so this is a very common pattern but you can also do this to make sure um you know if you have like a bank account balance or inventory count and you make a sale you want to make sure it doesn't go below zero so you can um you know if someone's making a charge in their bank account you can make sure it doesn't go below zero before accepting that charge to happen or if you're accepting an order you can make sure the inventory doesn't go below zero things like that um that can help you with your your logic there that's cool question oh yeah go ahead well i have a couple of questions first one what is that get client that you have there good client that's a great question let's let's skip to this and what i have is in my data i have a client.ts file and it's just a little function that is going to return a dynamodb client for me and and i like to do this for for two different reasons number one when i'm creating this dynamodb client as you can see right here there are certain parameters i might want to put in there to configure that client so i have uh in this case what i have is like in my http options i'm setting some timeouts right both a connect timeout and a request timeout um to be a thousand milliseconds so one second and what i'm doing is just cutting that off if there is some sort of connection timeout and you know in this particular case i think the default timeouts are like 30 seconds so occasionally you just have a request yeah yeah it's crazy like this is like the most common thing that i i see people have issues with like in lambda and things like that is they'll have a function timeout and they have no reason why no idea or your api gateway timing out first and you're like why yeah yeah and you just like have no visibility and it's almost always just like uh you know request gets dropped in certain places you put this in there and i think it'll just retry it automatically if it does hit that timeout so so put those things in there but what you can do is like configure your client with these things http options happens to be one thing but there's a lot of configuration you can put into your client and this way you don't have to do it in every single class right every time you're getting a client you don't have to do that um you can just do it in one place which is really nice so that's one reason i do this the other reason uh if you look at this is like outside of this function i'm like setting this client equal to null and um and then what i'm doing is within my function you know if the client exists return the client otherwise um set the client to this new client and return it and what that does is it like caches my client it makes it like a singleton there and so especially if you have like a lambda function the first time you make that request to dynamodb you know dynamo is using an http connection thing but the first time it does that it has to do the tls handshake and set up that http connection but then if you reuse that same client it already has that connection initialized you don't have to redo that handshake and and it can be much quicker to do that so this first request might take 80 milliseconds 100 milliseconds because it has to do that three-way handshake yeah but then subsequent requests might take six or ten milliseconds so this is just caching that client for you um so you don't have to think about that yeah and i have another question and i think this is a question that i get asked a lot how you test this can you test this locally can you mock somehow all these things that you are building or you will not recommend it yeah um and are you talking about like this sort of clients bus not specifically not just because now you have for example the create user you want to make sure that it's working before putting it to the cloud and i have my visions here my my perspective but i want to get yours until what point you test locally and until what point you test in the cloud or do you clone the cloud in your computer how how how you recommend it yeah it's a good question and i'm not very dogmatic on this one um i realize different people like different things i'd say if your application is pretty simple and straightforward i think it's okay to test locally and when i say that um i would say if it's api gateway lambda and dynamodb i think those are all the tooling around running those locally is is pretty good you can use dynamodb local which emulates dynamodb really well um it's also very easy to like sort of configure your client here to send it to localhost things like that that's what i was telling you when i saw the client it's just like so you could have an environment variable where like if you know process.m.test equals true um then set the endpoint to localhost 800 and that would send it back with your client and again you don't need to set that anywhere else which is which is really nice um so again like api gateway lambda and dynamo are pretty easy if you're doing those locally if you start adding in other parts to your application which i almost always do sqs event bridge step functions kinesis all those things a lot harder to test locally or emulate locally i think the tooling around that isn't as good and at that point what i would say is is test in the cloud and the nice thing about having uh you know uh infrastructure as code whether it's servo cml sam whatever you're doing is it's very easy to like spin up your whole application and what you can do is you can have your production application you can have your staging application but also you as a developer can can spin up your entire you can deploy your entire application to another aws account um and have a pretty high fidelity look at what your application looks like and test it that way and and it's quick to do so it's it's generally cheap to do so especially if you're using serverless friendly tools so api gateway dynamodb lambda sqs those are basically all pay-per-use type things and you're not going to be using it enough on your dev stage to where it's going to cost you anything i mean you can have a basically a high fidelity look at your production environment that costs you less than a dollar a month because you're just not using it very much um so i think that's that's awesome and so i would like i recommend people test in the cloud and just get used to having that sort of workflow um but you know if you do if you really like to test locally and you have a simple enough application i think you can test locally if you want to yeah and i think one one thing i will i will add is to do unit tests and then you can test locally in a safe way without touching the cloud because you need tests don't need internet and then go to cloud and do your other testing because i think people like to debug in the like locally that's basically what they're doing when they're testing locally is trying things monolithics at unit tests make sure all the jsons that you're sending to dynamo make sense and with that class model that you have built is quite simple to build those unit tests around it so i'll let you continue yeah so um yeah just a couple things we covered so far number one like having that bass class that has those those abstract pk sk that get client that we talked about i think is going to be useful especially if you're running in lambda to where you can cache that client and things like that and then using condition expressions to prevent overwrites the next pattern i want to talk about this is kind of a fun one um often in your in your application you have neat um access patterns with sort of two needs and i would say you have entities that need a unique id so like a uuid but you also want some sorting particularly around like creation date when that item was created and and how can you do that right because uuids aren't sortable by themselves but you can use something that's called a ul id so here i'm importing the ul id library and you can go check this out there are ul ids implementations for like almost every language out there or at least like the major languages and what a ul id is it's basically got the uniqueness of a uuid but it's it's prefixed with the timestamp at when it was created so um that you that ul id will be prefixed with that time stamp now within your dynamodb you know we talked about partitions and then within that partition it's sorted according to that sort key they'll be sorted in creation order date there so you get nice ordering um from that ul id so if i will use this from now on because i hate my ui ids that are random and then it's like i want to know which one is the last one it's awesome there are like a couple different types of implementations so there's ulid which is pretty popular there's also one called ksuid which came from the folks at segment a few years back but like if you want a sortable unique id it's really nice that you can get based like both of those in in one in one thing there so awesome use ui ids as those i'll i'll show like a quick example of how that works too let me go back to insomnia i i don't think i've implemented this photo which i should have but i'll i did it for likes so let's get back to insomnia and maybe let's create a photo and then like a photo a few different times too so um here i have a create photo access pattern where i'm just posting to you know a particular user photo and i'm sending up the url where that photo exists because i probably up upload it client side something like that so i'll hit send and now i have this this photo in my in the cloud here let's get that id and now it's like that photo so so now my um this access pattern is for this particular photo if you post to the likes endpoint you can you can actually like it uh as part of that body we're sending up the liking username that's just because i didn't implement auth in actuality you would probably pull that from auth which username that was but we're setting that up yeah so marcia like my photo here we go that got implemented there if you look at these ids these are each ksu ids here so you can see what that sort or not ksu id sorry ul ids you can see what that looks like so if you look at this the first you know six or eight characters that's actually going to be the time stamp in there and then the rest of that is just going to be some random values um that gives you your uniqueness i'm also going to like this again with alex debris you're liking your own photos you know i'm vain i gotta i gotta pump those likes up there right so and get that um but now let me copy that and we'll go to actually so we'll go to list likes for photos right where we wanna get all the the likes for a particular photo i posted that that photo id in there sent that back um and you can see it's ordered in sort of most recent to to least recent here so i have that's nice here's alex debris who liked it second and then marcia who liked it um first so you can see like hey who's like this most recently so you know if you look at instagram or twitter things like that that's probably uh similar ish to how they're they're implementing something like this that's super cool last thing i want to show and then um we'll hop back into the code here and show both how that sort of sorting stuff works how those queries work but also how do we doing how we're doing likes i'm going to retrieve that photo back so i did a get on that particular photo it returned that photo item here's the owning user the url the photo id notice how it has a likes count and a comment count right so now i have the likes count of two because you know i liked it marcia liked it it's good to it's got those two likes on it so we'll see how to implement that sort of thing here in a second so i'll switch back to vs code for that i because that's a common use of pattern when you want to count things yep yeah often when you have sort of a relationship you know a one-to-many or a minimum menu relationship on that parent item you often want to maintain like a reference count to show how that is and it's not very efficient to sort of grab that every time like you know read all the likes every time and say oh now it's five now it's ten rather just like maintain that counter as you go is is is a better way to handle that usually so you can see on my photo item here i have both a likes count and a comment count that i'm going to initialize to zero when we get started and now let's look at the like and see what we're doing there oops like okay so i have an access pattern called like photo and what i'm going to do is execute a transaction operation and what a dynamodb transaction does is it allows you to do multiple write operations in a single request and those operations will will succeed or fail together so if one of those operations fails the entire operation fails it rolls back all those other changes and it's a nice way to you know like a transaction in a relational database do a bunch of operations together and make sure that they all succeed um so in this particular operation we're going to do two in this transaction we're gonna do two operations number one we'll do an insert of that actual like right so we're saying uh marcia is liking this particular photo and and just like we did that put item with uh the user we're gonna do that same thing here you'll also notice we have a condition expression here same sort of condition expression we want to make sure marcia didn't already like this photo because we don't want her just sort of pumping the light counts on my photos things like that one like per per user here so we're making sure that that doesn't that one fails the whole thing fails exactly yep so if marcia already like this we're not going to do anything else this will fail and we'll roll it back um in addition that second operation we're going to do is an update operation and this is going to be on the photo so we're saying which item is it we're using the key to do that we're doing that photo and that's where that dot keys method comes in earlier to identify which photo i want to do and what we're doing is we're setting an update operation where we're incrementing the likes count so just this looks kind of funky but we'll walk through it here what we're doing is setting the likes count attribute equal to the likes count the current value of the likes count plus an increment value so let's go look at this um so this hash likes count what that doing is this is an expression attribute name here and it's actually saying uh it's it's the value of this its name is this is is the likes count there um and then this colon increment is saying we want to increment this value by one so when you have these update expressions you'll need you'll also have these expression attribute names and values which is sort of substitute um names and values in there but yeah that's what's going on here is we're just incrementing the value of that length that's always confusing when you see it for the first time but after a while you you learn how it works yeah yeah i'm pretty yeah i don't want to get into it here because it's like so confusing but yeah it is weird to see what that looks like for sure but basically what we're doing there is incrementing that likes count so we want those two operations to sort of succeed together and another thing we're doing you know on that update is we're making sure um in this condition expression when we're updating the photo we want to say make sure this attribute exists so we want to make sure this photo actually exists because we don't want marcia to like some you know photo that doesn't exist so again same thing if that photo doesn't exist this will fail we won't put that like item in there and and we'll just move on so those will all those will both go together in this particular operation um as we go and that's that's a very common pattern yeah using that that transaction with with reference counts like that because handling that kind of atomicity if not is very complicated because imagine if i'm liking the photo and at the same time you are deleting it then there will be a like in some word that doesn't have a parent and in this case is no cannot yeah yeah if you're doing multiple operations and you have to like roll it back with like a saga pattern it's like it's just it's hard yeah so it's nice to be able to do that in transactions and transactions are actually fairly new in dynamo i think that was 2018 or 2019 re invent um but yeah huge like that just simplified a ton of things uh being able to do that and in this case everything is in the same table but you can do it in multiple different tables so yep absolutely great point so um next one i want to do and this is following up on something we did right listing the likes for a photo so when we were retrieving those likes back how do we how do we get those out of there what you're going to use there is a query operation so a query operation is is sort of like a it's a list based operation it's a read operation on dynamo where you can read multiple items that have the same partition key and we talked about that earlier partition key items are grouped together and then they're sorted according to that sort key query is how you efficiently get that and that's what we're doing here we're actually using our secondary index in this one and you just have to specify what's called a key condition expression that says hey which which items am i looking for you have to specify the partition key you're looking for then you can also specify conditions on the sort key if you want to i didn't have to here but i'm basically just saying hey i want the gsi1 pk value to be equal to the gsi 1pk for this like i'm also saying scan index forward equals false which means like sort of go reverse order here and that's going to give me um reverse chronological i want the most recent likes first um and then the older likes um later and yeah queries are very powerful so i think if if you're learning dynamo definitely go and check what you can do with queries because you can like bring now that you sorted everything by time you can bring the latest uh five i don't know comments or something like that you can do i don't know really a lot of operations on and even do a little search if you want like uh it begins with like some characters you can bring those things that are in the sort key so i think yeah it's quite powerful what you can do with that uh expression yeah absolutely query's huge like the great thing about it is you know with dynmo that primary key is so important but sometimes you only know part of the primary key right like if you're if you want to pull somebody's orders you don't know what what the order id was you just know that sort of username and you just say hey give me all the orders for this username like you said it's it's sorted chronologically you can you can get the most recent ones first and uh and you're good to go there so it works it works really well so that's cool yeah i think this is something super important for people to to learn um cool one more pattern i want to i want to show off and this is about many to many relationships and how to handle some of those things so um actually let's let's hop back and do a quick one in um insomnia again so let's follow somebody right so we have a follow endpoint i want to follow particular people it's going to affect what shows up in my timeline things like that you know if you're familiar with instagram twitter any of that stuff so let's follow someone uh let's say alex let's say marcia wants to follow me so we're sending that one up here and you can see we've added this follow in here and now it's saying marcia is following alex debris we have two access patterns based on that right we want to find all the followers of a user so if i want to see hey who is following me i can go look at that and i can also say hey look at all the followed by for a user so who are all the people that marshy is following right so list followers for user let's do that one real quick and you can see that i have my my one follower here and the nice thing here is that it has all the information about martial it's got her whole user object it has her username her name her follower count following count things like that um so all that there we're going to see in a second how we how we do that uh same thing if we want to do the followed by a user we're going to fi find all the people that uh marcia is following oh i'm following you there we go wait did i do that wrong oh yeah there we go no yeah yeah there we go that's right okay so there's uh there's me in there so um off it like this is a many-to-many relationship right like i can uh follow many people i can also be followed by many people and it's not bi-directional like those can be distinct so uh many-to-many relationship let's see how we implement this one because this is a little trickier yeah but that's a very common use use case now in social media yeah yeah absolutely um so the thing about our follow record is it's going to be like a very bare bones record it's it's sort of just like a bi-directional pointer to um to the two people involved but i don't duplicate all the information about the user onto that follow record because it's so mutable right like if we looked at that user that came back it had a username which isn't going to change but maybe you know maybe you change your name and have a different display name maybe and also like your follower and following count is going to be changing all the time so it's hard to denormalize duplicate that data onto each follow relationship so the follow is going to be very bare bones just point to it point to the two users and then if we want to oh yeah here's a transaction for that follow user if you want to look in that i'll just quickly do it and and we're going to insert that that follow in there again making sure it doesn't already exist because you can't follow someone more than once then we also have two update operations where we want to update the follower count of the person being followed we want to update the following count of the person that's following so we have three operations in this transaction if any of them fails it's going to roll it back and the follow won't happen and then but it's going to update all those counts for us which is which is really nice um so there's that let's move into like those those list-based operations how do we do this uh and it's gonna be a two-step process here so list the followers of a user first we'll run this query operation so exactly we're doing before get that query operation which i only know part of the the key right i want to know the followers of alex debris or the followers of marcia i can put that in there and get all those followers and now i have those follows back once i have those follow items those items point to the person on the other end of that relationship and we'll come down here we'll create keys out of those don't even know about that too much but we'll run a batch get item operation which says hey i have 10 users that are following me right and i want to go sort of enrich all the information about that user i want to get their display name i want to get their following count all that stuff i'll go do a batch get item to retrieve each of those individual items from the dynamodb table i can get up to i care if it's 25 or 100 in a single operation using this batch get item request and i just get all those at once they come back to me and then i map over those and return those users and that's how i go from here those followers are just pointing to those people to actually enriching each of those items and sending them back in that response yeah and that's more efficient than doing a query for each user like iterating over the list and yeah exactly yeah you can just go straight to it and get exactly what you want there i think a lesson learned from your talk is learn look at the dynamo documentation and information there's so much many things that are coming through dynamo that you don't need to reinvent the way yeah yeah it's re like the great thing about dynamo is um you know you you need to learn like two or three key concepts about how they organize data how you need to think about data once you sort of grok those then it's like you learn five to ten to fifteen patterns not a ton of patterns like these five we talked about here are the most common ones by a long shot you learn those and it's just like oh you just re-implement those over and over it's just like this very uh basic building block that can allow you to implement all these different patterns and again it's going to give you that consistent performance as you scale and really nice yeah so that's super cool that's all i have for today i think yeah but that's really cool and i think this was super educational for everybody watching because uh i learned a lot of things and i'm pretty sure everybody else did learn but if they want to learn more i recommend alex book that is like mind-blowing into dynamo and learning all the secrets and what what you can do so i will leave the link in the description box for people to go and check check your book thank you i appreciate that and thanks for having me on this was uh really fun you know a long time for sure and fun to hop on and actually do this so yeah it's always a pleasure to to have a chat with you so i hope to see you next year in rain band absolutely yeah i'll be there so we will all be there i hope yeah yeah and all the links for the github repo and and all the links for finding alex are in the description box so they can go and ask you more questions and learn more from dynamo oh dynamo yeah so thank you very much alex for your time and for making this demo i think everybody really appreciate always to see code and see things in practice because it's so much easier than to see just slides absolutely absolutely glad to do it yes have a great day thank you thanks marcia thank you for watching the episode until the end all the links that we mentioned and the github repo and how to find alex in the internet are in the description box and i see you next week in another episode [Music] you

Channel: FooBar Serverless

Views: 3,307

Rating: 5 out of 5

Keywords: find access patterns, model data for dynamodb, dynamodb, dynamodb data modeling, lambda foobar, basic operations with dynamodb, aws dynamodb, dynamodb tutorial, use dynamodb as a pro, dynamodb primary key and sort key, dynamodb primary key, dynamodb aws, steps to model data for dynamo, advanced data modeling, dynamodb advanced design patterns, alex debrie, advanced data modeling dynamodb, dynamodb best practices, amazon dynamodb

Id: Q6-qWdsa8a4

Length: 71min 16sec (4276 seconds)

Published: Thu Jul 08 2021