AWS Lambda in Go - Lambda Microservice with API Gateway, Lambda, and RDS

hey everybody thanks for tuning back in um today we're continuing our series with AWS Lambda um last video we talked about um setting up a Lambda behind a API Gateway so we route in traffic from The Real World or from the outside world um hits API Gateway and that triggers our Lambda So today we're building off of that a little bit and we are setting up our Lambda to now talk to RDS uh relational database service we're going to be using postgrads specifically um and we're going to be putting our Lambda RDS um behind or inside of a VPC so our RDS is not going to be exposed to the outside world so if you knew the connection credentials you could not get to it uh from the outside world you would need to be inside of the vbc um to do that we're also going to put our Lambda into the VPC so we're still going to have real world out side connections coming in they're going to hit the Gateway just like we set up in the previous video that Gateway just like we set up in the previous video is going to hit Lambda um but the Lambda is now inside of our VPC so we can talk to this RDS that we wired up um and I just put cloudwatch down here just because that's uh by default where all of our logs are going to go so anytime we hit our Lambda and invoke our Lambda um there's going to be uh logs submitted to cloudwatch where we can kind of Monitor and see what's going on on same thing for RDS there's a lot of logs that go to cloudwatch as well that you can check out um yeah so there's a quick architecture diagram we're expanding off the previous video so if you didn't see that previous video take a look there um you'll need that code to start in um with this video as well uh but yeah uh with that being said let's get started and let's get uh started hooking up RDS here um so let's go ahead and hop over to RDS here and let's create a new DB um as part of this series we'll create um a database resource we'll give it um we'll set it up with an a VPC um so that it's not accessible by public and only uh adbs resources that we give access to it um can actually access the database so we'll click uh create database here uh we'll do standard creates postgres and if you want a specific version you can choose it I'm just going to leave their default here I'm going to switch over to Dev test um if you use production you can get these kind of multi- availability zones um we don't need that we just need a single so single DB database 1 that's fine for our username post CR that's good if you'd like you can autogenerate passwords um that's probably the most secure way to do it um for us I'm just going to type one in here here make sure you remember whatever you type in uh cuz we'll need it we'll set the lambda's environment variables to use that password okay so for here couple different classes these are all kind of the the box or the machine that the database is going to run on some of them are a lot more expensive than others if we see we're in standard class now we scroll down um it's estimating our monthly cost to be $118 we can get that a lot cheaper if we got to burst of classes and then we choose the micro version the smallest one that drops our price down to 15 and we can make this a little bit cheaper so we don't need Auto scaling um that's fine solid State's fine allocated storage 20 gigs is the minimum so we'll just keep it there so connectivity this is where we kind of do security things for our database we don't want to make it publicly accessible we want to select no so only devices inside the VPC can connect to our database that's good um if you want you can create a specific VPC like for example if you have one AWS account and you have a bunch of different services or applications running on it you'd probably want to create a different VPC for each one uh we're just going to use default for what we're doing default subnet group no Public Access and we'll just existing yep defaults good don't care about availability Zone um database Port that's this is 5432 is standard for postgrad so we'll keep that if you want you can increase so you can use like IM am access or ceros we're just going to stick with password additional configurations uh we don't care about automated backups here for our initial database name let's just use post encryption sure that's not super important but we'll use that and let's see we'll keep the rest of this okay so we see we get uh much reduced from that like 120 cost um we're down to about 15 bucks a month so let's uh create this here and remember the password you typed in and we're on that default VPC so now we need to jump back to our Lambda and and we need to do a couple of things here go to configuration um we're going to not no we're going to let's see that's fine we want to modify this uh roll right here so let's select the roll and when we created the Lambda this R got created for us and we want to attach another policy um let's see it's filter by Lambda and the specific policy we want here is VPC access execution role this gives us uh the permissions we need to kind of talk to other services and run commands um across the VPN okay so let's attach that policy this lets our land to talk to our RDS basically all right so this Ro is saved um it's updated let's go back to our Lambda and then on the left hand tab once we're in configuration you can see environment variables so remember we need a password so let's put and let's just type in the password okay we'll need a couple of other um configuration variables here let's see we'll need the host this is the actual the endpoint within the VPC that we're that the RDS lives on we'll need the host um Port is going to be 5432 so we don't need that let's remove this so let's get the host um I'll need to let's save and I'll need to actually go to RDS and we'll find the host here all right so on RDS now let's click DB instances and here's the database we just created it's still creating so let's see if we have the host available to us yet okay so we don't have the host available yet so I'm going to wait here I'll pause here until the host shows up when it does does um believe it'll show up right here under endpoint once our database finishes creating all right so our database has finished being created and we see here now under endpoint we've got our host um so let's select that and we can jump back to our Lambda and now we'll plug in that host variable name environment name um that host variable into the environment variables for our Lambda let's add a variable we'll say host and we'll paste in that variable okay so let's save it all right so we got host and password now let's jump to our code um so this is where we left off in the previous video If you haven't watched that we set up an API Gateway to talk to this Lambda um so that we can actually make external requests and communicate with our Lambda so we're going to make a new folder here and we're going to call this let's see we'll call this database you can call it whatever youd like and we'll make a new file we'll call it database. Go and the package is going to be database all right so we'll need a couple of variables here we're going to need um host and this is going to be get in we're going to pull this from the environment and that was host password we're going to pull this from the environment as well that was password Port is just um 5432 the user uh when we created our RDS with our postgres RDS um it was defaulted to postgres there was a field on the creation screen that you could change so if you didn't want to use postr you could put something custom up to you um and I believe we need one more database that's the default database here and um we made that to be post as well Okay so we've got um these environment variables here um let's hop to the next step so let's make a function uh git connection and let's have this return a SQL DB and an error okay so we're going to get uh let's change database to database um so we're going to build that and we're going to hit do open and postgres is the driver we're using and data source that's that psql info string that we just built here so one thing with these um with the data source you need to also import sorry with the driver name you need to import that specific driver um the one we're using is going to be p uh pql right here um it'll be this one here and basically we need to go get that and we need to import it so we can do that here yep so we'll get it and then this is going to be imported as well okay so you have to import this uh whatever driver you're using if you're using MySQL uh Maria whatever um you need to also import the driver that's required for this um for this SQL do open to function correctly so what does this return this returns exactly what we need and an error so we can just return this and what's it complaining about has wrong type of string Port okay so this needs to be an end okay so that'll return our connection so now in main we can set this up so let's make our DB now and let's do SQL DB and remember for Lambda these are going to be our Global variables that we can use between invocations so once you run Lambda once it's going to be um basically booted up and that thing is going to stay alive for some unspecified time AWS controls how long the Lambda lives um it could be 5 minutes it could be 3 minutes um something like that but if you have a bunch of requests coming in it will reuse that Lambda so we'll be reusing the connection we won't be creating hundreds or thousands of new database connections each time somebody queries our Lambda um there'd be one connection basically per Lambda okay um let's set up the DB here so we can say um DB let's call this um DB connection and an error is going to be git DB database. git connection okay and let's see if ER we'll handle that otherwise we'll set our DB to equal DB connection so if there is an error let's see let's just um we'll log that out and we'll do a panic so the Lambda would crash okay so if there's an error we're going to log that error um we'll also zap. error this basically formats it nicely and prints it out so we can see what went wrong otherwise we have a connection set up uh that's good one other thing we can do is just uh DB connection. ping uh we can ping this here to make sure it's alive so if ER doesn't equal nil so pinging the database make sure we can actually connect to it and we can communicate with our database pinging error pinging database and we'll also zap the eror otherwise we've successfully connected the database base we set our Global variable and we're good to go so we're going to do three new endpoints one of the endpoints is going to be for migrations we want to create a table um you can handle your migrations however you'd like uh for us it's we're going to put an endpoints and we'll just call that endpoint it'll be like get/ migrate and that will run a really basic migration script and create a table for us and then we'll have one endpoint to insert a row into the database and one to retrieve all the records from our table so let's set up that migration here let's do const um employees table create table employees uh let's pull up the post go docs all right and I'm just going to copy this down and we will um we'll have an ID uh we'll have an email and then we'll have a first and last name I'm just going to set these to varar uh this and again this video is not really about postgres um itself so if you're wondering what any of this is uh look up some guides for postgres um pretty straightforward to pick up okay so there's create employees table let's call this SQL now make a function to do this create employees table this will pass uh context so take in a context and this will also take a database connection and we'll just return an error and what we can do is just DB do exac um context we'll pass in the context and then we can just pass in our our create table SQL and what does this return this returns a result and an error so we just want the error and let's return the error from the function okay so this is going to try to create the table uh return the error if there's any um we use context here just good practice if you're not familiar with context and go um if the Lambda if the request basically gets terminated like Midway through processing um by passing this context the query will also get terminated as well so you won't have like um really long running queries if the user isn't even waiting for the response um okay let's make this Capital so we can use this outside the package create employ table okay so let's add we'll just modify this in point margarate and we'll run database. create employees table and this will take that context that we get from the Handler and this returns an error and if that's not nil we're going to return this error response could not create um employees table and let's just return this Comm otherwise um that so we get down to this line here we've successfully created the employees table um we're going to return okay and we're also going to return just this success here kind of a success message um let's see body let's see what is this complaining about needs to be a string okay so if this went wrong some kind of internal server error will return here all right uh this should be database and what is this complaining about also needs the DB so we need to pass DB and that's just our Global database variable okay so for migrate uh what we're going to do is try to we're going to call database. create employees table try to run the migration script the really simple one that we set up um if there's an error we're just going to return that error um otherwise we'll just return success all right so there's the first step here um inside of databases here let's create a new file we'll call this employees. go and this is going to be package database and let's make our SQL so the first one is going to be inserting a new employee so create employee SQL and we're going to say insert into employees um we want to do email first name last name and let's make this look a little nicer values one two and three okay that ID column here this serial is going to be autogenerated for us um this is kind of like the primary key and this will be autogenerated and auto incremented as we create more these values here 1 2 and three this is uh we'll pass these in as arguments into the query let's make function create employee um this is going to take a context as usual and we'll take an email uh also a database so SQL DB and then we'll take our email string first name and last name String and we'll do db. exec context and let's see we'll pass our context we'll pass our create employee SQL and then we need to pass our args so those args were email first name and last name and we just care about the error return error and we need to add the return type to the function okay so we're going to execute the query we're passing in the arguments and then we just return the error if there's any error so that's in create uh create employee um next let's do to get employees and we're going to do select star from employees just a simple query um we're going to need a type here as well so we'll say employee it's a struct and it's going to have an ID which is going to be just an an um it'll have an email which is a string it's going to have first name and last name Jason first name and let's set that up here last name all right and let's do the function now that'll actually run this query get employees we'll take the context we'll take a DB and that's all we need and we're going to be returning an array of employee objects and and error as well okay so let's see we're going to set this up make um this is an employee size zero so we're going to make an empty array here let's return employees we're going to make this empty array of um employees and we're just going to set the size to zero um and now let's query so we're going to do dbquery um query context here pass in the context and we're going to pass in our git employees SQL so get employ SQL and this is going to return a rows a SQL do rows object and an error so rows error let's handle the error otherwise for rows. next um we're going to read in data so let's see let's hop on the go. dev docs here and okay so one thing I forgot to do is deferring this close so we need to close the rows uh whenever we're done so usually the easiest way people to do that just add it right below your query with a defer statement this means it'll run at the end of the function uh when the function returns so for row. next um we're going to make a single employee object and then we're going to scan into that object and then we're going to append that onto our employees array and then at the end we'll just do a quick error check so let's bring this here I'm going to bring this down here okay and let's scan this in so we'll say employee employee employee employee so we'll scan the ID the email the first name and then the last name and we don't have this object turn otherwise let's bring this up right here and then we'll say we're going to append the employee we just made onto our employees array I just prefer this syntax versus um VAR um if you'd like to use VAR you can but I just prefer this um syntax here okay so row. error so we're going through each row we're scanning the row into this employee object that we just created above if there's any error we're just going to return nil we don't have any um there was an error retrieving employees and we'll return nil and that error otherwise we append that new employee that we just scanned onto our employees array at the end down here we're checking if there was any error pulling anything out from the rows if there was we want to return nil and that error otherwise we've made our employees array object and we're going to return that uh with a nil for the error okay so there's get employees and there's create employee so that looks good um we'll debug this when we deploy it and test this out okay so we have migrates um let's do else if and set this up okay else if event. path equals um SL employees and event. path equals um sorry event. method We Want U we'll say HTTP do post so this is going to be for creating employees here create a new employee and then this next one is going to be for getting all the employees so event. path okay so in the previous video we didn't talk about this really but you can also check the method right usually for an API you'll have a single um entity for rest you'll have an entity as the endpoint so like employees and then you'll have different methods different actions you perform on that entity so post we're going to create an employee um get we're going to get all the employees so let's do get first here um let's say database. employees this takes the context and the database and this should return employees and an error if the error isn't nil let's handle that and we'll see error fetching employees otherwise um let's build our response we'll build our response oops and we are actually going to want a new response type so this default response um is good for like errors or just basic requests but if we want to return like an array of employees um we'll make a new response for that so type get employees response we'll call it a struct and we'll just put one one item in here and this is going to be a type of database. employee it's going to be array of database. employees and let's call this employees okay so [Music] employees so if there's no error right here we building out the response object so we'll say get employee response that's the type that we're going to return and we want to say [Music] employees and we're just going to pass that employees array that we got from our database and we'll return that we'll return that right here okay so there's getting all the employees so let's get um let's create a new employee now so first also another thing we didn't cover in the last video um we can retrieve the request body um so let's do a json. unmarshall and this takes the data and then the interface that we want to unmarshal it to so we'll do unmarshal and event. body so this is the body of the request that got passed to the Lambda and then and let's say here's an employee so we can set up our employee object here pass the address and this returns an error if there's any issues let's handle the error so if the error isn't nil um if the error isn't nil that means there was some issue with the request body so maybe they're missing a Fields uh maybe it's a bad type right maybe email was an integer instead of a string like we're looking for so for this we'll say bad uh request and then we'll pass the error back also status bad request and we'll pass the the body that we just marshalled up here okay so otherwise we have that employee so we can create it so database. create employee we'll pass our context our DB and we need these values email first name last name so we'll do employee. email employee do first name and employee. last name and this just returns an error let's handle that error and in this case something went wrong at the database layer so this is like an internal server error this is like a 500 and what is it saying no new variables on left side okay so we already set up error above otherwise um we're successful and we've created our employee so this is okay this is okay here and we can just put in a simple message success okay so there's our two endpoints now we've got a post on SL employees that'll actually create one in our database we've got a git on SL employees which just gets an array of all the employees you've created and that first one up here was a migrate we'll say ran migrations all right so there's uh um a really quick setup um here's our database configuration um yeah so let's go ahead and generate our function and let's upload this to our Lambda now and we'll hopefully everything runs okay but if not we'll debug this live here go back to Lambda we'll go to code and let's upload from zip so we got our function and let's save it Okay so we've uploaded our Lambda um this may take a little bit to start running um but let's go ahead and run migrate here so don't worry I've got all the data in here right the email the first name last name um that doesn't matter our Lambda is not really going to care about that um but I have the request set to get and I have the endpoint set to migrate so let's run this okay so we're getting this internal server error um I think I found the issue here we set up our RDS to not take public connections and only work inside of a VPC so we also need to wire our Lambda up to live inside that VPC as well so inside of our Lambda and configuration vbc um we need to configure that so here um we select that default one and then let's get all of our subnets as well and we'll select our default Security Group so this is all the stuff that we put for our RDS just these default ones um okay and let's save that so this should give our Lambda access to the database now so we're updating the function and then once that's done um let's go ahead and rerun that that curl command Okay so we've updated that VPC uh We've updated our Lambda to um have access to the VPC and now if we hop back over here okay just the path any method um on Migrate so let's run that okay so we see ran migrations um so that should create the employees table and now if we run get employees we see we have um an empty setup here for uh we've got an empty setup um so let's go ahead and create our first employee so if we do post on that employees inpo now we're passing in our email first name last name and let's do that uh we see we get success back so um an employee should have been created let's call get again and we see yeah we get an employee back um so there is our first employee um we can create another and now we see we have two employees ID 1 and two and just the same data because that's what we created it with um yep so there we go um so that is a quick run through of how you set up um Lambda behind an API Gateway and then also connect it to RDS um we have three endpoints if you are going to use this um in production or kind of use this for your own product I'd recommend pull out this migration end point either put it in a separate Lambda maybe call that from your build pipeline um if you're on GitHub as part of GitHub actions um maybe have your build trigger this endpoint to run these migrations in case you make any changes or just have some migration scripts in here you could have some SQL files inside of your database folder and you could kind of import those and run those um when the Lambda starts um yeah so there is kind of how you do it um hope you found this useful um hope this is able to help you out and get you kind of up and running with uh Lambda and RDS and please make sure to like comment and subscribe to the channel really helps me out lets me know I'm doing something right and um keeps me motivated to make more videos for you um yeah thanks for watching and I will see you next time

Channel: devtopics

Views: 3,876

Keywords: microservice, golang, microservices, heroku, api, backend, software, docker, docker-compose, docker compose, deployment, deploy, postgresql, postgres, go, aws, lambda

Id: T5eOGsFtZ64

Length: 35min 18sec (2118 seconds)

Published: Sat Jan 29 2022