Golang Web Server and RSS Scraper | Full Tutorial

it's time to build a fully fledged back-end server in go from scratch on our local machines the purpose of the server will be to aggregate data from RSS feeds if you're not familiar with RSS it's a protocol that makes Distributing things like podcasts and blog posts really easy so what our server will allow users to do is add different RSS feeds to its database and then it will go automatically collect all of the posts from those feeds and download them and save them in the database so that we can then view them later before we get started there are four things you're going to need the first is a basic understanding of SQL the language that's most often used to query relational databases if you're not familiar with SQL yet that's okay I've got a full course on SQL I'll link that down in the description below go watch that then come back here if you're not familiar with SQL number two is you're going to need a text editor and a command line I'm using vs code and zsh here in the video so you'll see me using that feel free to go download those if you want to try them out but you can use whatever text editor you want as long as it can edit your files and you have access to a terminal to run commands number three is the go programming language itself if you don't have that you can go download it I will link uh the download page down in the description below and I'd also recommend just installing the go plug-in um if you're in vs code there is an official go plug-in Go download that it'll make your life easier with syntax highlighting and formatting and that sort of thing number four the last thing you'll need is an http client so an H D client will allow you to make get and post and put requests into the web server that we're building we'll need that for testing I use the Thunder client it's a vs code extension but you can use anything you like even curl on the command line or Postman insomnia there's tons of choices Google HTTP clients or if you don't already have a preference again if you're in vs code I'd recommend the Thunder client extension now that you have all of those tools installed and hopefully working let's jump into the project the first thing we're going to need is just a main.go file we'll create our entry point it will be a part of the main package and it will need a main function so Funk main takes no arguments return those returns no parameters and for now let's just print hello world make sure we can kind of build and run this program we're going to need to initialize a new module for our project so I'm going to do go mod init and I like to name my modules after their remote path right where they will exist um kind of out on the internet so in my case I use github.com my GitHub uh namespace which is wagslane Slash the name of this repository which again is where I'll be keeping this code on GitHub so I highly recommend you keep track of this code in GitHub this entire project should be checked into git and uploaded to GitHub or gitlab or whatever you prefer so I'm going to name this repository RSS aggregator RSS AG so we'll go ahead and create that that will create this new go module and now that that's ready we should be able to just go build and execute the new um binary or the new executable that will be created from go buildings I'll just run it once so you can see that created this new binary file here in my current directory and so from now on I'll be running this command go build and Dot slash RSS AG so that will build and run and we got hello world back so we're good to go like I said before we're going to be building this project with Git and storing our code in Source control as we go so I'm going to go ahead and create a new git repository and in vs code it highlights all the code that has changed but not yet been committed to Source control here in green but I don't want all of this in my source control the dot vs code file is configurations for my editor that's a personal thing that doesn't need to be in the project itself same with the RSS AG binary we don't want to commit the binary that we're building or the executable file that we're building we just want to commit our source code so I'm going to create a new DOT get ignore file and we're going to ignore the dot vs code folder and the RSS AG binary and next um we're going to add all of the secrets the like configuration secrets for our project in a DOT EnV file and read them out of the file itself so for example one of the configuration variables that we're going to set is the port that the server will run on so I'm going to set that port to 8000. and I'm going to also ignore that dot EnV file in the git ignore again because configuration data is something kind of local to my machine in production this port might be something different so we don't need to commit this file to Source control um but I do want it here in my repo now at this point I do want to pause and say if you have no idea what a port is or you have no idea what HTTP requests are or rest apis are um we're going to move fairly quickly in this project so if you're not familiar with that stuff again I will link down in the description below my HTTP course that would be a good one to go brush up on before working on this project so now we need a way to read this port variable into our program so that we can use it and the go standard library has a built-in function called os.getn so it's a an exported function called get end from the OS package and we can get the value of a variable by its key so in this case the key is port and we'll get back a Port string and then for now let's just uh let's just say if Port string equals the empty string then we'll say log dot fatal so log.fatal will exit the program immediately with arrow code 1 and a message and we'll say port is not found in the environment otherwise we'll say port and we'll print the port string okay cool let's go ahead and run that word is not found in the environment okay so the problem here is that Port the environment variable doesn't exist in my current shell session if I wanted to add it I could run in my in my command line export Port equals 8000. and then run this again and then we get Port equals eight thousand the problem is I don't want to manually set this environment variable every time I work on my server I want to pull it from this file so we're going to use a package that allows us to grab environment variables from a DOT ENB file and it is this package here github.com joho slash go.env that's also the URL of the library you can paste that into your browser go check it out but we're just going to install it here locally and that will add it here to our go.mod and then I'm going to run go mod vendor to copy that code here into my my vendor folder we get a kind of a local copy of that we'll run um here we're going to need to actually use it so we do go.env.load and by default load loads the dot EnV file I think we can also optionally pass in dot EnV as the file path and what's uh what's my error here could not import no required module provides oh that's let's go mod tidy that should clean up my imports okay perfect and do I need to do anything else what are we getting here could not import no required module I should probably go mod vendor again that should pull in the code okay so you can see we've kind of imported and downloaded all of that code from the package and now I'm not getting air any errors in my console okay so this will take the environment variables from my DOT EnV file and pull them into my current environment so then I can use os.getendv to load the variable so to test that let's go ahead and change uh the port to 8080. and rerun the server still says eight thousand so something went wrong maybe I'm misremembering how to use this uh use this package let's try just go dot into that load still port 80. what am I doing wrong does it not overwrite you know what it might not overwrite my current session I'm going to kill my current sell session shell session and create a new one and then we'll do this again so now I won't have that exported 8000 that I had in my terminal um run that again okay cool so now it's pulling it from the file because I don't already have it defined in my shell session now I want to take just a second and point out there are text instructions for this entire project over on boot.dev and I'll link that down in the description below we're going to be using a lot of text uh you know code Snippets from those text instructions and they'll be easier to kind of copy and paste and grab from boot.dev directly then trying to you know retype what you're seeing me type here on the screen now we're going to actually spin up our server and we're going to be using the Qi router to do it it's a third-party router very lightweight built on top of kind of the same way that the standard library in go does http routers and so let's go ahead and install those now we'll do go get github.com go Dash chai slash chai or chi I always struggle to pronounce that one we'll install that and we'll install this the cores package from the same uh the same name space the G namespace next we'll create a new router so I'll do router colon equals Qi dot new router fact I should probably go mod vendor have it there and then I'll do another go mod tidy and go modbender to bring it in cool so this creates a new router object next we'll connect up this router to an http.server so that serve colon equals a pointer to an HTTP dot server and a server needs a Handler which will be the router itself and we also need a or an address which is just a colon plus that Port string so in this case it'll be colon you know 8080. cool and then we can call http.listen and serve or sorry not http.listening server we want to call it on the server object so serve.list.serve cool and before we call that in fact I think that returns an error so let's capture that error say if error is not equal nil log dot fatal as in the error is a message okay listen and serve will block so when we get to line 30 our our code basically just stops right here and starts handling HTTP requests if anything goes wrong in the process of handling those requests then an error will be returned and we'll you know log it and exit the program but kind of the happy path for our code is that you know nothing should ever be returned from listen and serve because our server is just going to run forever before we run this let's just add one more kind of logging statement we'll do log dot print line actually let's do printf and we'll say server starting on Port percent V and we'll pass in that Port string okay cool with that let's go ahead and build and run again so go build see what we get Hello World server starting on Port 8080. I should probably remove I should probably remove that hello world at this point now that we have a running server let's go ahead and test it so I'm over here in the Thunder client tab again because I'm using the thunderclient plugin and I'm going to click new request and we're going to make a request to http colon slash localhost right we want to make a request to our own machine on the port that we're running on which I believe is 8080. okay with that let's go ahead and start up our server okay server starting on port 8080 so it should be good now you'll see I don't have a new prompt because my my server is still running if I send this get request perfect we get a 404 that's exactly what we'd expect because remember in our code we haven't actually set up any handlers or anything we just have a server running so we're getting a 404 because we're trying to hit a path in this case the root path and it doesn't have anything any logic there to handle that code if we killed our server and ran it again we just get the connection refused I've configured my thunderclient to actually store all of my tests or my HTTP requests as plain text here in the working directory but I don't want those going into my source control so I'm going to go ahead and add that to the git ignore Thunder Dash tests will ignore everything in there next let's add a course configuration to our router so this is so that people can make requests to our server from a browser and we're going to be using some fairly permissive configurations here and we'll use router dot use and then we'll pass in this cores.handler configuration this comes from that course package that we installed earlier and let me see what am I doing here we need one more parenthesis there and then I'm going to go ahead and vendor this as well so go mod tidy go mod vendor cool so we should have all of that code here in our vendor folder as well I'm not going to go too in depth on exactly what cores are you can definitely go look that up but just to give you a high level overview this configuration is essentially telling our server to send a bunch of uh extra HTTP headers in our responses that will tell browsers hey we allow you to send uh you know requests to http or https versions We allow you to use these methods we allow you to send any headers it's just a way to say hey we're going to allow you to do basically whatever you want there are ways you can tighten up this configuration for security purposes but for now we're just going to be running our project on our local machine so we're going to just open it up make it permissive to avoid any sort of uh kind of weird testing issues if we try to connect to our server through a browser rest API which means all of the request bodies coming in and response bodies going back will have a Json format so let's create a little helper function that will make it easier to send Json responses so I'm going to create a new file called json.go it's going to be in the main package and the function signature is going to look like this so we've got a function we're calling it respond with Json it takes as input a response writer this is the same HTTP response writer that HTTP handlers in go use it's exposed by the standard Library it will take a code so this is the status code we're going to respond with and it will take an interface which is just something that we can Marshal to a Json structure the first thing the function will do is Marshal the payload into a Json object or a Json string and the way we do that is with the standard Library so we need data and error equals Json dot Marshall and we pass in the payload so this function will attempt to Marshall whatever it's given into a Json string and it will return it as bytes and the reason it returns it is bytes is so that we can write it in a binary format directly to the HTTP response which is pretty convenient that fails for whatever reason then what we'll do is we'll write a header to the response and we'll use status code 500 we'll say something went wrong on our end right internal service error or internal server error um and then we'll just return from the function and actually if something goes wrong we should probably log it as well on the server side so that we can see our own logs and see hey we tried to do something and it broke uh so we'll do log dot print line failed to Marshall Json response and let's print the response or let's print what we tried to Marshall that's probably more more interesting we'll use printfs that we can interpolate that value there next we're going to need to add a header to the response to say that we're responding with Json data so we'll do w dot right header and or not right header I'm sorry w dot headers dot is it header dot add and we want to add the content type key so content type and the key will be applica or the value of the application Json so this adds a response header to the HTTP request saying hey we're responding with a content type of application slash Json which is the standard and a value for Json responses and then we should be able to write the status code so we do w dot right header 200 so everything went well and then we need to write the data itself so w dot write and pass in the Json data this will write the response body now that we have a way to respond with some Json data let's create an HTTP Handler that does that so we'll do handlers or Handler readiness again this will be in the main package and we're going to create a new function called Handler Readiness and this is a very specific function signature this is the function signature that you have to use if you want to Define an HTTP Handler in the way that the go standard Library expects so it always takes a response writer as the first parameter and a pointer to an HTTP request as the second parameter and then in the body of this Handler we can just call our respond with Json function so we'll say respondus Json we'll pass in that HTTP response writer we want to respond with a 200 status code and some some response payload in this case all we care about is the 200 okay status code so I'm actually just going to respond with an empty struct which should Marshal to kind of an empty Json object and now that I'm writing this I realize that we actually made a mistake or I made a mistake in the Json uh response Json code we should pass in uh we should use the passed in Response Code instead of hard coding the 200. so if everything goes right we'll use the code given okay now with that we need to hook up our Handler Now using the chi router what we do is we hook up a an HTTP Handler which is this function to a specific HTTP method and path okay so the way we're going to do that is I'm going to create a new router so V1 router and we'll use that same Chi dot new router to do it and I'm going to specify V1 router dot handle handle Funk excuse me I want to handle the slash ready path and I want to handle it with this Handler Readiness function okay so we're we're connecting the Handler Readiness function to the slash ready path and the reason I created this new V1 router is because I'm going to mount that so I can do router.mount to the slash V1 path okay so I'm nesting a V1 router under the slash B1 path and I'm hooking up the Readiness Handler at the slash ready path so the full path for this request will be slash B1 slash ready and that's just so that if we make breaking changes in the future we can kind of have two different handlers one under version one and one under version two for our rest API this is a fairly standard practice and actually I'm going to name this path health Health Z that's just a habit uh that I'm bringing with me from kubernetes Land that's pretty standard to have a slash Health Z path um that you can hit to see if your server is live and running so that's the purpose of this Handler it should just respond if the server is alive and running and everything's good okay so let's go ahead and run the server and make sure it's doing what we'd expect so go build and Dot slash RSS AG that starts up the server and then we can open up thunderclient and now instead of making a request to the root which we'd expect to get a 404 from we'll do slash V1 slash Health Z and make that get request and we get the 200. now here's the weird thing if I change this to a post request and I make that I actually still get a 200 but that's not really Our intention the health Z endpoint should really only be accessible by get request so I'm going to make an update here rather than using the v1rander.handlefunk I'm going to use v1router.get and this will scope the Handler to only fire on get requests okay with that let's go ahead and rebuild our server and check again host should fail method not allowed perfect but the get request should still work so we have a nice helper function for responding with arbitrary Json now I want one for responding with arbitrary error messages so let's do function respond with error it will look very similar but instead of taking a payload which is an interface it will take a message string and this function is basically just going to format that message into a consistent Json object every single time okay I would say if the code is greater than 499 we're going to log a message and that's because uh error codes in the 400 range are client-side errors so we don't really need to know about them it just means someone's using our API in a weird way but we do need to know whenever we're serving we're responding with a 500 level error code because that means we have a bug on our end and we should probably go fix it so we'll do log dot print line responding with 500 level error and we'll just tack the message itself on there okay cool after we do that logging we'll use the respond with Json function but we'll be responding with a specific structure of Json so let's go ahead and Define that as a struct so type um error response is a struct and has one field error which is just a string and we'll add this Json tag to just say this the key that this should Marshal to is error so in go we typically take a struct and add these Json reflect tags to it to specify how we want this json.martial function or on the other side the json.unmarshall function uh to kind of convert this struct into a Json object so in this case we're saying I have an error field it's a string and I want the key for the field to be error so this struct will Marshal into a Json object that looks kind of like uh like this error you know something went wrong right it wouldn't have uh actually wouldn't have that but it would look like that okay and we'll see that in just a second okay so now we get to respond with Json we pass in the response writer a cut the same code that we were given and then we'll just respond with an error response and the error message will be the message that we were given okay let's let's do another Handler so here we can do V1 router dot get create an error endpoint and oh I need I need an actual Handler so we'll create a new one called Handler Handler error and we'll respond with an error instead of passing in an empty struct we'll say something went wrong and we'll respond with a 400 status code client error right okay now we can hook up this error Handler here it will only work on get requests that seems reasonable and basically it's just going to call that respond with error function so it'll be a good way to test that okay let's go ahead and rebuild the server oh what do we screw up routing pattern must begin with Slash ah let's go fix that so you can see here we've got slash Health Z slash V1 we need to start these with a slash it's just the way the chai route or the QI router works cool um let's go open up the Thunder client and send a request do the slash error Handler cool we get the 400 bad request status code and this is that Json body um so every single time that we need to return an error from our server now we can just use this function and it will always use this consistent error format which is great because we can throw this in our documentation and just tell all of the users of our API hey this is what you should expect when something goes wrong now that we have a little bit of our boilerplate set up I'm going to take the opportunity to commit all of this to get so that I don't lose it um I will say that I generally recommend committing the vendor folder so you can think of the vendor folder kind of like the node modules folder if you're familiar with JavaScript land and in JavaScript you would never commit it it's way too big but in go we typically don't have all that many dependencies so it's actually perfectly fine to commit the vendor folder in most scenarios and I'd even recommend it so I'm going to go ahead and add that and commit it we'll say boilerplate or HTTP server complete for this project we're going to use postgres as our SQL database it's a production ready database in fact it's the one I used to build boot.dev you're going to need to install postgres on your local machine make sure that the postgres server is up and running and you have a client installed that you can use to make kind of one-off SQL queries against it I have detailed instructions on how to do all of that in the text instructions for this project over on boot.f so again go check those out if you need to figure out how to install postgres locally and get a postgres client up and running on your machine I use PG admin so that's what you'll see me using in this tutorial so if you followed those instructions then you should have a postgres server running on your local machine and a postgres client installed again I use PG admin that's what you're seeing here on the screen okay so because postgres is running locally I Have This localhost Server here in PG admin that I've connected to again that's the postgres server running on my own machine and under databases I have kind of the built-in postgres database but I want to create a new database that we're going to use for this project so in this case I'm just going to name it RSS AG and we'll create that database and then here within the RSS AG database as long as the uh kind of icons are gold then you're connected and everything is working at least up to this point let's run a quick query against the database just to really make sure everything's working so I'm right clicking here on the RSS AG database and I'm going to click query tool and from this tool I should be able to just write some raw SQL so I'm going to go ahead and do a select version this should just return the current version of postgres that I'm using I'm on version 14.7 and as long as you're on something 14.7 or newer you should be good to go now it's important to keep in mind here that PG admin is just a client for interacting with an SQL database right we're able to write raw SQL code here and run it against our database server if you think about it kind of in an analogous sense PG admin is basically just the same thing as the Thunder client where the Thunder client is a client for running one-off HTTP requests against our server PG admin is a client for running one-off SQL requests or SQL queries against our database directly next we're going to install two command line tools that will allow us to work with SQL databases from our go code much easier now these aren't fully fledged orms if you're familiar with that term these are kind of lightweight libraries that allow us to work with SQL databases using the standard library and just sort of streamline the process for us the first one is called sqlc and again you can find all of these commands in the text instructions over on boot Dev so be sure to be following along over there but we're going to use the go install command to go grab sqlc and install it into our command line once that's done you should be able to just run sqlc version to make sure it's working next we'll install Goose the same way so go install and then the installation path for goose again that link is over in the text instructions and then you can make sure that goose is installed working correctly by typing Goose Dash version the great thing about sqlc and goose is that they work based on Raw SQL there's no kind of fancy query language that's unique to those tools we can just write SQL queries and we're going to store all of that in our repository so I'm going to create a new folder which is called SQL and in there I'll create a new directory called schema and this is where we'll store all of our table definitions or more specifically our migrations so uh we'll start with a users table and the way Goose works is it runs the migrations in order so we're going to start with a 0 0 1 migration and we'll call it users.sql from a very high level the way that database migrations work is they have an up and a down statement so for example here we're creating a users table the up statement will just create a new users table and the down statement will delete that same table so any down statement should just undo the operation of the up statement and that just makes it easy to roll back changes to our database schema if we ever need to the goose command line tool Works based off of SQL comments so we'll start with a comment dash dash plus goose up and dash dash plus goose down and then anything we type here will be considered an up migration and anything here will be a down migration so let's start with the up migration it's going to be create create table users and the first field will just be called ID it'll be a uuid a universally unique identifier I prefer uuids to integer primary keys for a number of reasons um I'll link a blog post down in the description below um and that's just going to be a primary key next we're going to need a created at which is a time stamp not null but we must have must have it created at must have an updated at same thing and then a user will also have a name and we'll just make that a text field again let's say that's not null I need to remember to terminate my SQL statements with a semicolon and for the down by migration it's pretty simple we'll just drop the table so drop table users all right let's run our migration but first we're going to need to be able to connect to our local database from our program and from our command line so very first thing is we'll need a DB URL and we'll set it equal to the URL that we use to connect to our local postgres server so this isn't to connect to PG admin this is the same connection string that PG admin uses to connect to the database server we want to go directly to the database so it's going to look something like this postgres is the protocol so colon slash slash again this is just a URL and then we have the authentication part which in my case is Wags Lane because that's the user on my machine and then colon and then password if you have a password for your local database this is where it goes I actually did not set one up because it's just my local database and it's going to be at localhost colon 5432 which is the standard port for postgres and the last part of the URL is just going to be the database name that you created so in my case I believe it was RSS AG o okay so your url will should look very similar to this with maybe you know the username the database name something like that could be could potentially be swapped out on your machine okay to run our migration here I'm going to copy I'm going to copy this database URL and then I'm going to CD into this directory so CD SQL schema and then from here I can run goose postgres so I'm telling I'm telling Goose that hey I'm using a postgres database and then I'll paste in my connection string and type up so this will run the up migration a nasty error here turns out I forgot some commas we need to separate all of these field names with commas cool save that file let's try again so we got OK z001 users.sql no more migrations so that should have run let's check PG admin to make sure that it works so now over in PG admin under my RSS AG database I should be able to come into the schemas tab the tables tag and I can see here that I now have two tables Goose DB version so this is an automatic table created and managed by goose and then I've got the users table that I just created let's go ahead and do a select star from users and we should just be able to see those column names come back now let's make sure that the down migration works as well go ahead and run the exact same thing but this time down instead and you can see that it down migrated the same file now over in PG admin if I right click on tables and click refresh you'll see the user's table is gone and this query should fail now okay so let's re-up migrate to get that database table created again and then the interesting thing about migrations is you can rerun the same up migration and you won't get any errors because Goose knows that you're already migrated up to the most recent version of your migrations now it's time to write a query so we're using sqlc to handle our queries and Goose to handle our migrations so to get sqlc set up we need to create a new file in the root of our project called sqlc.yaml I'm going to paste in this configuration here basically it's just telling sqlc what version we're using what database engine we're using and where we're going to store our queries the raw SQL for our queries are going to live in the SQL directory under a new subdirectory called queries we've specified that here right and here I'm going to create a new file I'm just going to call it users.sql and again this is where the SQL will live and the way sqlc works is that it takes the SQL statements and it generates go code Type safe go code that matches the SQL every sqlc query starts off with a comment that starts with its name so name we'll do a create user statement and it returns one record so we're saying I want a new function called create user oops and it's going to return one user that statement will be insert into users ID created Created at updated at and name values dollar sign one dollar sign to dollar sign three dollar sign four okay so what's this nonsense right in sqlc each dollar sign number is interpolated with the parameters for the function so this statement will create a new function called create user with four parameters and the first parameter will go in right here the second one the third one the fourth one Etc so it allows us to create queries that take arguments as input and then we'll end the query with just return returning returning star okay we want to create a new user and return that record right we expect one record back now let's use sqlc to actually generate the go code for this query we always run sqlc from the root of our package rather than within the queries directory itself and the reason that works is because we have this sqlc.yaml file at the top level okay so if everything was written correctly we should be able to do sqlc generate and what happens is it goes and reads that query and it also reads our table definitions which we've specified here right SQL schema so it knows the shape of our tables and it knows the query we want to create and it can go automatically generate all of this go code in the internal slash database package now we need to actually use the database in our go code so here in main.go I'm going to create a new struct called API config and it's going to hold a connection to a database now this database dot queries type is actually exposed by that code that we generated using sqlc so you can poke around through this package and kind of get familiar with the generated code we never manually update this code that's generated by sqlc it's completely managed by sqlc we're just going to write raw SQL to generate this code okay next thing we need to do is import our database connection so here in dot EnV we have our DB URL and we need to grab that and pull it into our application we're also going to need to disable SSL mode so SSL mode equals disable and it just this just tells our code hey we don't need to be connecting to our local database using encryption we kind of trust our local database so we'll parse that as a string so I'll do DB URL and if the database URL is not found then we'll we'll report a message uh we'll log an error message and exit after that we need to actually connect to the database so the go standard library has a built-in SQL package we can do sql.open the driver name that we'll be using is just postgres and then we can pass in the connection string and this will return a new connection and an error and again if there's an error we'll just go ahead and log a message and exit can't connect to database now this is kind of a weird quirky thing about how go handles databases but we actually need to import a database driver into our program but we don't actually need to call anything from it so the sqlc docs mention this but basically we just need to include this line at the top of our program and we do need to import it so I'll do a go get on that lib PQ and we'll import it using that underscore just to say include this code in my program even though I'm not calling it directly okay with that there now we should be able to create a new API config and let's just call it API CFG and it takes as one of its Fields a DB where am I at I think I scrolled too far DB oh and I should probably go mod tidy and go mod vendor so that I stop getting weird errors in my in my vs code okay uh this API CFG takes a database.queries but if you look here we don't have a database dot queries we have an sql.db so we actually need to convert it into a connection to our package and we can do that with database dot new and we pass it as input the connection and we'll get back queries error here we can say if error it's not equal nil and pass in the queries to the struct oh did I do that wrong maybe this doesn't return an error mismatch two variables but database.new returns one okay cool so this actually can't fail it's just a it's just a simple conversion we could actually even just do this it's probably easier great now we have an API config that we can pass into our handlers so that they have access to our database let's write that create user Handler okay so I'm just going to copy paste this Handler Readiness and change it to Handler user and we'll update this to say Handler this will be the create user Handler now here's the interesting thing about HTTP handlers and go the function signature can't change but we do want to pass into this function an additional piece of data we want to add this API config so the way we do it is by making this function a method so we do API CFG is a pointer to an API config so our function signature Remains the Same right it still just accepts these two parameters but now we have some additional data stored on the struct itself that we can gain access to and let's hook up this create user Handler in Main so we'll add it to the V1 Handler we'll do V1 router Dot host we want this to be a post request to slash users and we want to use the create Handler create user method which we defined on this struct so we can pass in API cfg.handler create user and now our Handler will have access to um to the database okay cool this Handler needs to take as input a Json body it should expect some parameters so we'll do type parameters is struct and I think for now we just need a name and we need to parse the request body into this struct so we'll do Json dot new decoder and r dot body okay and this response or this returns a decoder and we do decoder dot decode and we want to decode into an instance of the parameter struct so we'll do params is an empty parameter struct and we'll decode into a pointer two parameters and this returns an error if anything goes wrong if there is an error then we should use that Handler function that we made earlier respond with error and say something like well we need to pass in W if something goes wrong here it's probably a client-side error right so I'm just going to pass in a 400. and we'll say um let's see error parsing Json cool and then we'll return because we're done at that point if there is an issue okay otherwise we have access to a name so we can use our database to create a user so we do API cfg.db dot create user now this is the method that SQL C generated for us right because it read our create user SQL and it created a create user function for us right and it created the parameters as a struct so that's pretty convenient let's see how this works so create user accepts a context and some create user param so I'm going to use uh I think it's CTX no r dot context there we go so that's the context for this request and then we pass in database dot create user params this is struct and it should have yep all of our uh types that we need to pass into the create user function okay so first things first an ID the ID is a uuid um and this is the first point at which I think we've needed to use them so we're going to have to import this package so github.com Google slash uuid this is a very uh well-known uuid package and go we will go get it with that installed we should be able to do uuid.new and that will just create a new random uuid and if you weren't familiar this is what a uuid looks like in string form it's basically just this really long random uh bit of I mean in this case represented as text that we can use as a primary identifier for every user so every user will get their own random ID cool um I should probably go mod tidy go mod vendor it's just good practice every time you install a new package to make sure you vendor it and go mod tidy kind of cleans up any unused Imports and resolves some issues there okay uh create an app I'm going to set to just time dot now dot UTC it's created now and then updated that should represent the last time it was updated uh which would also be now right because we're creating something new and the user's name will just be params.name right it's whatever was passed in to this HTTP request in the body oh and I just I'm now realizing that I messed up this Json Tech should look like this okay cool um so create user should probably return an error yep returns a new user and an error again if there was an error creating the user we'll want to respond with an error and we'll say couldn't create user uh 400 seems fine and then we'll actually respond with the user object itself okay database dot user and see how that goes well let's let's actually take a look and see what does a database.user even look like I'm curious yeah so all these I mean all these fields are exported so they should Marshall to Json just fine let's go ahead and run this and see and see what we get before we run the code though it looks like I've got a couple little things to resolve here so error is already defined there and then here oh I'm messing something up we need to pass that in we need to interpolate that cool oh and here as well percent V because those are errors okay cool um now let's go ahead and run build and run the code so go build and run RSS hack okay server has started so let's go ahead and open up the Thunder client and now we'll be sending a post request to the user's endpoint and we'll be sending in a Json body oh let me grow this just a little bit and we need to specify a name I'm going to create a new user called Lane and let's see what happens couldn't create user database wagslane does not exist it looks to me like I probably messed up my connection string let's go take a look at that so here in dot EnV yeah okay we forgot to or I forgot to add the name of the database here at the end so we we need to do slash name of database all right let's try that again let's rebuild the server and resend that request cool we got a 200 response it looks like that is a new random ID great of that updated at and the name next just to make sure that the record actually was created in the database server itself I'm going to pop back over here to PG admin refresh my tables there's our users table and rerun this select star from users perfect looks like we've got one record in here with all of the data that I would expect now I want to make one more optimization to our code here you can see in this Json response that the fee the key names in the Json object are the same as the exported key names here in the user struct in the database package now we can't change this struct manually again this is generated by sqlc so what I think we should do is instead create a new models folder models.go in the main package and here we'll create our own user type so type user struct and it will be nearly identical so let me go grab this one be nearly identical the only difference at this point is that I'm going to add Json tags so that I can specify you know what these names should be and we've been using kind of this snake case convention so I'm just going to stick with that so updated at and name and I'm going to create a function uh we'll say database user to user and it will take a DB user and return a user and all this does is return a new user struct where we've kind of populate it with all of the all the stuff from the database user so again the purpose of this is really just I want to own the shape that's being returned over the wire right on our HTTP responses and now I have the power to configure that easily within within my application so we'll go ahead and just uh paste these in here okay and then in my user Handler rather than responding with the database user I'm going to respond with our user cool let's re-run and build that and let's run our query again now remember we already have a user in our database so I'm going to create a second one let's call this one Rob and this time you can see we have those snake case keys and again I'm going to go check in PG admin to make sure that Rob is there perfect now we've got Lane and Rob and you can see they have a different randomly generated IDs and their timestamps are slightly different we're going to be using API keys to authenticate our users on This Server the nice thing about an API key is Not only is a little more secure than a username and a password but because it's so long it also serves as a unique ID for that user so we don't even need a combo of username password we can just use the API key in order to kind of uniquely identify people so we need to run a migration that adds a new field to the users table so that we can store their API Keys now we've already created this migration that creates the users table and we don't want to modify this because it's generally a really bad idea to go modifying your existing migrations instead we create a new one so I'm going to create a new one and we'll call it zero zero two because we want it to run after the first migration and again Goose uses these numbers to know in which order it should run the migrations and we'll call it users API key and the migration statements are going to look a little bit different okay so the up statement is going to be an alter table so alter table users we'll add a column and we'll call the column just API underscore key it's going to be a varchar so varchar 64. now the difference between varchar and text at least for our purposes here is that the varchar is exactly 64 characters long so we're saying we want our API keys to be 64 characters long and we want those API keys to be unique no two users should have the same API key we also don't want them to be null and we're going to set a default a default API key and this is important because if we didn't set a default we'd run into an issue when we run this migration remember we already have two users in our database currently so if we just try to add a column that has these unique not null constraints on it then what's the SQL database going to do how is it going to generate new API keys that are unique and not null typically it would just default the new um you know you know the field in the existing records to null but because we've said they can't be null and they must be unique we need to provide a unique default for every new record or for excuse me for every existing record in the database so the default value that we need to add again needs to be unique for every person so we're actually going to have to use some random number generation so we can generate a unique API key for every user and this is the snippet of code that does that again you can go grab this in the text instructions for this project over on boot Dev but let me explain basically what it's doing we're generating some random bytes and then we're casting it into a byte array and we're using the Sha 256 hash function to get kind of a a fixed size output so we're saying take a a big random slice of bytes hash them with straw 256 so that we get a fixed size output and then encode it in hexadecimal and that's so we get 64 unique hexadecimal characters this one makes more sense when we actually run the query and you see what the output looks like and then as far as the down migration goes we just need to alter able users and drop column API key again down migrations should just undo whatever was done in the up migration let's go ahead and run this migration so I'm going to need to change directory into SQL schema then we'll run goose postgres and then let me go grab the connection string now we do need to peel off this SSL mode disable Goose doesn't need that just our code needs that so I'm going to grab the rest of the string and we'll run an up migration cool so it looks like it ran successfully let's go see if those default values look good run the select star statement and there you can see the new API keys so big old hexadecimal strings that uniquely identify every user and should be kept secret by the user because just the API key is enough to authenticate the user now that we've got our migration and we've updated our schema we actually need to go update our query right we need to be creating new API keys for new users so let's go update our create user function it should now accept an API key as the last parameter and pass it in here actually you know what if we do it this way we're basically telling our application code hey you need to go generate an API key in the same way that we generated it here in our SQL I think it would actually be easier what if we just take this and plop that in here right so now we'll use we'll use SQL to generate the new API Keys we don't even need to update the function signature of our create user function cool so the SQL will just handle the creation of new API Keys every time a new user is created all right now we should be able to run sqlc generate insert has more Expressions than Target columns let's see oh right sorry we do still need to pass in the API key as the column name um the difference is because we are not using dollar sign five our function signature won't change this got a little confusing I was reading it like a function signature even though it is it is SQL okay run that again it went off without a hitch you can see it updated a few files in our database package now we should be able to go use that in our code but before we test our server let's add one more thing let's give us a way to get a user so we'll create a new function and this one we'll call get user by API key and it should return a single row and it's going to be a select statement so select star from users where API key equals dollar sign one and we'll run sqlc generate again to generate the code for that query you can see it created it here so in our Handler user function we actually don't need to make any changes to our create user Handler right we didn't change the number of parameters that we need to pass in for the API key it's handled kind of under the hood by the SQL query but we do need a new Handler for getting users so let's go ahead and add that so I'm going to copy paste that do Handler get user by API key maybe we can just let's just simplify this let's just call it Handler get user now this Handler is going to look very different this is an authenticated endpoint so in order to create a user on our API basically to register a new account you don't need to have an API key but if you want to get your own user information you have to give us an API key first this isn't going to be the only authenticated endpoint or the endpoint that you can only do if you're logged in so I think it makes sense to kind of abstract the logic for getting a user by their API key into a package so under the internal package here where we have our database code I'm going to create a new package and we'll just call it auth and in there I'll create a new file we'll call it auth.go and this whole package will just be called auth now the only function that we care to export in this auth package is going to be this one called get API key and its purpose get API key it will say it extracts an API key from the headers of an HTTP request so it's going to go into the headers look for a specific header and see if it can find the API key if it can it'll return it otherwise it will return an error now as the authors of This Server we get to decide what we want the authentication header to look like so I'm just going to say example let's expect an authorization header so the key of the header will be authorization and the value will be API key and then sum you know like insert API key here okay so we're looking for a header of this format so first let's look and see if we can find a value associated with the authorization key um we're just using the HTTP standard Library here so we can do headers dot get authorization and this should return let's see a string okay so the value associated with this with this header key now if value is the empty string then we can just say turn empty string and an error say errors.new no authentication info about so otherwise we have a valid value so we could do something like this uh vowels strings dot split bowel okay so strings dot split takes a string as input and a delimiter so we're going to say we want to split this string here right the the value given to us by the authorization header we want to split it on Spaces okay so next we can say if the length of vowels does not equal 2 then again we can return an error saying like you know maybe malformed malformed ah header right because we're expecting that the value of this key is two like two specific values separated by a space the first should be API key and the second should be the API key right the first would just be the string API key and the second should be the actual API key okay so if the length is wrong then uh next we should probably check and make sure they typed this incorrectly so we can say if vowels it's zero does not equal API key malformed auth header we could say malformed first part of auth header okay otherwise we can just return thousand one and no error right because by that point we're sure that all the all this part was correct and we're extracting just the API key okay what did I screw up here errors.new oh right you're not supposed to capitalize errors in go that is a a linting error stylistic error okay cool so now we've got the get API key function we can go ahead and use this in our get user Handler so let's go ahead and grab that API key so API key and error auth.getapi key and we pass in the HTTP headers so r dot header perfect if there is an error let's handle it we can just respond with an error saying auth error and 400 we should probably do like a 403 and in fact now that I'm thinking about HTTP codes creating a user probably should be a 201 instead of a 200. like you probably won't run into any issues for using a 200 but 201 is the created code so it's like a little more correct if you're looking at it from kind of a restful HTTP standpoint and then 403 is one of these kind of permissions errors so that should be good okay now that we have an API key we can use our database query that we created dot getuser by API key again we'll pass in the requests context and the API key I haven't really touched on this yet in go there is a context package in the standard library and basically it gives you a way to track something that's happening across multiple go routines and the most important thing that you can do with a context is you can cancel it so by canceling the context it would effectively kill the HTTP request I don't want to go too much into detail on how all of that works here you could definitely go read up on it but for now just make sure that it's important to kind of use the current context so every http.request has a context on it and you should use that context in any calls you make within the Handler that requires a context just in case uh kind of cancellations happen okay cool uh that returns a user and an error again if there is an error Let's do let's use a better uh string here maybe couldn't get user and this one let's just go with a 404 or a 400 for now cool um and then we can respond with Json this time it will just be a 200 code and again we should cast that database user to the user model that we defined here right with the the nicely formatted Json tags and that should be good okay let's hook up our get user Handler to our router so here we'll do V1 router dot get so same path slash users but we'll be hooking up the get user Handler to the get HTTP method so again same path different method all right let's test our new endpoint um first I'm going to go ahead and rerun sqlc generate because I can't remember if I've done that and then we'll build and run the server okay with that running let's head over to thunderclient and first let's create another new user so Json body let me minimize this a little bit um let's keep create a new user we'll call them Rand and it'll be a post request that looks good okay response came back Rand was created and this is rand's ID ah we screwed something up we're not responding with the API key let's go update that so here in our model I believe it's because we're not casting so we need API e string Json API key and then here we need to do that conversion so this is just getting dropped because we weren't we weren't setting the API key anywhere okay uh let's rebuild and we'll create a new user let's call this one Joe okay cool so Joe was created and it actually returned the API key perfect now let's go ahead and I'm going to create a new request this one also to localhost 8080 slash V1 slash users but this one will be a get and we're going to add some headers or specifically one header right we're going to add our authorization header and its value will be API key and then the actual API key just paste it in there okay let's go ahead and run that and it's returning Joe it's good to test failure cases too so let's just see what happens if we update this header and let's just like let's just make something broken let's just remove a section of the API key and see what happens cool we get a 400 battery Quest couldn't get user SQL no rows in the result set so essentially user is not found perfect so we've got our users set up and our authentication system now it's time to actually get to some business logic right this is an RSS feed aggregator so we need a way to store feeds let's create a new schema or rather a new migration in our schema folder this will be 003 we'll call it feed s.sql now this is going to be a create table migration right we want to create a beads table so we'll do create table feeds now a feed and the drop will also drop the feed stable a feed has an ID just like a user it also has a created at and an updated at and it also has a name like all of that is actually very similar what's unique about a feed is it has a URL which is text as well that is unique and not null and it also has a user ID which references sorry it's a user ID is a uuid which references users ID and we'll also add the on delete Cascade essentially what this does is it says we have a user ID stored in our feeds table that references the ID of a user in the users table right so this is this is relational data this is a relational database essentially what this means is if we try to create a feed for a user ID that does not exist in the users table we'll get an error which is what we want because we don't want feeds to be able to exist without a user who created them and then this on delete Cascade bit just says when a user is deleted I want all of the feeds associated with that user to be deleted automatically it will Cascade and delete all of them and let's run this migration so I'm going to hop into the SQL schema directory and from here we can run goose postgres postgres and grab my connection screen again we don't need the SSL mode stuff for goose grab the rest of it and up cool now over here in PG admin I can do a select star from feeds and make sure that that table exists with those fields next we'll need a query to create a new feed so I'm going to go ahead and copy this queries file update it to feeds and then we'll delete this one because we don't need it and we'll create a create feed and insert into feeds ID created updated at name there will not be an API key but let's see we need a name URL and user ID name URL user ID we won't be generating an API key here instead all of these values I think will just be passed in from our application code so one two three four five six we need six parameters for this function five and six and then we'll just return the entire feed row after it's done being created and then I'll just navigate back to the root of the project and run SQL sqlc generate to create the code for that new query next we're going to create a new Handler that will allow users of our API to create a new feed here's the thing that Handler is also going to need all of this same logic that we have in the get user Handler right we'll need to grab an authentication a token or an API key from the authorization header fetch the user and then use that user in the Handler and rather than copying and pasting uh this what 10 lines of code into every Handler that's authenticated instead we're going to build some middleware to kind of dry up the code right um let's go ahead and do that so I'll create a new file I'm going to call it middleware auth dot go art of the main package and here we're going to define a new type and it's our own custom type I'm calling it auth Handler and you'll notice it looks almost exactly like a regular HTTP Handler the only difference is that it includes a third parameter it has a user associated with it so if you think about this it makes a lot of sense for any authenticated Handler to accept three parameters where the third one is the authenticated user now the problem with this auth Handler type that we created is that it doesn't match the function signature of an HTTP dot Handler func right those functions with just the response writer and the request as the only two parameters so what we're going to do is create a new function called middleware auth that works it's a method on our API config so it has access to the database um but it its job is to take an auth Handler as input and return a Handler func so that we can you know use it with the QI router okay let's Implement that the way this function will actually work is we're going to return a closure so we're returning here a new Anonymous function with that same function signature as your normal HTTP Handler func the difference is that as we Define this function we'll have access to everything within the API config so we'll be able to query the database so we can basically just go rip out the code from our get user Handler and paste it in here all right we're going to go get the API key from the request well from the request headers at least and then we can go ahead and grab the user using that API key all right so we'll have access to the user here in the function finally all we need to do is run the Handler that we were given with the response writer the request and the user right so by the time we get to actually calling the auth Handler we're able to give it an actual user from the database and this is really great let me show you why so now that the middle middleware auth function exists we can remove all of this code from the get user Handler we can update the get user Handler to accept as input a database user user database.user look at how clean this function becomes right now it's just now it's literally just one line cool and now to hook it up you'll notice we have an error over here we just need to call our middleware auth function API CFG dot middleware off we call this function to convert the get user Handler into a standard http.handlerfunk I kind of move fast through that hopefully it all makes sense though basically we're just calling the middleware auth function first to get the authenticated user and then we're calling that callback the the get user Handler the nice thing is now we'll be able to reuse that middleware across many different HTTP Handler functions so now let's create the create feed Handler I'm going to go ahead and just copy this Handler user change it to Handler feed and for now we just need a create function so I'll delete this get and we'll call it Handler create feed and remember it's an authenticated endpoint so we can have it except the user directly so user database dot user we know who's creating the feed by the time we get to this function which is awesome all right in order to create create a feed use our new create feed function which takes create feed params and create feed params have ID created updated at name that's all the same the difference is a URL and a user ID so URL and a user ID which is a uuid which actually exists already on the user object so we just do user.id okay um what do we want as input we need a URL so we also are going to want a params.u well so we want the user that's creating a new feed to be able to just send us a name and a URL and we'll go about creating the entire uh kind of you know feed object in the database or feed Row in the database so this is what our parameters should look like uh what error am I running into here cannot use API cfg.create feed no new variables on left side of that's odd create feed should return a feed let's take a look at that definition yeah it does return a feed what am I messing up here oh first of all that shouldn't be a user that should be a feat okay that's the problem I was over I was trying to overwrite the database.user type with a feed type that won't work okay so we're creating a feed we're generating a new uid that's great we're using the current time perfect um this is getting messed up a variable of type uuid.uid as uuid.no uuid ah okay I see the problem the create feed params except a uuid.null uuid that's a problem we don't ever want a u the null uuid type from the uuid package is a nullable uuid but we don't want it to be nullable because we expect that every feed will be created by a user so let's go update our our uh I think it's our is it our migration let's go look schema feeds yeah user ID uuid not null cool it doesn't need to be unique a user can have multiple feeds but it should never be null all right um with that we're actually going to need to go uh rerun our migration so SQL um schema and we'll do a down to drop the table down and then back up to create the new table with the proper schema and then we should re-run SQL C generate okay did that work create feed create feed params looks like that error is gone it's now just a uuid type perfect now at this point we have a valid database feed I should probably update this error so that it actually says feed and we want to return it in our HTTP response trouble is remember we don't want to just directly return the struct let's go create a new model for a feed so we'll do type feed then I'm going to go just copy the types from here and we'll use those and we'll make our own Json tags for the type user ID URL name updated at and created that okay and then we'll want a what is it database database feed to feed and we'll return a feed all of this should be pretty straightforward and again this just gives us more control right now we in our code that's not generated by SQL C are able to Define you know what the shape of the response will look like if for whatever reason we needed to store some data in the database but never wanted to respond with it in our Json API we could make those changes here in this struct okay so now we've got database feed to feed we'll call that and now we should be good to go all right let's test out our new Handler so I'm going to rebuild our server and run it and over in the Thunder client let's see so we just created a new user Joe so we have our authentication key here or our API key I'm going to do a new request to HTTP slash localhost 8080 slash V1 slash feeds ah now that I typed this out I realize that we never hooked it up let's go hook up that feed so speed Handler Handler create feed we need to go paste this in to the main function so this one will go under slash feeds because we're creating a resource we're going to use a post request Handler create feed okay that should be hooked up now let's restart our server and over here again I need to grab that API key all right what do we want here post request localhost 8080 V1 feeds we do need to authenticate again so query headers then authorization API key paste in that API key okay uh what do we send in as the body let's take a look at our Handler again a name and a URL okay so name now remember this is this is this is not a person this is a feed right and a feed is a URL that kind of links to an RSS feed out on the internet so this one I'm going to put just lanes lanes blog and then the URL is going to be https colon slash wagslane.dev slash index dot XML so in case you're not familiar with what an RSS feed is let me just show you really quick um I'm here on my blog wagslane.dev and if you click RSS up at the top it'll take you to my RSS feed now every RSS feed will have a different URL it's kind of up to the author of the blog or the podcast what that feed URL is but you can usually find it by poking around on their website so in my case it's just wagsly.dev index.xml and it will look something like this if you open it up in a browser it's basically this structured XML document that describes what each post on the blog says at least from a high level it'll usually have something like a link to the post maybe a short description um basic stuff like that again podcasts also work on the same RSS structure so for testing you can use my blog or if you know of any other RSS feeds out on the web you can use them so now that I've pasted in that URL uh let's go ahead and create that feed s got a new URL created that updated at the name and the URL seem to have persisted correctly and that is the user ID associated with the API key that we use to create the feed next we're going to add the ability for any user to get all of the feeds in the database this is not an authenticated endpoint we need new query I'm going to go ahead and use the same file this query will just be called get feeds and it will return many rows instead of just one all right and this one will be uh select star from feeds super super simple query here we're just going to go grab all the feeds and return them okay from there we should be able to sqlc generate and let's go hook up that Handler I'm going to use the same Handler feed file uh but this one will be a little bit different you're going to call this one Handler get feeds it's not authenticated so we don't need to pass in a user and it doesn't even take any parameters right it's just going to go get all of the feeds so API cfg.db dot get feeds and it the get feeds function if you remember we just wrote it in SQL it doesn't take any parameters and here it just returns all the feeds that are currently in the database so this error should be say something like couldn't get feeds cool um now we need to return all of the feeds this is not a single feed this is now a slice of database dot feeds so not only do we need to return them but we need to actually convert them so let's go update our models a little bit let's create a new function this one will be database feeds to feeds and the difference is it will accept a slice of database fees and will return a slice of feeds let's do this slice feed we'll create a new empty slice of feeds and then four DB feed range DB feeds feeds equals append feeds database feeds or database feed and return Beats sorry not we don't want to append it directly we have to we have to call our conversion function okay so this function will just iterate over all of the database feeds one by one converting them into our new feed structure and then returning them cool now here we can use that function to do that mapping great let's hook up this Handler so in main.go we'll create a new entry here this is going to be a get request and it's not authenticated it'll be Handler get feeds cool let's regenerate I can't remember if I generated MySQL C so I'll do that couldn't hurt and then we'll build and run the server okay with that running let's go ahead and create a couple more RSS feeds so here update the body of my request sorry I'm so zoomed in so that you guys can see and it just makes it hard um no this was to create users this is users here's feeds let's let's just add the same URL with a couple well no we can't we can't add the same URL let's just use some garbage URLs just to test all right okay that created properly now let's test our new endpoint this one is going to be a get request to feeds and we don't need to add any authentication information okay run that awesome this looks good to me we've got an array at the top level and then two feed objects one for garbage blog and one's for one for lanes block so it looks like everything's working so we've given users a way to create feeds and a way to query all of the feeds now we're going to give users a way to follow specific feeds so that they can see kind of an aggregated view of all of the feeds that they care about on the system okay so uh let's go ahead and add a new migration we need a new table so this will be the fourth migration and we'll call this new table feed follows and this table is just going to store the relationship between a user and all of the feeds they're following so it'll be a many-to-many uh kind of table of user IDs to feed IDs all right the table is going to be called feed follows so create table feed follows every feed follow like every other record in our database will have an ID created at and an updated at but its unique Fields will be a little bit different first it's going to need a user ID which is oh my gosh why can't I type fingers throw the wrong place in the keyboard so a user ID is a uuid um that can be it doesn't see it doesn't need to be unique um but it does need to be not null then we need a feed ID also a uuid not null and then we're going to create a unique constraint on the combination of user ID to feed ID so unique user ID so again this constraint is going to make it so that we can never have two instances of a follow for the same user feed relationship right you as a user can only follow a certain feed once you can't follow it twice that doesn't really make sense right so we're gonna ensure that that's unique also I missed a couple things here the user ID should reference so references uh the users table ID field and on delete we'll Cascade so if a user is deleted we're going to go delete all of the data about what feeds they're following and then this one's going to be very similar except it references the feeds table with its ID and again if a feed gets deleted then we'll go delete all of the following data related to that feed cool okay let's go ahead and run this migration so I'm gonna go back up into the SQL or say back down into the SQL schema directory and from here I'll need my connection string do goose postgres connection string ah I didn't grab the whole thing let's try that again all of that goose postgres up cool speed follows databases or feed follows table is there so now we need a way for users to follow feeds alright let's go ahead and go create that so I'm going to copy and paste the feed Handler file and we'll call it Handler speed follows and update this so Handler create feed follow so remember in order for a user to follow a feed all we need to do is create a new feed follow record with that user feed relationship okay this is an authenticated endpoint right we so we we need a user and we need them to be authenticated have passed an API key right and let's see what do we need them to give us as input I think all we need is a feed ID right they just need to tell us which feed they want to follow so a feed ID is a uuid all right and now we should be able to create the oh we never we never made we never made the SQL query what am I doing what am I doing I'm getting way ahead of myself let's go add that query quickly so feed follows and to start we'll need a create feed follow okay what's in a feed follow right got all these all of these fields and I think yeah we're just gonna have them all passed in directly that seems like the easiest way so insert into feed follows ID created that updated at user ID ID that's five parameters right one two three four five cool that looks good now I should be able to go back and run sqlc generate cool now I should have a create feed follow function with create feed follow params all right it accepts a user ID and a feed ID so the user ID is just the author authenticated user the feed ID is going to be passed in his params right cool couldn't create feed follow don't need a get Handler quite yet and then we're gonna just need to make make that uh mapping function as well for read follows so in our models file I'll create a new feed follow struct and it's going to have a user ID and a feed ID and a new function database feed follow new feed follow all right DB feed follow dot ID by the way I'm not using GitHub copilot in this video just so that you could just just so you can see more of my thought process but I typically do use GitHub copilot and it makes this kind of function just like way faster to write it would guess this kind of function almost perfectly um so just so you know I I do recommend those kinds of tools to speed up the development process um I'm just not using it uh right now so you can see how I think through you know architecting this this application without all the AI prompts getting in the way okay now we should be able to database feed follow to feed follow and we're gonna be clear that this is a feed follow not a feed and that goes there perfect all right let's hook this up so we're going to need to go into main.go V V1 router dot post because we're creating a resource slash feed follows and this is an authenticated authenticated Handler Handler create feed follow okay let's test this new endpoint so we'll build and run the server and we'll need a new request this one will be kind of similar it'll be oops a post request to the feed follows endpoint and we're going to need to authenticate so let's go grab some authentication information get users let's go ahead and send this couldn't get user SQL no result okay I need to figure out what users I have available to me oh that's right we we changed this API key we wanted it to break let's go create a new user we'll make a new one called uh Billy and there's Billy's API key cool we've got some feeds but our feed follows need an auth section sorry in the headers we're doing it manually authorization API key there's Billy's key and then in the body we need to pass in the ID of the feed that we want to follow so let's do a get on all of the feeds and we can follow either of these let's follow Lane's block there's our feed ID paste that in there and create amazing new ID for the feed follow there's the user ID the feed ID what happens if we try to recreate it cool couldn't create feed follow duplicate key value violates unique constraint that's what we'd expect right we shouldn't be able to follow the same feed multiple times we're already following it we already have a record uh indicating that we are following it everything appears to be working just fine next let's give users a way to see all of the different feeds that they are currently following so we'll do get feed follows it will return many and the query will be select star from eat follows where user ID equals dollar sign one right so get all the feed follows for a given ID so let's get that hooked up need to run SQL C generate to create that query and then down here we'll create a new Handler this Handler will also be authenticated but it's going to be get feed follows have the user we don't need any parameters here and we're just going to call get follows and we'll just need to pass in the user's ID couldn't get feed follow this cool now we've got a list of feed follows or a slice of feed follows so we're going to need to convert an entire slice so again down here we'll write this type of a function going to be database feed follows to feed follows all right lots of copying and pasting here feed follows okay so now we have a way to convert an entire slice of database feed follows to our own struct that looks good to me there feed follows okay cool now we have a Handler for getting feed follows let's go ahead and update this so we need a new V1 router dot get slash feed underscore follows middleware off if feed follow us Perfect all right let's give that a shot so we'll build and run again and now let's see so this is um this is the request that we use to create so let's grab so hard working on such a small screen let's grab our API key and create a new request 80 V1 feed follows it's going to be a get request it does need to be authenticated okay see if that works cool we got the one feedback that we are currently following finally we need a way to unfollow feeds or to delete feed follows so let's create a new one new query we'll do delete feed follow now this one is going to be our first query that doesn't actually return anything um it's just going to be an execute right we're not returning one record we're not returning many records we're returning no records we're just going to run a a SQL query so uh it's going to be delete ROM feed follows where ID equals dollar sign one and user ID equals dollar sign two now it's important it's important to point out that we don't actually need the user ID here for this query to work right the ID is already a unique identifier the reason I'm tacking on this user ID is because this will prevent someone who doesn't own a feed follow from trying to unfollow a feed on behalf of somebody else that makes sense uh if for whatever reason another user got accessed let's say if if for some reason user B got access to the feed follow ID of user a if we didn't have this check here then that user who hijacked a feed follow ID would be able to like unfollow like force the other user to do an unfollow if that makes sense this ensures that only the user who actually owns the follow record can execute the unfollow command hopefully that makes sense okay uh from here let's just go ahead and generate that and go hook it up to a new endpoint so we'll do Handler Elite feed follow now this one's going to be a little different in that it is authenticated but we need to get a feed follow ID and delete requests so like HTTP delete request the delete HTTP method they don't typically have a body in the payload it's it's possible but I would argue it's not super conventional it's a little more conventional to pass the ID in the HTTP path so it's going to look something like this you and router dot delete feed follows slash feed follow ID and then this will be Handler uh Delete feed follow right so we want the feed follow ID dynamically passed in the path of the request so the question is how do we grab this feed follow ID in our Handler itself well the chi router has or Chi I'm never going to say that the proper way the G router has a I think it's Pat is it URL let's see URL parameter that's the one uh you are pro URL parameter function where we can pass in the request and a key and in this case it's going to have to match so feed follow ID matches whatever we type in here between the open and close brackets okay and that's going to return a string so this is the feed follow ID string great we're going to take that and we're going to parse it into a uuid so we'll do uuid.parse and that will return a feed follow ID and potentially an error if the error it's not equal nil we'll say couldn't parse feed Hollow ID and that will be a 400 level error perfect okay from here we should be able to do API cfg.database dot delete feed follow and we need to pass in the request context and feed follow params so database dot V delete feed follow parameters it takes an ID and a user ID so the ID of the feed follow we just parsed and then the user ID comes in with that user object because this is an authenticated request cool and that should return just an error right oh and it's just it's just given me yellow squigglies because I need to handle the error I couldn't delete feed follow perfect uh what do we respond with here I guess we have a couple different options um the simplest thing would just be to respond with like an empty Json object I guess uh what matters to the client is probably the 200 Response Code um so we could like for the sake of Simplicity just so we can use our respond with Json function we'll just return an empty Json object alternatively maybe we could return an object that says like message you know unfollow successful or something um but it doesn't matter too much I think as long as it's a 200 level code we're pretty much good to go okay and that's already been hooked up so let's go ahead and test it I can't remember if I generated let me do that again and then we'll restart the server and take a look okay so this was our endpoint it's returning the feeds that we're currently following let's go ahead and delete this feed follow so we need new request this is going to be a delete request we're going to unfollow a specific ID we're going to unfollow this we're going to delete this feed follow you'd follow with that with that ID and the headers we do need to be authenticated as the same person so authorization same API key okay let's run that delete we got a 200 response now let's go do a get and make sure that it's gone yep empty list or empty array we're good to go okay we've built out the majority of the crud section of our API but we haven't built the most interesting part which is the part of the server that actually goes out and fetches posts from the different RSS feeds that exist in our database again the whole purpose of this server that we're building is so that it can keep track of all of these different feeds in the database and then go out periodically and actually download all of the posts that are on each individual feed so for example we have a feed for my personal blog post this server will actually go out to my blog every I don't know 10 minutes and check to see if there's a new blog post to download and store in the database so the first thing we need to do is update the feeds table to have one more column we need a new column called last fetched at and it's just so we can keep track of when we last fetched the posts for a given fee so let's go ahead and add that we'll need new migration um and it will look kind of like this migration uh it's going to be our fifth migration so far it's going to be on the feeds table and we're going to be adding the last fetched at last fetched at field okay so alter table feeds add column last fetched at and this one is going to be timestamp and it will be nullable so we don't need a not null constraint um in fact that's it um it's okay like we don't need to specify any defaults that should be it and then as far as the down migration goes we'll just be deleting or dropping the column from the feeds table okay cool let's run that migration so goose postgres perfect so we don't need to update the create feed function we want the last fetch that field to default to null so no changes are necessary there but we do need a new we do need a new query this one's going to be called yet next oh my gosh get next feed to fetch can't type today get another speed to fetch and it will return a single row and this one should say select star from feeds order by last fetched at descending nulls first limit one okay so we always this the purpose of this function is to go get the feed that next needs to be fetched like we need to go get posts for this feed next and the whole idea is first we want to go find any feeds that have never been fetched before those need to take priority after that if every feed has been fetched then we want to go find the one that was fetched the longest ago like the farthest in the past right so we're ordering by last fetched at um nulls first in descending order actually scratch that we're going to want to do ascending right ascending would put the lowest the smallest time stamps right the ones further in the past at the top and then Ascend into the present okay so order by last fetch that ascending nulls first perfect okay just to make sure that my SQL code is valid we'll generate that looks good okay next we need one more query this one will be called Mark feed fetched Arc feed I guess as fetched this is one we'll call after we fetch a feed to say that we fetched it and we'll return the updated feed okay so it's going to be update beads set last fetched at equal to now and update it at also equal to now so we haven't really gone over this but the updated and created that fields are mostly for auditing purposes it's pretty standard practice to set these fields on basically every record in an SQL database just so you can see when they've been created and updated it's kind of again auditing purposes okay where ID equals dollar sign one and returning star okay so we update the feeds we set the last fetch stat and the updated at to the current time for the given ID that looks good to me let's go ahead and generate that perfect next we need a way to kind of take an RSS URL or a feed URL and parse it into an actual response body and in this case we're going to represent it as a struct let me show you what I mean so let's create a new file I'm just going to call it rss.go and it's going to be part of the main package and we need a new function and we're going to call it RSS to or actually let's call it url url to feed okay and it's going to take as input a URL which is just a string and it will return a new type so we need to specify the new type type RSS feed it will return both an RSS feed and potentially an error if there's something wrong with the request that it's making now that RSS feed struct that we just created is going to represent basically this giant this giant document here right so if you go to wagslane.dev index.xml which is a valid RSS feed then you'll see this giant document and really you can think of RSS as just a structured data in XML format and XML is just kind of like crappy Json so the way we parse XML in go is very similar to The Way We parse Json let me show you what I mean I've done the Dirty Work of scanning all of the valid values in that big RSS document and I found that basically these are the keys um for the RSS entries in my blog so RSS is kind of a standardized set of keys within XML and basically what I'm saying is these are the keys that we care about right at the top level of an RSS feed we expect a channel key right in the XML document and we expect a channel to have a title a link a description a language and then a slice of items and then items are kind of these nested objects that each have their own title link descriptions and publication dates right and each item is a new blog post and if you're asking how I came up with those names of all of the different Keys it's because I went and looked here in this document I saw okay at the top level we have a channel right and then we have this entry with a title a link a description right so I just kind of manually looked through this document and found all the stuff that I wanted to parse out so let's fill in the rest of this URL to feed function so first we're going to need an HTTP client I'm just creating a new client using the HTTP Library um we'll set it to a timeout of 10 seconds if it takes more than 10 seconds to fetch an RSS feed we don't want that feed anyway probably broken okay uh then we can use that client to make a get request to the URL of the feed and that's going to return an HTTP response and potentially an error if there's an error we'll just return let's just do um for cons for ease of use I'm going to make this a pointer to an RSS feed so we can just return nil and the error cool um if everything's okay then we're going to defer a close on let's close on the rest sorry it's not it's not the close function it's resp.body Dot close okay and then after that we want to get all of the data from the response body so it's going to be io.read all we have to read everything from resp dot body and that comes back as a slice of bytes and an error okay this slice of bytes we want to read into this RSS feed so dealing with XML in go is very similar to dealing with Json and go it's actually going to be XML Dot unmarshall pass in the data and a pointer to where we want to unmarshall today so actually I need to create an empty struct we need RSS feed is an empty RSS feed struct then we'll unmarshall into that location in memory that will return an error if everything goes well then we can just return the new populated RSS feed now as I type this out I'm already kind of dissatisfied with this pointer solution I don't think that needs to be a pointer I think we should just return empty structs either way would work I think this is a little cleaner though because it means the user of this function us right we'll get an actual RSS feedback and not a pointer to an RSS feed okay let's go ahead and test this really quick I'm just going to do a little kind of hacky thing just right at the top of main I'm going to call URL to feed and give it the URL of um my blog so wagslane.dev slash index dot XML that should return a feed and an error right and then if error not equal nil blog dot fatal error otherwise I want to just print out fmt Dot print line let's just print out the whole feed it'll be disgusting but at least we'll get to see if it kind of worked okay let's build and run that there we go cool so if you kind of scroll through this you'll see it looks like I mean there was no errors and then it looks like we properly at least you know at first glance looks like we properly filled out that struct it's kind of just dumping all of the data so now that we've done a sanity test on our URL to feed function let's go write the actual scraper create a new file just call this scraper.go again in the main package okay um the scraper is a long running job so this scraper function is going to be running in the background as our server runs so we'll let's name it something like start scraping and let me split up these parameters so we can really see so we can actually see what uh what we're dealing with here so we'll take three inputs a connection to the database um a number of concurrency units I guess the best way to think about this is how many different go routines we want to do the scraping on and then how much time we want in between each request to go scrape a new RSS feed cool and it shouldn't return anything because this is going to be a long running job now because this worker this scraper is going to be running in the background of our server I think it's really important that we have good logging that kind of tells us what's going on as it's happening so when we start scraping I'm going to do a little log message here so log dot printf we'll say scraping on percent the go routines every percent s duration as in the concurrency and the time between requests cool after that we need to figure out like how we're going to make our requests on this interval and there's a really cool mechanism in the standard library and go called a ticker so we can create a new ticker uh using the standard Library so time dot new ticker and we give it a duration in this case time between requests and it responds with a ticker and then we can use a for Loop to execute the body of the for Loop every time a new value comes across the ticker's channel so the ticker Has a Field called C which is a channel where every kind of let's say that you know time between requests was set to one minute in that case every one minute a value would be sent across the channel so by using this syntax here we could say run this for Loop every one minute and the reason I'm passing in an empty initialize and an empty middle section to the for Loop is so that it executes immediately the first time so the very first time we get to line 17 the body of the for Loop will request will will fire immediately and then it will wait for the for the uh interval on the ticker if that makes sense if we just did this oops if we just did what is it for range ticker.c uh then it would actually wait for the minute up front but I want to do it once immediately it'll make it easier to debug and work with now at this point I realize that I've made a mistake the purpose of this concurrency parameter here is to you know indicate to the start scraping function how many go routines we want to use to go fetch all of these different feeds and the whole point is that we can fetch them at the same time so that means that each time that this ticker fires we need to be potentially go you know going it out to the internet to fetch 10 20 30 different RSS feeds and download all of their blog posts at the same time which means we'll actually need to be able to grab a a multiple number of feeds we'll need to grab more than just one at a time so rather than get next feed to fetch let's change this to get next feeds to fetch and we'll have it return many and then rather than limiting to one let's limit to dollar sign one so we can actually pass in how many feeds we want as a parameter to this function okay then we should be able to regenerate that and we should oh we're not even using the function yet so okay so that was actually the perfect time to do that okay let's fill out the body of this for Loop so every Interval Timer tweet request we want to go grab the next batch of feeds to fetch so we can just call that function that we just wrote database.getnextfeeds to fetch it takes a context and a limit so the first thing we'll just use context dot background so again I haven't gone into a ton of detail on the context passage but basically context dot background is like the global context it's what you use if you don't have access to a scoped context like we do for our individual HTTP requests okay so that'll work for now and then we also need to pass in a limit so we'll just cast into 32 and the limit or the sorry the concurrency and that should return some feeds and an error if there's an error we should probably print something now notice I'm continuing here that's because this function should always be running as our server operates like there's no time in which we want this function to ever stop so if I returned here that would be a problem it would actually stop scraping completely just because maybe I don't know our database connection was down temporarily so for now we're just going to log and continue now that we have a slice of feeds let's write some logic that goes and fetches each feed individually and importantly fetches each individually at the same time so we're going to need a synchronization mechanism I'm going to use a weight group so the standard library has this awesome thing called a sync.weight group then we can iterate over all of the fees so four feed range feeds okay so the way that the weight group works is anytime you want to spawn a new go routine within the context of the weight group you do a weight group dot add and you add some number to it so here I'm iterating over all of the feeds that we want to fetch on individual go routines and I'm going to add one to the weight group then at the end of the loop I can do a weight group dot weight and within the loop I can spawn a new go routine so we're going to go do some function in fact I guess I should just name it kind of what we'll be doing uh let's call it scrape feed go scrape feed and here we're actually going to pass the weight group in as one of the parameters and within scrape feed takes a weight group which is a pointer to a sink weight group within this function we'll defer a weight group dot done okay so what happens here basically we're iterating over the the all of the feeds on the same go routine as the you know the start scraping function so on the main go routine on the main go routine we're adding one to the weight group for every feed right so say we had a concurrency of 30 we would be adding 30 to the weight group now we'll be spawning all of these separate go routines as we do that and when we get to the end of the loop we're going to be waiting on the weight group for 30 30 distinct calls to done so done effectively decrements the counter by one right done decrements the counter by one so we're adding one every time we iterate over the slice and then we're calling done when we're done actually scraping the feed so what this does is it allows us to call scrape feed at the same time 30 times so we go spawn 30 different go routines to scrape 30 different RSS feeds and when they're all done line 35 will kind of execute and will move past that until they're like before they're done we'll be blocking on line 35 which is what we want to do because we don't want to continue on to the next iteration of the loop until we are sure that we've actually scraped all of the feeds so we've sort of stubbed out the scrape feed function right now doesn't do anything other than call weight group dot done let's actually go scrape some feeds so it's going to need access to a database connection and it's also going to need a specific feed to go fetch so feed is a data base dot speed cool and then here we can pass in database and great the first thing scrape feed should do and and keep in mind we're deferring the weight group dot done so this will always be called at the end of this function um the first thing we should do is Mark that we've fetched this feeder that we're fetching this feed so it's going to be database.mark feed as fetched we can just use the background context again and we need the ID of the feed so feed Dot ID cool that should return an error if there was an error I think oh it also Returns the updated feed I don't think we care about the updated feeds I think we can ignore that say if error it's not equal nil now keep in mind we're not returning anything from this function remember we're calling it on a new go routine so there's nothing to return here if there's an error instead we'll just log there was an issue and return nothing and next we need to actually do the heavy lifting which is to go out and scrape the feet so we already wrote Our URL to feed function let's just use that so URL to feed and we'll pass in the feed dot URL and we should get back an RSS feed and an error and if error does not equal nil error fetching feed and we'll return there otherwise we need to do some logging so in the future what we'll do is instead of iterating over all of the items in the RSS struct that we get back and just printing them to the console we'll be saving them into the database but for now just so that we can test our scrape feed function um we're just going to log log all of this to the console so we're going to log each individual post or rather that we found a post and then how many posts we found the last thing we need to do is go hook up this start scraping function to our main function so that it actually starts okay so start scraping takes database concurrency actually I'll just open this up my screen's too small to be working in two tabs at the same time okay um we're going to need to call it before listen and serve because remember this is where our server kind of blocks and waits forever for incoming requests so we should probably call it I don't know right here seems like a good spot so um it's just it's just a function right yeah it's not a method takes database concurrency and time between requests okay so we go go start scraping remember we want to call it on a new go routine so it doesn't interrupt this main flow because remember start scraping is never going to return it's a long running function this is an infinite for Loop okay it needs a database connection so we'll actually need to save this in a new variable so that we can use it in the API config and in the start scraping function next we need the concurrency um let's just start with 10. seems good and then time between requests let's do time dot minute perfect I went ahead and added a second RSS feed so let's go ahead and check the database and see what feeds we have currently so I've got Lane's blog and the boot Dev blog so there's two now remember when you're setting a concurrency of 10 so we should definitely be able to fetch both of these blogs at the same time on the first iteration of that Loop if we say deployed this to production and allowed users to start creating feeds maybe we'd get up to 100 200 400 different feeds in here then we'd only be fetching 10 at a time right just so we understand how that mechanism works but for now this should be good enough to test I'm going to update our logs just a bit so we can see which blog each post is from so found post item.title on feed .name okay let's go ahead and run that found posts the properties of pointers and go on feed boot Dev blog perfect kind of scroll up we should be able to see some the boot Dev Blog has way more blog posts than my personal blog here it is here's some Lanes blog stuff okay so that looks like it's working we should be able to move on to the next step now where we'll actually save these blog posts into the database rather than just logging the titles to the console we're going to need a new table in our database so let's start there we'll call it posts so the purpose of this table is to store all of the posts that we are fetching from all the different uh from all the different RSS feeds okay um what am I doing this is this is queries we need to start with migration so let's grab this we'll make it zero zero six posts dot SQL all right Goose up is going to be create table posts what goes in a posts table um we are going to need an ID to create that and update it that pretty much never changes um what else does a post have well it has a title right title um is that text Dot null that makes sense uh posts also typically have a description text I'm going to allow that one to be nullable I think it's okay if a post is missing its description hosts also typically have a published at date published at which is a time stamp um should we allow that one to be nullable no let's make that non null what else does a post have it has a URL the URL should be not null for sure um I'm also going to say that it should be unique unique I don't think we ever want to store the same Post in the database twice there's no point right if we have a post I don't see why we would need it a second time so let's let's go ahead and make that one unique and then lastly let's just put in a feed ID and the feed ID is going to be a uuid that uh we're going to want it to be a reference of references feeds ID right and that's going to be does that need to be unique it doesn't need to be unique but it should be not null we should always have the feed ID of a post and let's here put on delete Cascade if we delete a feed we'll Cascade and delete all of its posts and I forgot to put the type here so URL text not null unique okay that looks good to me let's go ahead and run our migration so it's going to be we need a CD into SQL schema and run goose postgres we'll need this connection string up perfect next we'll need a couple of queries to interact with this table the first one is going to be just a way to create a post so let's do posts and we'll do create oops free post return one thing and we'll just kind of be inserting a bunch of stuff I think let's take a look at the post table so we've got ID created at updated let me just grab this so I don't forget it okay so insert into posts ID created at updated at we also have we have a lot of stuff so I'm actually going to start spacing this out a little bit create it at updated at it's going to be a title and a description publish that URL and a feed ID okay and the values one two three four how many do we got eight five six seven eight returning star straightforward right insert into posts all of these fields no fancy logic that should be good okay let's run sqlc generate add that function to our internal package and then we just need to go use it so down in the scraper now instead of just logging all of these posts to the console let's save them to the database I'm going to leave this log message it just says feed blank collected blank posts found so that'll just log all of the different feeds we're collecting but each individual post I think it's wasteful and kind of busy to log everything so we're not going to do that instead we'll just call DB dot create post context.background and what does it take create postparams database dot create post params okay cool I kind of like how sqlc breaks down the parameters into um into a struct makes it pretty simple to work with okay um we've got an ID created that updated that okay ID is just going to be a new ID created that updated that that'll just be time.now dot UTC what's next title description idle item.title [Music] I just realized oh I just realized now that I'm not putting the field names kind of embarrassing okay title description do they not have a description what does an item have let's take a look at an item items have title link description yeah it does have a description what am I messing up here can I use item.description variable of type string as sql.null string ah right okay so we need to do sql.null string a null string has the string itself and whether or not it's valid so we just put in the string and then we say it is valid string dot valid true although actually this is a problem right this is a problem because if item.description is blank if it's an empty string we're going to be putting in an empty string and saying that it's there even though it's not so let's not do this let's do something a little different let's go description it's a new sql.nil string why is why is that giving me trouble oh because I'm doing it within the call to create post do it right here okay so we'll create a new sql.null string and then we'll say if item dot description does not equal the empty string then we get to set description dot string equal to item.description and description dot valid equals true and then we'll use the description here does that make sense so if if the item's description is blank then we'll set the value to null in the database effectively um otherwise we'll create the valid description entity okay next we need a published at let's see we've got an item dot Hub date which is a string okay so we're gonna need to parse that string to parse that date there is a Time dot parse function in the standard library and we're going to use this RFC one one two three Z layout so this is the layout that I'm using on the boot Dev blog and on my blog to be more robust and support all of the different publishing formats for all the different blogs that we want to scrape we'd probably need to make this logic a little bit more robust but for now I'm just going to say we're parsing it this way if it's not that way I guess we take a hike okay if there's an error so if error does not equal nil we can say log dot print line didn't parse date sent V with air Cent V and we'll pass in the actual Pub date and an error and that's going to be a printf oops cool and then if that's an issue we'll just continue so if we don't get a valid time uh then we'll just we'll just log it log in move on okay so published at pass in that I shouldn't use single name variables like this let's do Hub at okay what else do we got a URL and a feed ID so URL it's just going to be the item.link and feed ID so we have access to the feed ID here we do we have the feed right because we passed in the feed here e dot ID now db.createpost does return an error so we need to handle that error what am I screwing up here oh it also Returns the post itself I don't think we care about the new post though I think all we care about is if it failed so if error equal nil go to create post with error okay let's give that a shot okay build and run now remember we're expecting this time to get logs that just say that the blogs were collected so 21 posts from Lane's blog 321 posts from the boot Dev blog were collected I'm going to go ahead and kill the server and let's check PG admin so now if we select star from Posts you should see a bunch of stuff in here IDs created at titles descriptions published at dates awesome this is looking fantastic we scroll down to see how many rows there are 342 that looks right to me now I think we have an issue here let me show you what I mean if we run this again so remember we've scraped both of the feeds and pulled in all of the posts so if I rerun my server at this point yeah we're getting all of these issues failed to create posts duplicate key value violates unique constraint posts URL key right now this makes sense we didn't want to store duplicate posts in our database so we have a unique constraint on the post URL which means when we go try to recreate the posts it fails because we already have the posts in our database so let's do a little string a string detection so that we don't log this crap every time this happens because this isn't really an error this is expected Behavior so we can do something like if strings dot contains air dot error uh duplicate duplicate e then do we continue what are we doing here then we continue otherwise we'll log the air so we're only going to log the error if it's not a duplicate key error okay so let's run that again and make sure we don't get those errors perfect we have one last feature to add to our RSS aggregator we need a way for users to be able to get a list of all of the newest posts from the feeds that they're following so we'll need a new query we can call this one get posts for user and it will return many posts now let's think about this query for a second it's a little more complex than the other queries we've done in that I think we need to do a join so we have our posts table right posts have IDs created ad updated at but importantly they have a feed ID so we know what feed every Post in the database belongs to and we also have a feed follows table that tells us which feeds an individual or user is following so if we join those two tables together right if we take all of the feed follow information kind of join it to the posts table then we should be able to filter by all of the feeds that a user is actually following so what does that look like you do select host dot star so everything from the post table ROM posts join feed follows on oops on post dot feed ID equals feed follows dot feed ID okay so this adds essentially all the feed follow information to our like the virtual table for this query right we're joining those two tables together so now we should be able to filter um the way we want Okay so we've we've joined them together where hosts dot wait post no where feed follows dot user ID equals dollar sign one okay so we join the tables and then we filter all the posts down or rather the entire table down by the specific user ID so all of the posts that belong to feeds that the user is not following should at this step get trimmed out then we can order by let's do Post dot published at descending so give us the newest stuff first and we'll limit by a configurable amount so dollar sign too cool let's go ahead and try to generate that looks like it worked let's see if uh see if when we run the actual application it does what we expect let's hook that query up to a new endpoint so I'm here in the users file that seems like a reasonable place we'll do Handler get posts for a user it will be an authenticated endpoint so we'll need that that user data but here we're going to call DB dot or sorry API config dot DB dot get hosts for user and then we can pass in the request.context and user dot ID oh we also need I think a limit oh get no no no sorry we take we take database dot get post for user params right because we had multiple parameters here so we'll need the user's ID and a limit the user dot ID and for now let's just say a limit of 10. and that will return to us some posts and potentially an error if we get an error we'll respond with it couldn't get posts otherwise we need to return the posts themselves now we should go create a special posts model right so that we get our our own tag so type post struct and go mostly copy this from our internal host model wherever it ended up here it is okay that's Json tags goodness typing is hard now here's an interesting thing this sql.null string is not something that we're going to want to use in this struct because this is a struct that Marshals to Json the null string object is a nested struct so if we marshaled it directly to Json we would actually get description as a Json key and then string as a Json key and then valid as a Json case would be a little nested object there that's pretty bad user experience because Json natively supports kind of null in the sense that you can just omit the key or use like the actual value null so what we're going to do is instead do a pointer to a string and the way Json marshalling in go works is if you have a pointer to a string and it is nil then it will Marshal to what you'd expect in Json land which is that null value okay and then published at URL and feed ID okay did I miss anything let me look over this really quick that's looking good and next we need the conversion so we'll do database post to post now this one gets a little hairy right we probably need to do some logic here so we'll say bar description pointer to a string and then if DB post dot description dot valid then we'll set description equal to the address of dbpost.description.string then we can just directly use the description variable there all right published at what else we got URL and feed ID okay and last but not least we need a way to do it uh do the conversion for an entire slice bunk DB database posts to posts and the logic will look pretty much identical to that but I actually think it'll be easier to type it out so we'll do uh posts slice post database post post s in the database post okay there we go a lot of conversion logic there but now we should be good to just respond with some Json and we can pass in those posts as database posts cool what am I getting here struck literally uses unkeyed Fields oh yeah let's not do that we want an ID and I think it's a limit right what am I messing up user ID perfect okay let's build and run the server again oh I realize I made a mistake I need to actually hook this up to something so let's go back into main.go we need a new endpoint we'll do get um I don't know user feed now feeds probably a loaded term in this uh in this application we should say uh let's just do posts and it's going to require middleware so middleware off and API CFG Dot Handler get posts for user okay so get slash posts it's an authenticated endpoint Perfect all right let's rebuild and run that okay opening up thunderclient first we need to check well actually let's just grab some auth information so um this clearly has some off let's grab that API key create a new request HTTP colon vocal host V1 posts headers close that authorization API key okay so if I make that request I'm getting back no posts now we know we have posts in the database but my my user that I'm currently logged in as is not following anything all right I'm getting back the empty array when I when I check my feed follows um but I can check which feeds Exist by running this API request so let's grab this feed let's grab the Wags Lane feed and let's go follow that one so we'll post to feed follows here this feed ID okay so now I should be following let's check my feed follows great I'm following them so now if I go get my posts there we go I should just be getting posts from the wagslane.dead block perfect let's try following let's try following um the other one so where is it feed follows no feeds let's try following the boot Dev blog as well so post feed follows send that check my feed follows now I'm following both now if I go get my posts perfect we're seeing stuff from the boot Dev blog that's it thank you for sticking with me through all of this mess we've created an amazing blog aggregator that will actually work pretty darn well at scale you could run this thing uh you know over a long period of time collect millions of blog posts and it would do pretty well I hope you had a ton of fun with this project I do want to remind you that this is a server right we've kind of been running it stopping it restarting it but at the end of the day you can just turn it on ADD new feeds and follows and interact with it directly and it will once a minute go out and collect all of those blog posts so you could just keep this running on a Raspberry Pi in your house to aggregate you know blog posts podcasts all that kind of stuff um I will point out that we have done a bit of happy path programming so happy path programming is when you're not necessarily handling every Edge case out there you're you're handling kind of you know the thing that you expect to happen most of the time so for example um we only had one type of date parsing for the published at dates in our RSS feeds but maybe there are RSS feeds out there that use a different date format and will fail to parse them um so one way that you could extend this project would be to just add a ton of new RSS feeds and make sure that you deal with the issues as they come up make sure you improve the logging so you can see the issues when they come up anyways I hope you had a ton of fun with this project and that you learned something I just want to remind you that we do have an entire back-end learning path over on boot.dev in go lay so if you liked this project if you liked this course and are looking for some more content definitely go check out boot.dev we also published a lot of different ways that you could potentially extend this project to make it cooler for example maybe you add a front end or a command line application that interacts directly with the API so that you don't need to use a manual client like thunderclient every time that you want to interact with your posts and then I also just want to remind you before I go that you can find me on Twitter at wagslane or on YouTube at boot.dev definitely go subscribe to our YouTube channel as well thank you again to free codecamp for allowing us to publish this course and this project walkthrough I hope you enjoyed it and I'll see you in the next one

Channel: Boot dev

Views: 45,245

Keywords: learn to code, web development, backend development, learn programming, programming

Id: dpXhDzgUSe4

Length: 169min 34sec (10174 seconds)

Published: Thu Jun 01 2023