SEO in Next.js 14 - The Ultimate Guide (Metadata, Sitemap, Robots, Google Search Console, Caching)

cona in this tutorial you will learn everything you need to know about SEO search engine optimization in nextjs we will learn how to properly prepare our website for Google's search crawlers by setting static and dynamic metadata for each page by creating a Sit map and the robots txt file and by configuring the cache for each page properly so that our Pages load as fast as possible which is an important ranking Factor on searge engines we will also learn about some other little tips and strategies when it comes to SEO and in the end I will show you how you can connect your website to the Google search console and the zel analytics to follow along you should already know the basics of nextjs and the app router I have a full tutorial on that here on YouTube I will put a link to it into the top right corner if you are new to nextjs just watch this tutorial first and then you can come back here now a few of the things I cover in this tutorial are already covered in this beginner tutorial but I need to cover them here as well because they are important for SEO to follow along with this tutorial I already prepared a little starting project it's a very simple block that just has some different posts so each of these lines here is a title of a post when we click it we get to the page of the post now I put an artificial loading time in here so this takes a second to load this is on purpose because later we will learn how to R render and cach our pages in advance so that we completely skip this loading time in production the data for this block is coming from dummy jon.com it's just a rest API with some dummy data so that we can make a fetch request here and get some post that we can show on our website you can download this starting code from GitHub I will put a link to the repository into the video description just clone this repository open it in vs code then run npm install in the command line to install the packages and then as usual you can just npm run def to start this project in development mode and then you should see the same block as me again the link to the repository is in the video description please also subscribe to the channel if you haven't yet for more nextjs content and please also leave a like on this video if you like it and then I wish you a lot of fun with this tutorial okay and now we will step by step improve our starting project to make it more optimized for search engines the first step is to set the metadata for all our Pages because this is what actually shows up in the Google search results so the page titles the page descriptions and also the faff icon and so on let's start with the faff icon when we create a new nextjs project we already have such a fuff icon. file with the bzel logo this shows up in the tab of the browser but also for example in the Google search results so you usually want this to be your own logo or some kind of icon that represents your website there are different free generators online to create these faff icon files I usually use this one here I don't have any affiliation with them but this one just works so here on real F icon generator.net or just type f icon generator into Google we click on this generate button and then want to go into our project because I already prepared a logo that we can use in The Styling code in the srz folder and then inside assets there is this logo I created for the website this is just a normal PNG file we can select this and create a f icon out of it we get a preview how this will look we can do some additional configuration like setting a background color but we don't need any of this I just want to go all the way down and click on generate R your fa icons then we wait until this has finished and then we can ignore most of the stuff we just want to download the fa icon package we open the folder where we saved it in and in the zip file there are a bunch of different files but we actually only care about the fa icon these other ones are for example if you want to turn your website into an app and put it into the app store or Google Play store then there are variations of our logo for these stores as well but we don't care about them we only need this F icon I file so we copy this file and go into our project folder again srz app where already there is the old fa icon we delete this one and paste the new one in here now it still shows the old icon but I think this is just a caching problem let's open this file yeah and as we can see it's actually the correct new F icon so I guess Windows just caches these image previews and shows the wrong one and the fa icon has to be placed directly inside the app folder this is how it works in nextjs you don't have to set your own meta tag in the head of the website like you do it in raw HTML instead nextjs takes care of this for you but this only works if you put the F icon directly into the app folder this is where nextjs expects this file and it also has to have this exact name F icon. IO we can close the fa icon generator and if you refresh the page you should see your new fa icon in the top left corner sometimes this doesn't work right away you can try pressing contrl F5 to bust the cache and reload the page but as long as the correct F icon here is shown in the project this will be set correctly and it will work in production next I want to set the open craft image that's the image that's shown when you paste a link to your website on social media Twitter for example or Facebook then you can add such a preview image that will be shown together with the link it's recommended for this image to have a width of 1200 pixels and a height of 630 pixels but this is just a normal PNG file and you can create this file with any image editing tool I like to use gy for that it's a completely free alternative to a Photoshop it's really powerful but you could also use Microsoft Paint for example if you want I will put a link to this OG image into the video description as well you can download it from here and again we have to follow a certain naming and folder convention for this to work so the file has to be called open graph minus image in order for nextjs to detect it and again we want to put this directly into the app folder so we open the app folder where our fa icon is already in and we take this open craft image and paste it in here and then this will just work again under the hood next JS set some tag in the head of the page we will later take a look at this but this is all abstracted a way for us to make this process simpler this open craft image will now be used for our whole website for all pages later we will also learn how we can set specific OG images for specific pages and we can also change this OG images dynamically for example to use the thumbnail of a blog post as the OG image for this page but more on that later then we go into the root layout. TSX file as we learned in the nextjs beginner tutorial this root layout wraps our whole application and this is the best place to set base metadata for our website so this is the metadata that will be used if we don't set more specific metadata on other Pages we do this by exporting a const called metadata with this exact spelling and we set a type to metadata which is a next import because this way we get autoc completion in here again this is an abstraction provided by nextjs under the hood this generates specific Texs in the head tag of the website we will take a look at this in a moment and for all of these different tags there are options available in here but we don't need all of them important is of course the title of the website because this will be shown in Google and also here in our browser tab so this is set to create next app by default now we can change the string in here but we can actually also pass an object with curly braces and then we get some more granular control so for example we can set the default to a string I'm going to write my awesome block because this is the title of this website this default will be used used when we don't set a more specific title on another page this has the same effect as just passing this string to the title directly but with this object what we can also do is we can define a template which is a really cool feature because this way we can interpolate our default string with a more specific page title for example I want the root URL to say my awesome block but when I open a blog post I want to have the title of the blog post and then a dash and then my awesome block behind it and instead of repeating this pattern for every single page we can use such a template string so we write dollar sign s this is where the title of the child page will be put in and then here I write Dash and again my awesome block or whatever you want so this part here is completely up to you and then later this part here will be replaced with the title of of the child page we will see this in a moment let's also change the description of the website this one is not shown in the browser but this is shown for example when we post a link to our website on social media so this is a really important piece I'm going to change this to come and read my awesome articles write whatever you want a lot of other meta Texs are automatically inferred from these tags here for example there is a TW Twitter description tag where we can set a specific description just for Twitter but if we don't Define one then the default description of our page will be used for Twitter the same for example with the open graph image which will automatically be used on Twitter and Facebook LinkedIn and so on if you want to set more specific metadata just use aut complete and somewhere in here should be what you're looking for for example we have this Twitter option which expects an object and here as I already described we can set a more specific description but we don't need this but one value I usually set in here is card and I set this to summary large image this defines how our open graph image will be shown on Twitter because it can be shown as a small image or as a large image we want to show the large image which just looks better in a minute I will show you how you can check how the social media previews for your website will look without having to deploy your project first but before we do that I also want to set the metadata for some other Pages next let's open the page. TSX file in the about folder this is the about page and there is a link to this page here in the pH as I explained this uses the default title that we set in our layout TSX so this is the fallback basically but we can set a more specific title for this page for this we go above the component and again we export const metadata make sure the spelling is correct with the same casing that's important and we set the type to metadata from next and now we can set the same values in here for example the page title I want to set this to about and as you can see this title is now put into our template string here and it looks like this about- my aome block this way we don't have to repeat D my awesome block on every single page it's just more convenient now on some pages you might want to not have this Dash my awesome block and overwrite the title completely you can do this as well by again passing an object here and using absolute instead when I said this to about the rest of the template disappears but we can still keep it in our root layout which is really useful but I don't want this here I want my template and then let's copy this into the Privacy page as well here at the top again outside of the component change this to a privacy policy and we have to import a metadata type and this is this privacy policy page which I prepared and as I already explained nextjs automatically creates certain meta tags in the head of the website and we can see them by opening the Chrome Dev tools you do this with F12 then we go into the elements tab here we can expand head and here we can find all the meta texts generated for this page for example here is the description tag with the description we set up in our code our Twitter card setting Twitter title is generated automatically it's it's all in here and somewhere in here is also our open craft image that's this one and this points to our open craft image file now there are certain websites we can use to check our social media previews like social share preview.com the problem is of course our website is running on Local Host it's not actually on the internet so we can't paste a link there so we either have to deploy our project to a check this or we can use something like sr. us which is a way to run your Local Host website on the internet just temporarily for testing purposes there are also other tools that can do the same but what I like about this one is that we don't have to sign up for anything we don't have to install anything we just need to copy this command and run this in the command line you might have to set up SSH first for this to work but if you can't get this to work this is also not a problem you don't need to follow this step I just want to show you the previews of our website but you can skip this step if you want so this is my temporary URL here I can copy this open it on the internet and there is my awesome block and this is actually running on the internet with an https connection isn't this cool and we didn't have to install anything and now we can take this URL and pull it into this social share preview Checker now sometimes this tool hangs for some reason it doesn't load the OG image there's also another similar tool called open craft. XYZ here I pasted the same URL and here we can see our preview with the page title the description below and the OG image let's try slash privacy which shows the same OG image and description because this is the fback data that res set in the root layout but we see our page specific title nice later we will learn how we can set this metadata more dynamically from the data of the page itself for example you might want to use the title of the blog post as the page title or the thumbnail of the blog post as the OG image you can do this as well more on that in a moment for now we close this and I want to quickly talk about a few other things in the rud layout a new nextjs project automatically uses this Google font imported from next SL font gooogle this is actually pretty useful for SEO because these fonts are locally hosted on our own server on our own website whereas many other websites load Google fonts from the Google servers directly however this is actually bad for user privacy and in the EU it's actually kind of illegal because when you load a font from Google Google the IP address of your website's visitor is actually sent to Google servers and this is illegal without the users's permission because the IP address is considered personal data and you can't just send this around willy-nilly however we avoid this problem by using these local fonts because again they are stored on our own server and they also load faster which is great for SEO because we don't need its connection to Google this is why you should prefer them and whenever you use images somewhere in your nextjs app you should use the next image if possible this is this uppercase IE image and it's an import from next SL image the biggest benefit of these next images is that nextjs automatically resizes them to the size in which they are actually shown on the screen so even if the image file is 2,000 pixels big but you only show it with 100 pixels width and height then nextjs automatically resizes this image file for you if you use this image component which is very important for loading speed which again is important for SEO we don't have any images in our practice project here but we use many different images in my other nextjs project tutorials you can find them here on my channel I'm going to remove this image here because we don't need it and then let's continue with Dynamic metadata often this static metadata is not enough because we can only hard code strings and images in here but we can't change this dynamically for example I don't want to show the exact same title for every blog post instead I want to show the title of the blog post in the page title so let's open the blog post page we can find this under post id/ page. TSX here in this folder again we go above the component by but instead of exporting const metadata we export an async function called generate metadata and again make sure the spelling is correct with an uppercase M and all the other letters in lowercase again this function will return metadata but since it's an Asing function we have to wrap the return type into a promise so this returns a promise of type metadata again then by setting the type of the function explicitly we get Auto completion in the return block for example our good old title and description in the page component itself down here we fetched a block post from the API and then show it in the layout we need the same data the same block post also in here to generate our metadata but we can actually not share data between these two functions instead the way this works in the the nextjs app router is that we have to fetch the same data and generate metadata again and for this we get the same URL param that we also receive in the component this URL param is the post ideal I explain this in my next JS beginner tutorial it's this path up here in the URL that's different between different block posts when I Chang this to four we open a different block post that's this post idea and we use this IDE to fetch the blog post from the API and and the generate metadata function can receive the same arguments as the page component itself so we copy this whole part here and paste it up here and now we can fetch the same block post with the same post ID in generate metadata and this will be executed before the page is rendered and then we want to fetch the same data so we copy these two lines above the return statement and generate metadata I I don't want to destructure the post here so I change this to post and then we can simply use the post data to generate our metadata for example we can set the page title to post. title we can set the description to the body of the post for example or a summary what exactly you put in here is up to you and our blog posts don't have any featured images but if they had we could also set the OG image the open craft image of each block post dynamically we can do this by adding this open craft value which receives an object in here we have images which receives an array so scrap brackets and in here we can put an object that expects the URL key and this could for example point to a post. image URL now again our blog post don't have an image URL so this will not work but let's imagine we had a thumbnail for our blog post then this image would be used as the OG image for this page and it would overwrite the open graph image PNG file but since we don't have this value I will comment this out and just leave it in here as a reminder and when we save this we will already see the title of the blog post here in our tab to together with our Dash My aome Block template cool but there are few things I want to clarify first of all you might be wondering isn't it wasteful to fetch the same data twice in the page component and in the generate metadata function especially if we render this page dynamically every time the user opens the page without caching it then we would make two requests to the same endpoint to render one page however nextjs actually automatically duplicates fetch requests so when we make a fetch request to the same URL it doesn't matter in how many places we do this request it will only be executed one time when the page is rendered so since these two fetch requests have the same URL there will only be one fetch request executed and the data will automatically be shared between these two functions and nextjs does this automatically but only if we use fetch if you use an OM like Prisma for example or if you use exos instead of fedge then this automatic D duplication does not happen then you have to do it yourself and this works the following way you create a separate function you can put it in here or into a separate file depending where you need it we store this function in a variable called for example get post the name is up to you and then we have to call a function called cache but autoc complete doesn't work right now so I import this manually this is an import from react so this cache function belongs to react itself it's not part of nextjs or Library it's part of core react and this cache function does the same thing I just explained it D duplicates these requests so that they will only be executed once within the render cycle to this cache function you have to pass the function that you want to Cache so for example we could put an async function in here that expects the post idea we make this an error function function and in here let's say we retrieve our post from Prisma now I'm not using Prisma in this project so I will comment out this code in a moment I just want to show you how this would work for example Prisma dopost do find unique to which we pass the post ID then we would return the post from this function and then when we fetch the post we would call await get post here and also down in our component and again this way we D duplicate these requests so that they are only executed once but again this is not necessary if you use the native fetch function so again I will comment this out just leave it here as a reminder I have a bunch of different tutorials where we use Prisma and there you can see again how to do this exactly I will just leave an explainer comment here manually the D duplicate requests if not using fetch so you can take a look at the code later and remember what this was for it's also important to note that you can only export this generate metadata function and also static metadata from a server component you can't export this from a client component so if you need interactivity on this page if you need JavaScript features like onclick listeners here then you can't make this component a client compon component because then this part here will not work anymore instead you have to extract the page content into a client component so let's say for example we want this whole page to be a client component for some reason because we need a lot of JavaScript here then we could cut out this part create a new file called for example blog post page make it a client component paste the code that we just cut out and then we render this blog post page comp component this client component here in our server component this way we can use client component features in here but we can still export our metadata okay but I don't need client component features here so i r this instead of setting a specific OG image here in this metadata return value we could also put another open graph minus image.png file into this folder and again it would replace the one in the root folder but just for this page but usually you want to set a dynamic OG image depending on the post data and this is how you do it however there is one more cool alternative we can also generate an OG image from a TSX file so a component this is how it looks you can see the instructions in the documentation you can put diffs and styling in here and you can generate an OG image dynamically and again you can also fetch data in here like the blog post now this is beyond the scope of this tutorial but I can show you an example of this from my personal block there I use these open craft image TSX files so for example here I have this blog post and this is the featured image right and the title of the post is five AI tools I can't live without as a developer when I copy this URL and paste it on social share preview.com this is how it looks so the page title is put on top of the OG image with this dark background to make the text more readable and this is especially useful on X or formerly Twitter because they don't show the page title in the preview anymore they only show the image and the URL and there it's really useful to have the title layout on top of the image again how exactly you do this is described in the nextjs documentation if you want a detailed tutorial on this let me know in the comments below okay this is all about metadata next we will take care of caching our Pages properly so that we get rid of this loading time because a long loading time is terrible for SEO okay the easiest way to find out how our pages are cached is to just build the project for this we stop the execution of our development server and then we type in npm run build instead of runev we execute this this takes a few seconds depending on the size of the project and what kind of potato machine you're running it on but after it's done we get this output and we have a list of our different pages and these little icons to the left of them as you can see down here these icons tell us how each page is rendered and cached now whenever possible nextjs automatically caches a page so it's rendered when we compile the project and every time a visitor opens this page they get sent the readymade HTML and we don't have to wait for anything to load anymore and we can see this effect on the front page but first we have to execute npm Run start to run the project we just built then refresh the page and on the front page I also have this one second delayer but when we build the project nextjs renders this whole page it fetches the data it generates the HTML and whenever we then open this page we get just sent the HTML and we don't don't execute this part here again including the delay this is only executed at compile time and this is why we don't have our loading time anymore I can refresh the page as often as I want we don't get this 1 second delay anymore and if it was 10 seconds we wouldn't have this delay either this page loads instantly this is one of the big strengths of nextjs it caches these Pages for you so they open as fast as possible but when we open a single block post we still have our loading time of 1 second why because we have a dynamic URL param here right at the moment we are on post 4 but we can also go to post 20 which has different data the problem is nextjs doesn't know in advance what different values we can put in here this is why the data for this page has to be fetched when we actually open the page so this code here will not be executed at compile time time it will be executed when someone visits the page including the 1se second delay and all the loading time and when we take a look into the console again we can also see that this has this Lambda icon here which means this page is dynamically rendered whenever possible we want to avoid Dynamic rendering we can't always avoid it sometimes we need Dynamic data when the page is opened but statically cached pages are much faster to load so they are also better for SEO again the problem is that nextjs can't know in advance what different post IDs we can put into this URL so it can't render these Pages at compile time however we can tell next JS what values we can put into the post idea and then static cashing will work even for our blog post for this we have to implement another function and again I already covered this in my next J beginner tutorial but for completion I also want to cover it here so somewhere in the same same file outside of the component we export another asnc function called generate static params again it's important that the spelling is correct and from this function we want to return an array that contains all the different post IDs that we want to render in advance so how do we get these post IDs well we need to fetch all our posts the same as we did on the front page right so we copy this part and the second line as well paste it in generate static params also import this type here from the models folder and again this fetch request will automatically be duplicated with this one here but it would also not really matter if we execute this twice because this will only run at compile time one single time so it's not a big deal to execute one more request so now we have each post and now we want to turn this into an array of post ideas so we return post. map parentheses another pair of parentheses and then a pair of curly braces because this way we can destructure this post object in this map call and in here is the idea of each post then here between these two closing parentheses we make a right arrow and right idea this way we map the ID of the post into an array of IDs which we return from this function so this array will look something like this one for post one two for post two and so on so this is literally the array re return from this function this tells nextjs that we want to pre-render all these different post IDs which will then be put into these URLs the data for the page will be fetched and then the HTML will be cached so we can open it much faster so let's save this and let's build our project again again npm run build and as you can see now we don't have this Lambda icon here anymore now we have this fil Circle which means that this page is also pre-rendered and statically cached but by using our generate static params here they say get static props this was from the old Pages directory but in the app rouer we use generate static params instead the important part is that we now see this circle here and when we now start our app again our block post should load instantly just like our front page so no matter which post I click boom it opens instantly because again all the data is fetched at compile time and then we just get served the static HTML this is especially useful for something like a block because it's usually enough to fetch this data at compile time and it's really really great for SEO because page loading speed is an important ranking factor and nextjs makes it possible to open your pages pretty much instantly to see another example of this let's take a look at my blog again I do the exact same thing here Boom the page opens instantly now the images still need a moment to load but that's fine because they are lazy loaded anyway but the blog post itself all the text is there instantly there's very Lally zero loading time which is really amazing now one more thing I want to mention here sometimes you have a lot of pages like hundreds or thousands for example on an e-commerce website where you have thousands of different products and you might want to have them statically cached but you can't render all of them at compile time because maybe you don't want to fetch the data for thousands of products because that would overload your server now the cool thing is when you implement G generate static params and we open a blog post with an ID that we did not return from this function the page will then be rendered when a user opens it for the first time so we still have our 1 second delay the first time this page is opened but then nextjs will go ahead and cash this page for any user that visits the page after it but this only happens if you implement generate static params in the first place if we don't add this then we will always have our loading time we only get this fallback Behavior if we implement this function and return something here to see how this works maybe let's only pre-render five of these pages I do this by calling slice and now we basically only put the first five post IDs into this array right let's also increase the loading time just for a moment but I will reward this later this is just for presentation set this to 6 seconds then we execute npm run build again followed by npm Run start and now when I refresh the page only the first five pages should be pre-rendered and cached so they should still open instantly and they do but when I open a post that was not preener and I want to open it directly via its URL for example post 30 which would be the last one then we have a loading time of about 6 seconds before this page opens because again we don't return this idea from generate static params anymore so it's not pre-rendered but as I explained as soon as we implement this function this page will still be cached after it was rendered for the first time so when we refresh the page it actually opens instantly we don't have this 6sec load loading time anymore let's try another one 29 again we have a very long loading time in which nextjs fetches the data renders the page but then caches it so now if we go back to a thir here it opens instantly and the same for 29 so only one user has to wait until the data is fetched and every user afterwards gets this very fast instant page loading speed this way you don't have to render all your pages in advance which again is very useful if you have a ton of them like thousands then you can just render the most important ones or the first 100 or whatever and then use this fallback strategy for the other ones okay but I'm going to reset this delay here and remove the dot slice to render all pages I'm also going to start this in development mode again so we can see our latest changes npm runev one more thing when we open a page with an ID that doesn't exist at the moment we get an empty page which is not great for user experience because it's confusing to change this we go down here into our component function and since we make a fetch request we can check the result of this request we can check if the status of the response is equal to 404 because this is what the dummy Js API gives us back if this post wasn't found if you use something like Prisma for example then you can also check that your array that you get back is empty we just want to know that we didn't receive any data here then we want to call not found which is an import from next SL navigation and this will take care that we get redirected to the not found page which I already included in the starting code again I covered this in my next as begin inner tutorial this file has to be called notus found. TSX and this is where this not found function will redirect us to if this data for this page can't be found and if we insert a URL for which we don't have a route we get redirected to the same not found page you want to make sure that you redirect the users to the not found page properly because if we render an empty page then this would be a pretty bad signal to search engine crawlers because they don't find any data and they will probably think that our website is useless instead we always want to give the search engine crawler the correct signals if we didn't find some data we have to tell it and this not found page also has the appropriate headers and Status codes to give the correct information to the search engine crawler so the crawler knows exactly what's up next I want to talk a little bit about server components versus client components I explain what server and client components are in my my next JS beginner tutorial but they also have implications for SEO generally you want to fetch as much data as possible inside server components this is what we are doing here we don't have to use client directive at the Top This is a server component this is why we can fetch data directly inside the component and down here we render it and this is great because this way all the JavaScript and all the packages we need to render these pages will not be shipped to the client they will only be used on the zver to render the page and then a finished HTML will be sent to the client now in this example we don't use any packages to render our Pages because they are so simple but for example we could have a markdown format in here as a third party package but by using this markdown renderer in the server component we don't need to send the JavaScript off this package to the client we only send the finished HTML to finished markdown but server components don't support any JavaScript features we can't do any interactive things on this page at least not anything that requires JavaScript for example button on click listeners don't work in zver components for this we need a client component but the trick is to make the client components as small as possible so let's say for example we want to put a button here on this page that we can click to clap for this post like on medium.com which is similar to liking a post just as an example here now instead of wrapping this whole page into a client component so that we can use our button in here we can instead just wrap the button itself into a client component and put it into the server component for this I have prepared this clap button in the starting code which contains the button and the JavaScript logic and also the state and again State can only be put into client components not into zver components but by putting the use client directive directly here into this button component we can put it just like this into our server component without having to make this whole page a client component it's enough if the interactive part is a client component and this is much better because then all of this stuff here is still rendered completely server side and when we look over here so this part is rendered on the server and our CP button down here is rendered on the client this is why it supports JavaScript feed Fe es okay cool and next we will learn how to set up a dynamic sit map and a robots.txt file which are also important for SEO the S map is a special file with which you can tell search engine crawlers about the different pages of your website that you want to have indexed in Google or any of the other search engines like Bing they all read from this sitemap now having such a s map is not mandatory these crawlers can also just crawl through through all your links and find your pages this way but this way you make sure that the Crawlers don't miss anything and it's especially important if you have Pages where you don't have any links to these pages on your website because then crawlers can't find them at all unless you have such a Sit map long story short it's a good idea to always have a sitemap on your website if you care about SEO in next JS there are two different ways to create such a sitemap we could create a zap. TSX file and write the S map in here directly this is the syntax of these s Maps has this URL set and these different URL Texs it's an XML file but usually you want to generate the S map dynamically because we don't want to put every single page in here by hand right if we have many different block posts then we want to just fetch these block post and generate these different entries automatically and we can do this with a dynamic zit map for this we create a zap. TSX file in our our app folder so let's do this let's go into our project right click on the app folder and here we put a zit map. TS file not TSX because we don't put any layout in here and the name has to be exact again otherwise this will not work sitap dots in our lowercase in here we export a default async function which again we call sitemap and our lower case it's an async function because we want to fetch our block Post in here and again to get proper autocomplete we set the return type to a promise of type metadata route which is coming from next dot sitemap this is the return type and in the end we want to return an array with all our different pages and for each page we add the JavaScript object and now we get autocomplete in here these three values here are optional the one we have to put in here is the URL of the page for example let's say we want to index the about page then we can put this URL in here and it has to be the full URL we can't just write slab this is not enough it has to be the full URL so we need the base URL in front of it and I put the base URL into the N file which is part of the starting code next public base URL so here I want to put process. nf. nexu base URL slab this is our first sitemap entry and when we save this we can already open our sitemap make sure your app is running in development mode so we see the latest changes sitap dot no. XML actually that's the file type and there it is that's the first you URL in our sit map but as I explained I also want to add entries for every blog post but I don't want to add them manually instead we want to fetch these block Po and then just map them into this array again we want to fetch all block post so we copy our fetch request from the front page in here import this type and then we want to map these post into the same U structure down here so let's create a const post entries and I give this the same type metadata route. sitemap because it will also be such an array and this way we get better autoc completion and type safet here and recreate this from posts. map again two pairs of parentheses and a pair of craes to destructure the ID as we did before parentheses curly braises because from this we want to return an object and in here we now get autocomplete because we set this type here again only the URL is mandatory again we want to start with the base URL and then do slash and our blog posts are under slash posts slash and then the ID of the post if we know the last time we updated a blog post it also makes sense to set the last modified value this tells the crawler when you last updated the page so it can crawl it again but if you set a wrong value here like setting all pages just to the latest date for example Google will actually ignore this value completely so it has to be correct now we don't have an updated Tim stamp in our blog post but let's say we had one then we would turn this into a date because usually the Tim stamp is stored in form of a string for example post. updated at and the last modified value expects a date so we turn this into a date object but again we don't have this value here so I comment this out and just leave it here as a reminder and we could also set a change frequency for which you have these different options how often you change this page I don't set any anything here again I just leave this as a comment as a reminder and we could also set a priority here which I think has to be a number yeah and this defines how important this page is compared to the other ones I've never used this value anywhere I usually just use the URL and maybe the last modified Tim stamp but if you want to implement this as well you can do it in here and then we want to put these post entries into our array down here so we can just spread them directly into it like this let's also put the last modified Tim stamp in here just to see how this looks in the generated s map I'm just going to set this to the latest date but again if you actually use the latest St for these timestamps for all your pages then Google will just ignore them because Google is smart enough to notice that this is not the correct value nevertheless let's save this refresh our sit map and now we have our dynamically generated block Post in here all of them and we don't have to add them one by one and later I will show you how to upload this sitemap to the Google search console so make sure to watch this tutorial all the way to the end next I want to set up the robots txt file with this file you can tell search engine crawlers which pages to ignore because sometimes you don't want certain pages indexed again we can hardcode it in static form by just putting a robots txt file into the app folder or use robots TS to generate it dynamically and even if we don't fatch any data here I actually like this syntax more so let's create the robots. TS file in our app folder so again right click on app robots. TS and again the naming has to be exact from here we export a default function called robots again in our lower case and we return metadata route do robots we don't wrap it into a promise this time because I don't want to fetch any data here but you C if you want and from here we return an object and in here we can set rules host and S map let's start with the rules which is an array in Array of op objects and again we get autoc complete here with user agents we specify which crawlers we want to Target if we want to Target all craws then we put an srisk in here in form of a string you can also be more specific and for example only exclude Google crawlers from certain pages but here let's say we want to exclude certain pages from all crawlers we start by allowing all pages with just a slash like this because we want most of our Pages indexed we just want to exclude a few of them which we can do with this allow which gets an array and let's say we want to exclude SL admin so this doesn't show up in Google and let's say SL privacy for example we still want these pages on our website and users can still open them we just want them to not show up in Google and after the rules we can also set the the zit map the location of the zit map which again is process. n dot let me copy the name next public base URL slash sitemap.xml right this is where the crawler can find the sit map to be honest I don't really know what the point of this is we can also submit our s map later without this robot txt file but I assume that this just helps the crawler find the S map if we don't submit it ourselves this is what I would guess anyway let's save this and then we should find it under slash robots do txt and there it is let's also put a slash in front of privacy to keep it consistent I think it actually doesn't matter if you put a slash on here or not but this way it's more explicit so such robots txt file is not mandatory but if you want to block pages from search engine crawlers completely then you need one so again we say okay listen all crawlers you are allowed to crawl all pages except for the admin and the Privacy page and this is where you can find the sitemap you could also make this function async wrap the return type into a promise and fetch data in here if you want to generate these entries dynamically but here we don't need it now when we go to the metadata of a specific page let's go to the Privacy page we also have this robots field in here this is very similar but it gives you more fine grain control for a specific page so right now the Privacy page is completely excluded from crawling this means that unless we remove this string here whatever we put in here doesn't really matter because the crawler doesn't even get to the page but I still want to show you the different options you have in here again we can use autocomplete for example we could set index to fults and follow to true if we removed this string here we would still allow the crawler to craw this page but with index false we say we don't want this page to show up in Google but with follow true we allow the crawler to still follow links on this page so if we had any links on our privacy page which we have because the header is a link and we also have for example the about page here then the crawler is still allowed to follow these links and index these pages so you have all this fine control with this robots value here and you can learn more about this by just Googling for robots metat tech there you can see all the different options you have here but again as long as we this allow crawling for this page Al together then this stuff here doesn't really matter because the crawler does doesn't even get to this point I still want to leave it here as a reminder if you care about SEO it's a good idea to connect your project to the Google search console because here you can monitor your Google traffic you see how many clicks you're getting and if there's something wrong with a page then Google will tell you about it you can also see what people have been searching for now this is a new website where I don't have any blog posts on but I can see that people have been searching for smart d and these different variations my name and so on so you have all this data in here it's pretty useful and it's also free to use so just go to search. google.com/ search minus console or type Google search console into Google and you will get here and to add a new website you can click up here and then on ADD property and here you have to insert the URL but without the https without the WW and any trailing slashes just like this my awom blog.com as an example then we click on continue and then you have to verify that you actually own this website the way this works is that you have to copy this value here then you have to go to the provider where you bought your domain this could be Vil but for a smart diary for example I used name cheep and here you have to find the DNS settings you have to check the documentation of your domain provider to see how to get there on name chep we do this via the advanced DNS settings here and we want to find something that looks like this with these different records inside it and then you have to add a new record of type TX tier for the host you just type in an ADD and then you paste the value we just copied from the console then you create this record I don't do this here because this is not the actual domain and after you addit this record you click on verifier then it will check if it can find the record and if yes Google knows that you actually own this website because you have access to the DNS settings and after this is done your website will show up here in this list and then you can start monitoring the data now it usually takes like two days or so until the data shows up but then it will update daily and you can see your search performance and the Google search console is Al where you should submit your sit map so here in the site bar somewhere we have this sit Maps menu option and here you paste the URL of your sitap so it's your websites URLs sitemap.xml you submit it and then Google will automatically check it again from time to time now it will take a few days until your pages start showing up in Google in order to check which pages are already indexed you can type this into the Google search but with your own on URL site column in this case it's smart diary. and this way we find all the pages of smart di in this case it's just the front page and the sign in page because I don't have any other pages but if you have a block for example and this block post are indexed then you should see these Pages here as well and there's actually a trick to get your pages indexed faster you can submit them manually via this inspect URL field up here you have to do this for each page by hand so you can't just pass a large list to it that's the downside you can also only do it for like 10 or 12 pages per day but from my experience this indexes these Pages much faster you will insert the URL you press enter and then if the page is already in Google you see this check mark which means the page is already indexed then you don't have to do anything if the page is not indexed yet then you can click on request indexing and you can do do this for any Z URL for example let's say if I had a blog post blah blah then I could Index this page via this input field and then usually it's indexed within a single day just as a little tip if you are in a hurry and if you host your project on Vel and you have the Vel Pro Plan which costs 20 bucks per month then you can also use Vel analytics for free which is a simpler and easier to add alternative to a Google analytics as you you can see it's also privacy friendly which Google analytics is not and to activate it go to your project open the analytics tab click on next in the Pro Plan it's included until we hit a certain threshold and then you click on enable and then you get some very simple instructions for some code you have to add to your project I don't want to go through these steps here because it's very simple just follow these instructions and then you can see in real time who visit your website at what time what pages and so on it's really powerful if you want to monitor the traffic to your website I hope you enjoyed this tutorial on SEO in nextjs one more thing I recently created this free react best practices mini course for my email newsletter you can get it for free under coding andf flow.com react bestest practices I will also put a link to this into the video description here you can join my email newsletter and then you get this free email course where you get one email with one small video Lesson for a couple of days and you also join my newsletter which I highly recommend because I send regular coding tips with new stuff I learned to it it's of course free and you can unsubscribe at any time so check it out leave a like on this video If you haven't yet subscribe to the channel for more nextjs and debde tutorials and then we see us in the next video Happy coding take care

Channel: Coding in Flow

Views: 3,622

Keywords: next.js 14, nextjs 14, next.js 14 tutorial, next.js app, next.js app tutorial, next.js app folder, next.js app dir, next.js app router, next.js seo, next.js seo friendly, next js seo meta tags, next js seo example, next js seo head, next js seo guide, next.js sitemap, nextjs sitemap xml, next.js dynamic sitemap, next js dynamic sitemap, next js dynamic routes, next.js robots, next.js robots.txt, next.js google search console, next.js vercel analytics, next.js 14 seo

Id: wTGVHLyV09M

Length: 62min 14sec (3734 seconds)

Published: Tue Dec 05 2023