Web Scraping Full Course 2024 | Build and Deploy eCommerce Price Tracker

have you ever wondered how websites know exactly what you talked about a few hours ago are they watching your every move well in a way yes they suggest products to buy analyze their competitors and monitor their Brand's reputation Big Shots like Amazon eBay Twitter Instagram and even your new favorite Chan gbt are all into web scraping they're using this trick to make their products even better if not the absolute best but hey why should you care well imagine truly understanding how all of this works under the hood and using it to your advantage picture yourself creating software that monitors the actions of platforms like Amazon notifying your users precisely when it's the optimal moment to make a purchase you wouldn't only develop software that Wows future employers but a handy tool that saves you money that's exactly what we'll do in this video in the JavaScript Mastery Style you'll learn what web scraping really is and then build a tool to leverage it who knows one of you might even take it to the next level and turn it into a software as a service business that others would want to use and pay for even Mr Beast scrapes the web we just paid someone to sit there and refresh and then book it for when it came out so like someone would cancel six months out yes but hey there has to be a better way to do it right we're developers we can surely figure out a more efficient solution what about a programmed scraping bot that can jump over all the most common obstacles developers face while scraping such as handling captchas anti-scraping measures IP rotation and anything else that may require human intervention unless most developers out there you will know how to build it so let's dive into the world of web scraping so I can tell you more about website Crawlers what libraries you need to build a web scraper and how to do scraping in the smartest way possible and immediately after develop a nextgs 13 scraping app that easily leaps over all Dimension scraping obstacles an application that perfectly combines two best Concepts next.js and scraping with a simple yet sweet landing page showcasing a nice header with a carousel a surge bar for scraping the products and a section showing so far scraped products all we need is to provide an Amazon product Link in behind the scenes we'll scrape the details such as product image product price and a special section that we calculate showcasing the average highest and lowest price that this item has reached so far then there's a details section showing product descriptions scrape from Amazon and finally a section displaying similar scrape products to discover but there's more than what meets the eye we'll Implement a special track option a model where you can provide your email address and then it will track that product for you first it'll send a welcome email showing how the Futures emails will look like for example if a product that wasn't available is now back in stock this application will send you an email to notify you the moment it's back and many more use cases such as notifying you when the product reaches its lowest price ever and will this process be manual well not at all we'll use a special programming technique called cron it's a special technique that runs the programs automatically at specified intervals imagine a Cron job is like a magical timer for your computer you can tell this timer hey do something for me at specific time every day for example you can say at eight o'clock in the morning I want you to open up my favorite game the timer will remember that and make sure to open your game every day at 8am just like magic it's like having a little helper who does things for you automatically when you want them to without having to do it yourself cool right and that's exactly what I will teach you the trigger are Scraper on a periodic basis so we'll keep our application up to date with recent data coming from the original Amazon website and then depending on the conditions we'll send emails to the user I hope that sounds exciting as these things are rarely taught on YouTube and we are about to explore all of them and combine them into a functional application so without any further Ado let's get started with the first part of this video which is what's web scraping in the first place if anything is going to play a significant role in this era and Beyond it is Data photos you share on Instagram tweets and even this video you're watching right now are all data having access to large amounts of data is powerful it is this very data that openai and the myth Journey used to train their models and we all know how impressive the results are as the amount of data grew really fast smart people saw an opportunity to use it to make their businesses better and quickly web scraping came into the picture have you ever copied stuff from a website and written it down somewhere like on a piece of paper or like in a text file or Excel well you've just scraped the web web scraping means taking information or data from websites but instead of doing it yourself we make programs to do it automatically so think of web scraping as grabbing useful info from other websites without directly taking their permission to do so we do this to help with all sorts of things like creating new products automations or even checking what our competitors are up to currently many big organizations scrape data in various formats from images videos text to reviews and pricing so when you look something online like wanting to buy a fancy MacBook M2 you'll see suggestions from different websites right at the top and if you click that shopping button at the top you'll see the complete list with different filters that's Google shopping Google collects details from online shops only to show them to their Google shopping page companies like Google or open AI are scraping the data available on other sites to build their amazing products what's Chad GPT anyway a wild wizard that went around grabbing all the public data I could find to answer this question let's first clarify the difference between a web scraper and a web crawler although the main functionality of extracting information is the same for both they slightly differ in their process web controllers generally navigate the web and web scrapers extract specific data from the pages web crawlers follow all the links to discover new pages but web scrapers Target specific Pages for data extraction usage is much different crawlers are mostly related to SEO analysis and web scrapers can scrape absolutely any piece of data in the recent World it has been mostly used for data set generation for machine learning training so as you can see web crawler is what Google or other browsers use to index websites they too navigate through our whole website first find links and rank them accordingly but on the other hand scraping targets specific types of websites on their page it's more about focusing on what truly matters but with that in mind how do web scrapers truly work it all starts with sending an HTTP request we send requests to websites we want to scrape this process is automatic it's not like you said the whole day pressing a button to make requests our code will do it on its own next we get a reply to our scrapers request with the website's content this content usually includes things like HTML CSS and JavaScript Pages as we receive all the content we need to go through the process of parsing and carefully selecting the specific things that matter to us if we're doing a price comparison we're interested in the product and its price details but how do we actually do the parsing well after we get a response from a website it's most likely going to include an HTML page and let's say you want to extract price from a product page we have to directly pinpoint the element that holds the price information for instance you inspect the price of a product on an Amazon website you'll notice it's located within a specific type of element called a span element which is identified by a unique class named like a price hole of course it can be different for different websites that's the goal here we'll focus on this specific element to retrieve our desired data whether it's the title pricing or any other information from this page once we've parse the content we extract only the data we're interested in and lastly you write the necessary code and store the extracted data in your database use it for training machine learning models or building new apps I hope that by now you have just a bit of a better understanding of the typical scraping process but now let me teach you how to create web scrapers is there a ton of code that we have to write well not at all people often think web scraping is something that takes a lot of work to build most of the time it's only about focusing on business logic which means figuring out what kind of data you want and where you want to get it from some smart Minds have already created amazing open source scrapers we can install coal and we're good to go with one of the most popular scrapers is Puppeteer right now you're most likely watching this video through a browser a browser that has a user interface you can interact with this is called a headphone browser on the contrary The Headless browser doesn't have a graphical user interface or GUI for short it has no visual interface like the traditional browsers we use daily it runs in the background and allows us to do automated web actions programmatically without displaying the web page on the screen developed by Google it's one of the top choices for many developers who do web scraping and here's a quick example of how we can use it first you import required modules then you create an Express app Define a route to handle web scraping launch a headless browser using Puppeteer open up a new browser page navigate to the Amazon product page extract all information from the page close the browser and send the scrape data as a Json response there are also other tools like Cheerio that help us parse the HTML content more easily and Cheerio can be paired with other scraping tools for example with puppeteer import record modules create an Express app Define a route to handle scraping launch a headless browser using Puppeteer open up a new page navigate to the Amazon product page get the HTML content load the content into Cheerio extract information from the page using Cheerio selectors which is much simpler than doing it manually and then close the browser and send the scrape data as a Json response although the difference with and without Cheerio may not be noticeable right now once you build something big like what we'll do today you'll immediately notice how useful it is sounds simple so far well here's where things get a bit more complicated there are many web scraping obstacles it's not us trying to make these requests but the code we wrote designed to act like a human but still it's not the websites we scrape can understand who's using their site and if they notice Something Fishy they'll immediately block your IP web scraping is all about imitating a human and working on their behalf it has to simulate the behavior of a real human being for that reason you may have seen some captchas like solving puzzles and identifying objects and images they've gotten pretty crazy lately requesting us to do all sorts of things to prove we're human websites use captchas to prevent automated Bots and scrapers from accessing their content so if you're a human being watching this video like it and comment Down Below on how you like these videos that start with a crash course and then Advance into a build and deploy I'm always searching for new ways to provide you with the best content possible in your opinion truly means a lot to me the second way websites block you is by doing IP blocking and rate limiting if you send too many requests you're done the third obstacle we face is dynamic content as web developers we build websites using react that loads the content after the initial page load we send minimal HTML and execute our JavaScript code based on the conditions traditional scrapers might struggle to capture this Dynamic content you might be reminded of the Headless browsers we just discussed although these headless browsers are great they still can't crack a few of these problems so where do they fail well all of the obstacles we talked about from handling JavaScript heavy Pages captchas anti-scraping measures user interactions like searching something before extracting data from a page device detection complex navigations IP rotation managing requests legal concerns and and literally anything that may require human intervention while scraping the name bright data being graded as one of the top most voted for web scraping bright data kind of does it no they haven't hired an army of people to do the job instead they've engineered a head full browser termed scraping browser which will do the things as if a human is actually using the website it almost completely imitates a human and IP rotation it does it automatically so the creator of any website will never know who is using their website it can be you or another bright data service no one will know and that's exactly the reason we'll use write data in our project to build a project without worrying of any above obstacles don't trust me blindly here are a few examples to show you why we'll use bright data from big companies including upwork everyone out there is recommending bright data if you really want to build a proper web scraping project and the scraping browser isn't the only feature bright data has to offer they go beyond that by providing big companies data sets with a unique web unlocker feature a way to getting unblocked it's going to help you overcome any website blocks with its AI automated features and the best thing is we'll use the same amazing feature to scrape the data from Amazon websites in a couple of seconds within the app you are about to build so without any further Ado I hope you learned a lot about why scraping is important how to do it what the obstacles are and now you are going to build your own unblockable web scraping tool using bright data to get started building our phenomenal price track replication we can start as we usually do by creating a new empty folder on our desktop we're starting from scratch and let's call it price wise then open up an empty Visual Studio code window and simply drag and drop that folder in foreign or half screen go to the top left View and then terminal this is going to open up a built-in Visual Studio code terminal once you have it we can initialize our next JS application and next.js13 is the react framework for the web it is used by some of the world's largest companies enables you to create full stack web applications by extending the latest react features and integrating powerful rust-based JavaScript tooling for the fastest builds so let's go ahead and click get started immediately right here you can read more through the next GS documentation and learning about most important features such as routing rendering data fetching and more and then you can go to the installation process where we can follow the documentation to install it I would never want you to just go ahead and type out the commands without knowing why or how I got to them in the first place so that's why I want to refer to the documentation whenever we're doing something and that's the exact premise behind our ultimate next gs13 course if you haven't already go to jsmastery.pro forward slash next 13 and here you can see why many of these companies are developing their applications in xjs and how we can join them in doing so while understanding all of these Concepts we've seen within the documentation and more so if you're interested in learning next which I'm sure you are because you're watching this video You're Gonna Love what we do within the next GS course turning this into something like this so let's get started with the installation of our project by copying this Command right here going back within our app pasting it and then just saying dot slash to create it in the current repository where n once you do that you can press y to install the package that's going to install our app and then you can answer a couple of questions in this case we do want to use typescript so we can say yes for eslint I'm going to say no just so we have a simpler installation process for team CSS we're going to say yes no Source directory app router definitely we don't need to customize the import Alias and that's it I was mostly pressing just enter to go with the default options so now let's wait until the dependencies have been installed and then we can start developing our application and the initial file and folder structure and there we go success created price wise add desktop right here I'm going to clear my terminal and I'm gonna immediately run npm run Dev which is going to open up our app on localhost 3000. once it runs it you can simply press ctrl-click to open it up in the browser and you'll be greeted with this great nextgi starter screen so let's drag and drop that all the way to the left and that's going to allow us to have enough space for the development of our application while at the same time looking at the deployed site let's close our terminal and let's start looking into the base file and folder structure so first of all here we're going to have the app and within the app we have our home but we don't need any of these so you can simply just remove everything and run rafce this is going to create a base react Arrow function component and something known as page but we can rename it to home there we go so this is our home page and now you can see just home if RAF CE didn't work for you that must mean that you don't have the es7 plus react redax react native Snippets installed so just install this extension and it should work immediately great so with that said now we have our page but we do have some weird lines appearing right here that's because our globals.css is applying some weird Styles so what you can do is go to the description of this video and there you're gonna find a GitHub gist with the modified globals.css file so simply copy it and paste it over here if you save it and reload the page you're going to notice that we're missing the font enter and this font will have to be applied as a custom class within our Tailwind config so in that same GitHub gist you'll find a modified tailwind.config.ts which you can simply override right here once you do that you're going to notice that here we keep almost everything the same but we do change a couple of primary colors just so we have a consistent application coloring and we also add the font family of inter so another thing we need to do is we need to head into the layout.tsx and this is where you actually import fonts from Google or elsewhere so alongside importing enter what we can also do is we can also get the space grotesque like this and we're going to Define enter as subset of Latin and we're going to also Define another font right here by saying const space grotesque and that's going to be equal to space underscore grotesque where we Define a subsets of Latin like so and we can also add different kinds of weights right right here by saying weights which is going to be an array of 300 and I believe those have to be strings so 300 we're going to also have 400 500. 600 and 700 so we want to go really bold and we can see that our typescript is complaint right here it is going to be just wait not weights so immediately you can see how useful it is to have typescript have your back and with that everything works we're back with home and we don't have those weird lines because we have modified the globals.css and the tailwind.config TS files now bear with me we didn't do any additional crazy styling here we've just done the setup same thing for the global CSS so here we have some often used cards which we're going to apply which apply a couple of different basic styles from Tailwind but the majority of the styling of this application will still be done exactly by you by following this tutorial so you don't have to worry about that this is just the setup to make her life easier in the long run now that we're on our home let's also modify our layout so in the layout we do import globals.css we import the metadata and finally we import the enter and space grotesque font which we Define right here after that we have our metadata where you can change your application's title ours is going to be called price wise and you can add a description where you can say something like track product prices effortlessly and save money on your online shopping which is exactly what our app does and it's really cool that you can immediately just change the title in the description right here which is going to modify your SEO metadata now after that here we have our body and we have our main inside of which we have our children but here we also want to add a nav bar because we want our nav bar to appear everywhere so let's wrap our children in a main tag and put it right here within it let's give this main a class name of Max W Dash 10xl so this is going to be a Max width of 1440p we always want to have a specific Max width even if a user has an extra Ultra hard monitor that Samsung Odyssey thing that goes all the way around you still they cannot read the text if it spans more than let's say 1000 pixel or 1440p so we want to just provide a Max width on the entire container and we can also Define a margin X meaning marginal left and right to auto and now this allows us to create our first component which is going to be called navbar so here we can define a self-closing navbar component we can then go into our not app but we're going to go outside of it and we're going to create a new folder called components within components we can create a new file called navbar.tsx and there we can run rafce to quickly spin up our navbar now if we go there you can simply double-click the nav bar and press Ctrl space or command space if you're in mac and this is going to automatically add the import from add forward slash components forward slash navbar so if you do that now we can see the nav bar and you can see the home as well which is great now since we're doing this in the layout which is a special nextgs file again more on that in the next GS course that we're doing essentially what layout is doing let me find it in the docs layout is going to allow you to share different parts of the application across different pages the nav bar is the best example of doing so you can define a layout by default exporting a react component from a layout.js file and then you can add a navigation bar like we did right now so now we are right here where we have the nav bar in the home we're done with our layout file we can do the nav bar later on and we can do the page later on but let's quickly look into the rest of our files that we have in our application so right here we have the app and within the app we have our favicon for now this is just a regular Versa logo but we want to update it so what we can do is in that same GitHub gist down below you'll find a link to a zipped public folder simply download it unzip it and then delete this Current public logo and just paste the new one right here in the root of your directory there you'll be able to find some assets such as icons we're going to use in this application also some images which we're going to use such as the hero images and more and finally you'll be able to find a new favicon.ico which you can simply drag and drop to the app and then replace it right here that way if you now reload the application you'll be able to see this little fire with a discount on top which is the logo of our application sometimes it's not going to update automatically so you have to do Ctrl shift R which is essentially a hard reload but sooner or later it's gonna update so now we know everything that's within our app we have the favicon globals layout and Page within the components we'll be adding more components then we have the node modules for all of our packages we have the public with the assets images and icons we have a git ignore as we don't want to push everything to GitHub especially not our dependencies and with that said we can get started with developing our navbar our navbar is going to use a semantic HTML5 header tag and within it we also want to use a nav tag this way all devices that need accessibility know that this is indeed a navigation bar our header is going to have a class name equal to W-4 and this is going to provide it a width of 100 if this syntax seems weird to you that's because maybe you haven't heard of Tailwind and if that is the case you definitely should learn more about it essentially by adding utility classes you're writing a real CSS but much faster while still keeping the entire customization flexibility but just allowing for a much faster workflow so while going through this video if any of the classes I write are unclear to you you can just go here and just search so if I go for wool you'll be able to see that it is indeed a width and here you can see what that property does if it's still unclear to you I would recommend installing the Tailwind intellisense package which is this one right here and what that's going to do is if you hover over it it's going to give you the exact CSS properties of the utility class applies great so let's also add another class name to our nav which is going to say nav this is not a tail in CSS class this is one of the classes we added within our initial Styles so if you search for nav you should be able to find it within our globals.css which essentially just applies a few more daily classes so sometimes you're not sure what this does you can just go to search search the class name find it and then apply it great so now we can see that we cannot see anything but that's because we don't have any content So within our nav we of course want to add a link and this is going to be a link coming from next forward slash link so you can automatically import it right here at the top and give it an href equal to forward slash meaning it's going to point to home we can give it a class name equal to flex items Dash Center and then gap of 1 to provide some spacing between the elements then that link can also render an image this image is of course also an optimized next forward slash image so we need to import it at the top and then give it a source property equal to forward slash assets forward slash icons forward slash logo.svg if you save it it's immediately going to be visible or at least when you give it a width property of 27 and a height property of 27 as well and then an off tag equal to logo if you save that you can see our logo appear on top as it does on the final website but now let's add the second part of this which is going to be the P tag this B tag is going to have a class name equal to nav Dash logo and we can also say price and then create a span element give it a class name of text Dash primary so now we're using this primary color and then we can simply say wise so this is going to be price wise and you can see it nicely right here the next thing we want to do within our navigation bar is if we go to below the link render a div that's going to have a class name equal to Flex so the items appear in a row items Dash Center and GAP all five within here we can Loop over the elements we want to show so that's going to be search heart and then profile so we can do that by declaring them right here at the top by saying const nav icons is equal to an array where we have a couple of objects the first one is going to say Source forward slash assets forward slash icons forward slash search dot SVG and we can also give it an ALT tag equal to search we can duplicate this two more times the second one is going to be a black heart dot SVG and it's going to say heart the third one is going to be user and it's going to say user what this allows us to do is now simply open up a new Dynamic block of code and say nav icons.map where we immediately get a specific icon we want to map over and then you can immediately return something and that something is going to be a self-closing image that's just going to have a key equal to I can dot alt because we're mapping over it it's going to have a source equal to icon.source it's going to have an ALT tag equal to Icon dot Alt and it's going to have a width of 28 a height of 28 as well and a class name equal to object Dash contain if I now save this you can see three icons appear and if we didn't have this array of icons then we would have to do all of that here manually and we would have to display three different images which you never want to do if you can make it dynamic and with that said believe it or not this is it for our simple navbar this now allows us to much better focus on what is Yet to Come which is the main part of our application the home page so now we can exit the nav bar and go into our app and then page which is our Primary Home component so let's do that next to get started with our home we can start with the UI in the ux of the home meaning what we can visually see on the screen before we start implementing the functionality so with that said we're going to wrap everything in an empty react fragment which allows us to add more elements within it and then first we're going to wrap the first part within a section HTML5 component this section is going to have a class name equal to padding X of 6 and just to be able to visually see it I'm going to add a border of 2 and a border red 500. now you should be able to see how this component behaves right now it's just a line but on medium devices we can give it a padding X of 20 and a padding y of 24 and that's going to make it expand to form this section within this section we can create another div and this div is going to have a class name equal to flex Max XL flex Dash call so on Max Excel devices which means with a Min width of 1280 pixels it's going to be a column so it's going to display more so in a tablet way like so and then we can give it a gap of 16. great so within this div we're going to create another div and this one is going to have a class name equal to flex Flex Dash coal and justify Dash Center finally within it we can start creating our layout so right here we can create a P tag that's going to have a class name equal to small Dash text and it's simply going to say smart shopping starts here okay that's nice and then we can render an image tag which we of course have to import coming from next forward slash image at the top and we can give it a source equal to forward slash assets forward slash icons forward slash Arrow Dash right dot SVG give it the alt tag wolf Arrow right and a width of 16 as well as a height of 16. and now if you save it you can see smart shopping starts here and we're starting to replicate what we have on the deployed application but of course this primary H1 as well as the paragraph here in the search bar is what's going to make our app pop so let's head below this B tag and let's Implement an H1 tag that's going to have a class name equal to head Dash text here we can say Unleash the Power off and then we can go to a new line and enter a span element with a class name of primary and there we can say price wise just like so but by giving it an extra space within there we go Unleash the Power of price wise and of course we want to do this primary or rather text Dash primary which is going to actually color it in our primary color I gotta say red is a bold color to choose but in this case we're cutting the prices so we want to be really bold with our coloring and design as well now below this H1 we want to have another P tag that's going to have a class name equal to Mt of 6 so margin top of six to divide it a bit from the top and then within it we want to just copy what we have on the deployed site and as a matter of fact I would want you to have this site opened as well so you can go to pricewise dash 10.4 cell dot app and you can copy this description from here and paste it here essentially it's going to say powerful self-serve product and growth analytics to help you convert and gauge and retain more finally below this we will want to render a new component which is going to be called search bar for now we can simply return this as a regular text element but soon enough we're going to turn this into a special component now we want to go below this div and here we want to display the hero carousel a hero is usually a component that's being displayed on the home page such as this entire part and then a carousel is this Carousel that moves around many different images so this is going to be converted into its own component really soon as well with that said let's go down below this div and below the section and we can create a second section this section is going to have a class name equal to trending Dash section and we're going to render an H2 within it in this H2 is simply going to say trending and it's going to have a class name equal to section Dash text so now we can see trending and of course this is going to portray our second part of the application or we can see the products that want to get the discounts and prices for and finally within it we can create a new div this div is going to have a class name equal to flex Flex Dash wrap so the elements nicely wrap as you can see on the finished site on smaller devices Gap Dash x-8 to provide some horizontal spacing as well as Gap Dash y-16 to provide even more vertical spacing within it we simply want to get some products for now I'm going to mock an array of products for example let's do Apple iPhone as we see it on the right side screen let's also do something like a book and let's do sneakers there we go and we can say dot map or we get each individual product and for each product we automatically return something for now that something can be a div that's going to just simply render the product name just like so so now if we go back you should be able to see three different products appear right there of course later on we're going to turn these into actual product card components so it's going to look much better so with that said let's go to the second component of the day which is going to be the search bar so with that said let's dive into our second component of the day which is going to be the search bar component so to create it you know the drill we go into components and you create a new component called search bar dot TSX run rafce to quickly create it go back here just call it as you would any self-closing component double press it and then press Ctrl or command space and then press enter to Simply import it right here at the top from components search bar once you have it you can navigate to the search bar component and we can start implementing it together as before we're going to first start with the UI and then we're going to move to the functionality so let's start by removing this div and rather returning a form component this form is going to have a class name equal to flex Flex Dash wrap gap of 4 and margin top of 12 and of course we need an on submit which is going to call the handle submit function so of course we need to declare this but first let's make this all visible in one line we can declare our handle submit right here within the search bar component by saying const handle submit is equal to just an arrow function and you're going to soon notice that as soon as you do that you get an error saying that we're using an on submit right here and that is considered an interactivity so if you need to turn this into an interactive component you need to turn it into a client component which means the two at the top have to say use client this means using a used client directive to turn it into a client-side runner component while everything else such as this page is going to be server side rendered within next.js docs they talk a lot about rendering environments where you want to render things and when should you render things where I believe it's somewhere on here different rendering strategies but then they explain how it streams how we can have client-side components within server-side components caching and all sorts of other things but sometimes it's not so easy to just go through all of the docs and actually you need much more real life reference points to be able to fully understand how the server and client components work this right here is a really nice table they created where they say when should a component be a server-side component and when should it be a client-side component and essentially whenever you add on clicks on changes or states that means that it has to be client-side in any other case it is a server side and that's also another thing we cover in depth in our next.js course where we really say that you cannot just use next 13 just as good old react you have to know about all of these server Concepts and without realizing those things in depth your app is going to turn into a slow client-side mess which is exactly what happened to us but then yeah once you learn all of these server-side Concepts you can turn it around and you truly can feel the power of nexjs so for now I'm going to pin this as well on the top and it's pretty cool to pin these tabs as you can quickly get back to them later on so in case we need to reference the finished site or the course content we can just go here and here we have our Local Host so yeah with that said we turn this into use client and our entire page original page is going to be server side rendered the only reason why return to use client here is because we're using interactivity and we'll also be using hooks great so now we can move over right here by creating an input within our form what would the form be without an input right so we can give it a type is equal to text element and we can give it a placeholder equal to enter product link if we save it immediately you can see some kind of an input and we can also give it a class name equal to search bar input which is just going to immediately style it a bit better next to the input we want to render a button component and this button is going to have a type equal to submit because we wanted to submit the form and we can give it a class name equal to search bar vtn and it can say something like search if we save this it's looking much better and we can close our search bar for now we're going to come back later on to give it interactivity but now we can focus on our hero Carousel which is I think the most noticeable component of all considering how much space it takes and how interactive it is so with that said I think we can remove this ugly border because now we can visually see the layout of the application we're creating and we can focus on creating the hero Carousel by going to our components and right here say hero carousel.tsx run rafce and then simply import it right here by turning this into a self-closing hero Carousel component and importing it from components hero Carousel closing the page and moving into the hero Carousel so let's implement this component next and believe it or not this is the first time in our application so far that we're going to use a third-party package I always say don't rain when the wheel if gray developers have already developed a nice looking Carousel that allows you to switch images why would you have to develop it on your own when we are in the world of the open source code so with that said let's go ahead and open our terminal by pressing Ctrl and then backtick or you can just go to view and then terminal right here and then see your shortcut there now what you can do there is open up a new terminal by pressing this plus and then we can install a new package by running the command npm install react Dash responsive Dash Carousel and once again this is not just the package that I randomly found this is a well-maintained npm package with about 370 000 weekly downloads it works really well with nextgs apps and here's how we can use it so go to this npm page you can just search for the react responsive carousel on Google and then we'll have to import this demo Carousel example so import just two of the import lines this is going to give us the loader for our CSS and then we have the actual carousel right here and then you can also copy the demo which is the carousel and you can put it right here inside of this div and of course we'll want to indent this properly so let's just select all of this I'm holding Ctrl and then alt and then now I'm moving everything with a backspace so now within here we have our Carousel we have imported the CSS and we have imported the package itself so now if we save this and go back we can close this you can see that it says that the super expression must be there now or a function not undefined okay that's an interesting one I'm guessing that's because we didn't provide too many props for our Carousel and also that our images are actually fake uh these are just copied from the documentation so now we can actually modify it and also it's not a good practice to use just a basic HTML5 image in next if you can use the optimized nextgs image tag so what we can do here is create a new let's call it a utility or a new constants array that we can say const hero images is going to be an array or we're going to have multiple objects the first one is going to have the IMG URL of forward slash assets forward slash images forward slash hero dash one dot SVG and it's going to have a I'll tag of I believe this is a smart watch there we go we can say smart watch now you want to add a new one right here below and we can duplicate this a couple more times however many you want to have in this case we can have five so I'm going to duplicate this four more times the second one is going to be two three four five then we're going to have a bag we're gonna have a lamp we're gonna have an air fryer and we're gonna have a chair I think this is pretty diverse so it means that our app can do anything and now instead of just manually rendering all of these images right here we can just open up a new Dynamic block of code and say hero image or rather it should be images we can change it later that map where we get each individual image and we immediately return a new image tag just like so by importing it from next image a couple of things let's rename this to images as arrays should always end with plural meaning with an S and then let's give it a source equal to image Dot imgurl alt is going to be image dot alt width is going to be about 484 pixels as well as the height 484. I found this value to work the best it has to have a class name equal to object Dash contain and it has to have a key since you're mapping over it of image.alt since we know that all of these are unique so now if we save this you can see it's still going to complain which is okay but now we can provide some additional props to our carousel and by quickly opening the react responsive Carousel again we can see right here below there are many different props you can pass depending on what you wanted to do so in this case we can do a couple of things we can say show thumbs is going to be false these are thumbnails of the images we can do the autoplay we can also turn on the infinite Loop so it's always going to Loop we can add the interval of about 2 000 milliseconds so two seconds per image show arrows is going to be set to false as well as show status is going to be set to false after playing a bit with this I found those values to work the best and here's the thing with all of this with properly implementing our carousel the page still breaks now I want you to think about it because this is really important although it can seem like a random error it's really important for you to understand why we're seeing this error well first of all we can see that this error is coming from react easy swipe react swipe we don't even know what that is we haven't even installed it but the carousel package did behind the scenes and if you look at the final Carousel what is it doing is it listening for some clicks is it interactive is it doing something the answer is yes and although as we've seen in the react docs you're not manually calling the on clicks or on changes or different states or stuff like that you are using a package that does that so what does that mean well it means you need to turn this into a use client component because it deals with interactivity so once you fix this as you can see we're back and we can actually Loop over these different images through our amazing Carousel so this was really really important because as you saw even though you're not manually using something you have to be careful of the packages we're using when it comes to the client and server side components similar thing goes to framework motion and all of these other packages as well as some State Management libraries like react context for managing States and also redex toolkit how are we going to use these states if you want to keep your app server-side rendered we dive deeper into all of those Concepts within the course first of all with the Deep dive lectures to truly understand how it works then we have a building deploy but of a really complex application and then finally there are active lessons where you can code things yourself to truly learn how to do it like a champ so yeah just wanted to put that out there now let's give this div a class name of hero Dash carousel and if you save it it's going to be a bit more manageable a bit smaller but yeah I'm glad that you now know that even though you're not using those States or interactivity maybe your packages are and you still have to declare it as you as client and now the last missing piece of the puzzle is actually seeing this on a larger size right here on the finished side and you see this nice air right here although it's small it really draws the user in to enter this link because this link can be anything it can be a new bag you want a new lamp or even an Apple Watch so it really draws you in and we want to implement that arrow on our current application as well which we can do simply by inserting a new image right here below the carousel with a source equal to assets icons hand drawn Arrow dot SVG also notice how important it is to name your svgs and icons in a meaningful way as you can easily refer to them alt is going to be Arrow width is going to be 175 and height is going to be 175 as well if we save it you can immediately see it but we have to absolutely position it so let's give it a class name and here we can say Max Dash XL it's going to be hidden so on devices with a Min width of 1280 it's not going to be there but if it is we want to position it absolutely with a minus left 15 like so bottom zero and then Z index of zero and bottom I have to spell it correctly and save it and now this is going to bump it up also left I have to spell that correctly too if you do that it's going to look good even if you expanded it still looks great but on smaller devices it disappears there we go how easy it is to use Tailwind for styling and responsiveness now we're done with our hero Carousel so we can close it and we can go back to our original home page right here so we have almost everything we need when it comes to the initial layout we have a nice home screen where we can enter links we have our trending where we'll be able to see the links that we created later on but now it's time to scrape the data not even just scrape it save it in the database and even update our users via email when the prices change which means that we have to periodically scrape it as well for updates and if you know anything about scraping you know how hard it is to do it properly especially considering many websites have blockers for scrapers captchas checks and more thankfully we'll use bright data in this video so right here bright data offers many many different solutions predefined data sets and more but in this case we'll be using their web unlocker scraping solution and in upcoming videos we're going to also play with more powerful scraping browser features for now let's go to the web Unlocker and here you can see they called it unlocker because it can unlock and scrape the toughest websites so you can access any public website at scale and leave unblocking to them this is going to simulate real user Behavior overcome any bot detection and so much more so there's a lot of things that you can use this for but in this case I'm going to give you a great example by scraping Amazon products from their website so with that said let's start implementing the functionality of our application and of course to do that we'll have to start from our search bar so let's collapse our browser a bit more to the side as a matter of fact we can collapse it all the way to our mobile view it looks good as well and we're going to have these Ever Changing images on the right side although we can just maybe stop it for the time being by turning off the autoplay and the 2000 millisecond interval if we do that we're just gonna have a sticky image which might be a bit easier for us while we're developing the functionality of our application so with that said let's go ahead and go to our page and then to our search bar because this is going to be the first point of interaction right here we'll have to enter the link and then we'll have to deal with some logic so how are we going to do that well first we have to keep track of the URL we entered and we can do that by using a use State Property which is going to be called search prompt that search prompt is going to have its set search prompt and it's also going to start as an empty string and then we have to import a use State coming from right here at the top import use state from react now that we have this set search prompt or no rather it's going to be search prompt and set search prompt just like so we can now use it within our input now this input can accept two things the first one being a value of search prompt and the second one being an on change Handler that's going to be equal to we get an event and then we simply set search prompt to be equal to e.target.value so this is simply going to keep track of our value that's in the input within our state and what that allows us to do is to do something when the submit is triggered so we can go into our handle submit and here we get an event and since we're using typescript we need to define the type of the event so it's going to be a form event coming from react so you have to import it and it's going to be of a type HTML form element right here and you have to close it like so now it's going to know exactly what this event has so here we just need to say event dot prevent default the default behavior of the browser once the form is submitted is to just reload the page in the modern react and xjs applications we don't want that to happen because our apps need to have more of a native mobile feel you're on an application you're not on a website so we want to prevent that default Behavior then once we do that we need to check if the URL we entered is indeed a URL so we can say const is valid link and that's going to be equal to to the call of a function which we can declare above so here we can say const is valid Amazon product URL and that's going to take in a URL of a type string and it's going to return what is it going to return let me ask you this is going to be a string an array a Boolean can you know well you can if you use proper naming which is also something with each inner course you want to start all Boolean variables with an is that means that you immediately know what the output of this function is going to be it's going to be a Boolean and now that we have created this function we can simply call it right here is valid Amazon product URL and we can pass in what our search prompt which we're saving in the state right here so now this variable right here is going to be the output of whatever check we Implement right here so let's ensure that check is good so our application doesn't break here we can open up a new try and catch block and in the try we can first parse the URL by saying const parseed URL is equal to new URL and you pass in the URL or passing as a string then you want to get the hostname of that URL which you can do by saying parsed URL that hostname this is going to be just the first part for example for JS Mastery Pro it's going to be jsmastery that Pro so what we want to do next if we want to check if hostname contains Amazon.com or Amazon dot something followed by country code there are many different Amazon sites worldwide right so we want to say if hostname dot includes Amazon.com or hostname dot includes maybe something like Amazon dot right or maybe hostname dot ends with and we want to put the Amazon in there so we have just a few different checks which we can display on a few different lines so it's easier to understand what we're doing so if it is any of these cases it means this is a valid Amazon link and we can simply return true and otherwise we can return false right here I think in the catch we can also return false because something went wrong so it's not a valid URL so now if we go back to our current website let's simply do an alert right here saying alert and we want to render is valid link then valid link else invalid link so now if I type something like test and press search we're going to get invalid link even if I type a real URL something like https column forward slash forward slash JS mastery.pro it's still an invalid link why because it's not an Amazon link but if I do something like let's do amazon.com forward slash MacBook and we do search it is an invalid link still let's try to turn it into a URL https column forward slash forward slash and now it's a valid link so this is just a check we're doing which we can just collapse here to ensure that it indeed is a valid link so now if the link is invalid we can say if not is valid link we can simply return to exit out of the function and then do an alert please provide a valid Amazon link there we go so now we have the is valid link and then in all the blocks of code below we know the link is valid so we can open up a new try and catch block here we can turn on the loading so I'm going to create a new use State field called is loading and set is loading initially set to false so once we click this is definitely going to take some time right in which case we want to turn on the loading so here in the try we want to start doing something so I can say set is loading to true and we can add a finally block which is going to happen either way on try or catch and whatever happens we want to then stop the loading this loading can further be used within our application within the button by saying something like right here if is loading then this is going to be searching else it's simply going to say search and we can also use our current search prompt to define the disabled state so if search prompt is triple equal to zero then the button is triple equal to an empty string then the button is going to be disabled as you can see great so now we have implemented that one more check and in the catch what we can do is simply console log the error but the magic will truly happen in the try so right here we'll need to scrape our first product this is going to be quite interesting and to be able to do that we're not going to declare all the logic here we actually want to create our first server action so how do you declare server actions well let's close our search bar and our page and let's go all the way to a new folder we'll create called lib as in library within it we can create a new folder called actions and within actions we can create a new file called index.ts here you can declare that file as a use server this means that all the code written here will run only on server and there we can create a new function and we can immediately export it by saying export async function scrape and store product and this is going to be a function that accepts a product URL of a type string and now we can start creating that function for scraping the product as well as saving it in our database because we want to be able to know and track the price later down the line so let's focus on implementing this first server action and with it our first scraping functionality of the day so within here we're gonna pass a product URL as the first and only parameter and then of course we want to check if no product URL meaning if it doesn't exist we simply want to return means exit out of the function if we do have a product though we'll have to do try and catch block as we'll be dealing with asynchronous actions such as scraping and also database access so what we can do here is in the catch we can simply throw a new error and we can pass a template string of failed to create update product and we can also render the error.message great so now this is great we can declare the error as type any but more importantly what happens here within the try block here we have to actually scrape the product so let's say const scraped product is equal to a weight scrape Amazon product and now it's not as simple as that we pass the URL but now this is not a built-in function this is something we'll have to develop and I'm going to teach you how to scrape all different kinds of applications so let's create a new folder within our lib which is going to be called scraper and then within it let's create a new index.ts file within here we can create an export async function scrape Amazon product and it's going to accept a URL which is going to be of a type string and then we have our function so now we can go back here and we can import scrape Amazon product from dot slash scraper right here at the top and now it's our job to dive into the scraper function and implement the scraping functionality first we want to make a check if a URL actually exists so if there is no URL we're gonna simply return and then we want to do a bright data proxy configuration this is going to ensure that we can actually use right data scraping to be able to get the product data so just visit the link in the description to go to the bright data's website and then sign up that way the link will most likely include some special credits for you to use so you can follow along with this video without any issues and without making any payments so the process for you might be a bit different but in general you need to sign in for the platform once you sign in you'll be greeted with a dashboard that looks like this you have a new scraping browser and after getting credits specifically for watching this video you should be able to see your dashboard then right here you'll be able to see many different services such as proxy manager proxy Chrome extension serp API and in my proxies you'll see even more stuff right here you'll want to choose the web unlocker tool so go to my proxies and then click add on top right and add a new web unlocker tool here you can see monthly cost but once again you don't have to worry about that you're getting all of these benefits and you have a solution name we can name it something like price wise and I don't think we'll need special features for now nor the country so for now simply click add we can easily change this later on and yes we want to create a new Zone called pricewise once you've created it you can see your host your username your password and all the other things so let's utilize this within our code right here we want to say const username is equal to and now we want to get a string of we can just copy this here and paste it directly but what we want to do is we want to put it in the environment variables so we want to ensure that all the data is safe so let's create a new DOT EnV file and within it we can create a new bright underscore data underscore username variable which is going to be equal to the string that we just copied and we want to do the same thing for the password so that's going to be bright underscore data underscore password and that's going to be equal to the password that you have make sure to use your data right here once you have those we can declare this username just by saying process.env and then bright underscore data underscore username we want to repeat this by saying password and then we want to do bright underscore data underscore password great we also want to define the port that we want to call it from and we can see some of that data right here in the base curl so this is a huge thing that you have so let's copy it and let's just comment it out for now here you can see the port number so what we can do is we can declare it by saying const port is equal to 22225 we also want to declare something known as a session ID so session underscore ID and we can use some random numbers such as let's do one million so that's one with six zeros and we want to multiply that by math dot random and then we want to do a single or zero so this is going to give us a session ID we also want to Define some additional options so we can say const options is equal to we want to have auth within it which is going to contain the username and the username is going to be a template string of username and then we want to add a dash and then we want to define the session and then we want to add the session ID as well so this is how we Define the username in the authentication part and the password is simply going to be password finally the host is going to be coming from here BRD super proxy.io so we can Define the host right here below the auth so host is a string of brd.super proxy.io then we want to pass the port which is port and then we also can do something called reject unauthorized which is going to be set to false and essentially here we have everything we need we have our options object with this options object we can actually make a request to get the data from Bright data so we can remove this curl and we're going to do it right here within our code so let's open up a new try and catch block within the catch we can simply declare the error as any and say Throw new error where it's going to be a template string of failed to scrape product and then error message but again in the try is where magic happens so here we'll have to do the actual fetching by fetching the Amazon product page so let's write that down fetch the product page and first we need to get the response from that page by saying a weight axios.get or we get a URL and then we pass some additional options like so and of course we have to install axios which is going to be the second package of the day so we can open up the terminal and run npm install axios and here we're going to also use something known as cheerio such a cheerful name so let's install those two packages and let me show you how we can use them I think everybody knows axios it's just used to make API calls but cheery on the other hand with such a cheerful name is the fast flexible and elegant library for parsing and manipulating HTML and XML a set of calls which you can use to really quickly parse through HTML and what is data scraping if not parsing the HTML content you see so we're going to use that so let's immediately import right at the top axios from axios as well as import everything as cheerio from Cheerio there we go so now we have our response and what we can do is we can just cancel log it let's see what do we get within this response or rather let's do response.data so now we can save this we're actually doing our proxy configuration using bright data we should have some funds in there everything should be properly set up and now we can go back to our actions and you can see that we're actually scraping this product I mean we're trying to call this scrape in store which is then calling this crepe product which is then console logging the response but of course to be able to see the output of this function we have to be calling this function and therefore we have to call this function scrape and store product which if you remember correctly was all the way in our search bar here we were just preparing to scrape the product so now we can actually do so we can say const product is equal to a weight since we're doing a weight we have to turn the handle submit into an async function and then we can call scrape and store product coming from lib actions and we can pass in the search prompt great so now we should be seeing the output and let's go ahead and go back to our Local Host now my question for you is do we open up the console or not where is this console like going to appear think about it I'm going to open up this terminal right here and go back to where we have our app and now I'm going to enter a product link but this time let's ensure it's a real product link so let's go maybe to a MacBook Air let's do the first one right here so let's go maybe to a MacBook Air and let's go ahead click the first one we have the price right here which is good and let's copy the URL go back and then paste it right here within our application and press search as you saw for a second it said search but then it broke at least it seems so let's open up the console right here as well and you can see that we have error failed to create update product failed to scrape network error coming from scrape and store product so now we know that we have an error right here coming from scrape and store product so let's see that is here where we call it and that is this error that we display and this error is most likely happening with this function because this is the only function doing some actual data fetching what else we can do is open up the network Tab and just try to repeat that request so if we now expand this a bit so we can see it better and go all the way here and click search you can see that we have a network request that fails and we're getting a typical course error so let's go ahead and fix it together hey guys editing Adrian here I was just editing the video and I wanted to make one point in this point in the application I also needed to add the next dot config.js file and within here we need to turn experimental server actions to True right now they're working absolutely perfectly and we're using them within our next GS course but they're still experimental so you have to turn them on specifically so the entire modified next.config.js is going to be in the GitHub just down below so simply copy it paste it right here in the root of your directory and hopefully this is going to help fix the error but after some time I just retried it and it worked still it can be region dependent so since I'm based in Europe I'm also going to try this Amazon from Germany right here you can see by the ending so I'm gonna give it a try run it and as you can see it's spending and then we get another status of 200 which means that it worked so feel free to test this out if you have your own Amazon in your country your region you can try that but what I can assure you is that on the deployed version of the application all Amazons are gonna work worldwide so you'll even be able to expand on this application and create different versions based on the countries and the regions how cool is that but now even cooler is what we get as the output of the console log of the response.data right here as you can see we have a lot of characters and this is the entire scraped Amazon product details website now it's our turn to make some sense of it so let's do that next right here we're going to initialize Cheerio for the first time and usually how you do it is you name the variable dollar sign and then you say Cheerio dot load and you feed it the response.data that so far we just simply console logged so no longer do we have to hanza log it but now we can use this dollar sign to extract the product title and other information so we can say const title is equal to dollar sign and then we get a product title like this and then we extract the dot text and then we can even trim it so I hope you can see why Shiri is pretty good it allows you to easily get those values now what do we do with it well let's simply console log just the title right now that's going to allow us to see if this is actually working so I'm going to open up the console keep in mind everything is on the server side and I'm gonna run search one more time it's pending and there we go we have a full title right here Apple 2023 MacBook Air that's great which means that this is working so now let's extract all the other data but before we do that let me show you how you can figure out which data to extract so the only thing you have to do is go to Amazon and then click inspect and then you can try figuring out what you want to get from here so we want to get the title so you can select it and you can see that this is an ID that has the title but then we also have a span within it with an ID of product title and see what we did we simply said hey give me the text of an element with an ID of product title in the same way if you want to get the price or anything else on the screen you would simply figure out what the ID or the class name or whatever is for that specific thing that is it so now we can go ahead and together extract all of the meaningful data for our application and the next part we want to do is extract the price but as you can see price has multiple spans over here it's within a span and then we have more spans and unfortunately it doesn't have a specific ID so sometimes you have to play a bit with the elements to extract the proper price or some other elements on the screen so let's go ahead and figure out how to do that we can say const current price is equal to and we're going to call a function called extract price and no this is not a built-in Cheerio function this is a utility function we'll create to keep this function clean so let's create a new file right here within our lib and it's going to be called utils.yes as in utility functions and within there we can export function extract price and it's going to get many different arguments we're going to pass into it and here we can immediately spread them by seeing elements just like so so all of them are going to be put in this elements array and now we can map over those Elements by saying four const element of elements and then we want to get the price text by saying const price text is equal to element dot text Dot trim once we do that we want to figure out if price text exists and if it does we simply want to return price text Dot replace and what we want to do here is we want to eliminate all of the non-digit characters so we can just keep all the characters of 0 9 and then the dots as well so we want to replace all the characters that are not zeros or nines or dots and this is just a regular expression that does that so you can copy these characters or in this case I would even use chat GPT for things like this such as write JavaScript replace function that removes non-digit characters it's really useful for things like this and as you can see it's going to give us this specific string we can use that or you can use the one that I wrote here let's give it a shot with the one that Chad GPT gave us it seems to be just a bit simpler so it's just capital D now what we want to do is if we're not in the loop we simply want to return an empty string great so now into this extract price we can first import it from utils and we can pass everything we think might be related to price sometimes this is going to depend on different Amazon versions so on amazon.com it might be different from amazon.india for example you'll have to dive deep into it and figure it out but I believe this is going to work for most so here we can pass the dollar sign dot price to pay span dot a price hole we can pass a second one which is a dollar sign a DOT size dot base dot a dash Color dash price and then the third one is dollar sign dot a dash button Dash selected dot a Dash color Dash base again sometimes this is going to be tedious work figuring out where all of these elements on the screen are but once you do it properly we can now also console log the current price and ensure it is good so let's go back to our website let's save it let's reload the page and now we can enter that same link that we had here and paste it open up the console and as you can see we get current price is an empty string which is not good what we can do is try to figure out if we're maybe missing something so here in this case we're trying to get the price from the span and here it is a dash price and a Dash text Dash price so maybe we should instead trigger these values as well so let's try to select these by a class name so let's target an element that has a class name of a dash price and Dot a dash text Dash price and this might do the trick so if I go right here and select this link and paste it let's see what do we get I get a 500 again as I said this is that problem with scraping worldwide websites but this is only happening on localhost so once we deploy the website there's not going to be any problems like that so for now I'm going to switch back to our European German Amazon and I'm going to give it a shot of course for you it should be still working so let's click search okay this one is working and now we do get back the price as you can see right here if for you some of these extractions don't necessarily work as they should you can always check how Amazon is doing it and get the right class names IDs and everything and then put them right here but of course we're gonna try to keep this scraping file updated so in case you're experiencing any problems there's going to be a full version of this scraper file in the GitHub just down below but for now I want to teach you how to do it so you can follow along and then if something doesn't work you can always copy that version great so now let's go back to our price wise and let's try to get our original price as well so this could be a discounted price but we want to do const original price and we're gonna also call the extract price function and we can pass some different things we can pass an ID of price block and then underscore our price sometimes it's going to be listed like that sometimes it's going to be listed as a DOT a dash price dot a text price or a span dot a off screen I might have put this here but rather it should have been here because this is the original price that we're seeing and there are many different ways that these elements can be displayed for example this one doesn't seem to have a discount but if we try to search for something else that does have a discount let's see if that's available maybe if I go here and I guess I should be looking at the German Amazon just to ensure it works this one right here does have a discount which is pretty cool so let's try to see how we can get the original price maybe it's this one this one has the a off screen as you can see right here and this is what we're trying to get a dash off screen this is the original price and then we have our price right here as well so let's see how we're trying to fetch that if I go here this is under a price a text price we're trying to just get everything by properly figuring out how to extract those values so let me show you how to do a few more of these to teach you how to actually extract values and then we can move further so for example original price can also be under list price so we can do a dollar sign and then put the list price right here that's an ID sometimes it is also going to show up under dollar sign and then an ID of price block underscore deal price if it's a deal and then the last one is going to be dot a dash size Dash base dot a dash Color dash price many of these different class names so now if we save this and if we want to console lock the original price as well we can do that go back here and click search and it's loading and as you can see we get the original price although this definitely doesn't seem to be parsed in the right way so we might need to fix our extract price function for that I do believe that the original regular expression that we wrote made more sense so let's put everything in a regular expression array and then within it we can put the top carrot which is shift and then six I believe and then you can do the backwards arrow and then D dot like this and you can also ask cha GPT to explain regular Expressions to you so we can do something like tell me what this does and now you can see in summary this expression is used to globally match and remove any character in a string that is not digit or a period so now if we save this and go back we can try one more time and you can see that we get this number and this number or rather these are strings right now we'll have to turn them into numbers but I wanted to include this as a part of the video I didn't simply want to cut this out because multiplying values and extracting values figuring out how to extract values and then how to parse them is the main part of parsing the data in the main part of scraping because once you get the data you have to make sense of it okay so we're just figuring out different ways how to clean up the data okay so now that we have the price we have the original price as well what we can do next is we can figure out if it is currently out of stock so we can create a new variable const out of stock and that can be equal to a dollar sign we have a hashtag of let me just fix this there we go availability span okay and then we do a DOT text dot trim dot to lowercase and if that is equal to currently unavailable then it must be out of stock and we're going to immediately parse it as a Boolean because we have this equation right here so if we save it and click search one more time you can see out of stock is false which is good so sometimes you'll have to do some cleaning and figuring out how to display those values now let's try to get the image as well so we can see const image is equal to dollar sign and then this is going to be hash IMG BLK front and then dot ATT which is the attribute of data Dash a dash Dynamic Dash image or maybe this doesn't exist so we're going to have something else so we can duplicate this and that's going to be Landing image and then the same attribute okay so sometimes you're going to have to have different versions of these things and now we can get the image save this let me show you how we got it first so you can see this image right here in this case it's a video but there we go if I select it take a look what it says right here image and it's going to have an ID somewhere I cannot see it there but I can see it when I hover see Landing image okay so this is how we're getting that image and then we're getting the attribute of data a dynamic image which is the URL so now if I go back and if I just search again if I open up the console you can see we get all different sorts of images and if I click it there we go great so that is working well as well let's see what else do we want to fetch once we get the images we want to parse them so we want to say cons image URLs is equal to object dot Keys Json dot parse image and this is not really an image it's rather images considering it's plural and we can say cons properly so now we're passing those images we can do it here as well it's complaining saying that it's a string or undefined so we have to add another or and then put that as an empty object so we can properly parse it because if you notice correctly this was returning an object right an object of different strings within it so we just want to get the keys so if we save this one more time but now instead console log the IMG urls it should be much more readable so let's try it one more time open this up and you can see now we get just the images so as I said once again it's all about parsing the data properly we can also get the currency by saying const currency is equal to extract currency and then we're going to pass in the dollar sign and then dot a dash price dash symbol okay but now this is another function that we have to create so let's go into our utils right below our extract price and let's create export function extract currency where we get an element of a type any and then there we can extract it by saying cons currency text is equal to element dot text Dot trim dot slice and we only want to get the first character finally if the currency text exists then we can return the currency text else we can return an empty string could we have done this right here within this function we could have but this is just a bit cleaner right we extracted it in another function so now we save it and we also want to do the currency right here just to see if we're getting it properly so I'm just going to click search once again and at the bottom in this case it should be Euro if you're doing amazon.com it should be the dollar or in India it should be a rupee so this is good and now what we can do is we can continue by getting the discount rate so that's really important when building a discount application so const discount rate is equal to dollar sign and then dot savings like this savings percentage and then we want to get the dot text dot replace and we want to remove everything that is not a percentage so how do we do that well we do a forward slash to start the regular expression within an array and then minus percentage we close the array close the regular expression say G for Global and then this so this is going to replace it and let's try to just console log it this count right right here at the end and we can try one more time and now as you can see we get 13. so this is just a number without the percentage it's going to remove it from the equation whereas here you should be able to see there we go we have 13. great so now we're getting a lot of stuff from here maybe the most important parts are going to be just the title of the product and then the current price and the difference between the prices so we figure out when is the best time to buy as well as the images on the deployed side we also have a description but it's going to be really hard to parse the description as a description of an Amazon product contains a lot of stuff so you can see you have a lot of things here and then you have a lot of images a lot of different stuff and sometimes this can be really hard to properly parse if you want to play with that that's okay give it a shot you might end up with something that looks like this right it's really hard to properly parse into unique blocks of markdown or text that is not simply just crunched together so in this case I would even just completely skip this part right here and just focus on what matters the most which are all the different prices the title the images and more so with that said this is more or less the most important part that we need right now for the product but now instead of console logging it we simply want to format into a data object so we want to say construct data object with scraped information and now here we can say const data is equal to and here you can pass anything we might need we can start with passing the URL of the product we can then pass something like a currency so we can say currency is either going to be a currency we fetch or it can default to the dollar sign something like that so we can do an or right here then we need to pass an image which is going to be imgurls or image URLs of xero then we want to pass the title we want to pass the current price which is going to be a number of a current price we want to turn it into a number same thing for the original price we also want to create a new array called price history which at the start is going to be set to an empty array but later we can update it then what else do we have we have a discount rate which is going to be a number of a discount rate and later on if you want to play with additional things try to fetch the category for now I'm just going to default it to category if you want to play with it even further and test your knowledge hey get the reviews count get the number of stars the product has like this for now I'm going to default it all to something like 100 and I'm going to do the stars of 4.5 there we go and then is out of stock we have that right is out of stock is going to be out of stock variable now let's go ahead and console lock this data to see how does it look like so all this time we're just figuring out how to extract values and how to clean them so we can then create our application based off of this clean data and there we go let me just scroll down here and you can see we get all the images we get the currency symbol the URL the title the current price as well and not too much after we have our next successful call but we still have some issues we can see that some of our current price and original price values are none so let's go ahead and fix it well here we're replacing the values but it's not always going to be the same sometimes we'll have Euros some people will have dollars and some people will have rupees and all other kinds of different currencies so we have to give our best to try to make it work for all of these different websites of course that requires a lot of work and sometimes you might have to make things work for your own country or website but I'll try to cover as much as possible and also whenever Amazon changes the way the website looks or their class names or IDs you'll have to modify your scraper too it's just how it is so for that exact reason in the description Down Below in the GitHub gist you can find the updated utils.ts which will try to keep up to date so if you notice that something is now working right now you can just paste this one right here here we have some different formatting for numbers and also we updated our extract price thing right here so this extract price just get the clean price and return it in the proper way and also we have some imports here which for now we can comment out and we also have some other functions using those inputs so we have the extract price we have extract currency we have extract description and then we have get highest price as well as get lowest price and get average price and get email notification type and all of these things use specific values coming from something known as price history item and this is nothing more than a typescript type same as product and others so let's immediately get everything we need from our Imports both the product and the price history item will be coming from ad forward slash types so let's create a new folder right here called types and then within it we can create a new index.ts in the GitHub just down below you can find the updated index.ts where you can see all the different kinds of types we'll be using such as our product type our notification type user and price history item we're going to go through all of these once we use them later on in the code but for now this allows us to import them from types and when it comes to the notification and threshold percentage these are used only in one function which is the get email notification which we're gonna use later on so for now we can simply comment it out so we can proceed with our application great so now we have our utils and the extract price should be working perfectly so if we go back right here into our scraper and reload the page and again use the same product we used before and press search and then open up the console as you can see now we have the correct price this is great and we can also get to the description so right here we can say const description is equal to extract description and then we pass the entire dollar sign and this is coming from utils right here great so now we have everything and let's ensure we're properly passing everything if we don't have the current price here you can put or and then say original price in the same fashion if you don't have the original price you can say or and then current price because Amazon sometimes uses these interchangeably and I gotta say their IDs and class names are not the best and now we can also add description is equal to description and finally we need four different values which are going to be used internally within our application and these are the values that make our application special not the current price in the original price but the current average highest and lowest this allows us to properly track how the price is changing across time allowing us to buy it at the best price possible so right here at the bottom we can add lowest price is a number of current price because we're right now right currently it is the lowest one because it is the only one right then we can add the highest price and similar this is going to be the original price and in both of these cases if the other one exists we can make it the other one so this is going to be the original and here if this one doesn't exist it's going to be the current one next we have the average price which by default we can set to something like current price or original price and this is it I know it might not seem as clean as possible but when you're parsing data and extracting information it can get messy so for one final time let's just try to call this URL and see what kind of data do we get and how does our final beta object look like so as you can see here we have the URL we have the currency the current price the original price the title category which you can extract later on if you want to price history stars out of stock description lowest price highest price and the average great so this is looking great to me and the only thing we have to do is instead of console logging it we have to return the data object right here from this scrape Amazon product function now that we are returning our new product data object we have to see where we're using it and the scrape Amazon product function is being used within our actions scrape and store notice how scrape is only one part of the goal of this function and we have done that successfully we have scraped the product so we can say if no scrape product we can exit it but if we do have the scrape product we'll have to store it in our own database and this is where things get interesting and where we really get to the bottom of the functionality of our application we want to get all of the products that the users input into our application and store them and then periodically check for the changes in the price by scraping them not on demand but periodically as I said and then we can send automatic email alerts as soon as the price drops how cool is that so the next thing we have to do here is either find the product in the database or create the product in the database which is going to allow us to keep track of the chain changes in the price so for now we know that this is working properly I'm going to keep all of these Links at the top but I can just pin them so it's a bit easier to see we also have our bright data here you should have gotten your 15 bucks of credits so don't worry that's going to be more than enough to test everything out and even to deploy your application to the internet and I'm gonna just leave price wise here unpinned and we can collapse this all the way because now we're going to start focusing on the back end of our application or more specifically interacting with the database so to do that before we actually scrape the product we can try to connect to DB and this right here is a function which we can create in another file just to keep everything nice and tidy this is going to be within a new file called Mongoose that's within the lib folder as that's going to be the data modeler we're going to use for our mongodb database so mongoose.ts and within here we can import Mongoose coming from Mongoose and as a matter of fact we can also install it so let's go to our terminal and let's run npm install Mongoose Mongoose is going to help us make a connection to our mongodb database let's just reopen this file so we can see that everything is good and then here we can create a variable let is connected which is going to be set to false by default and this is a variable to track the connection status once we have this variable we can then export const connect to DB and this is going to be an async function that's going to do just that connect us to the database so now that we're exporting it let's go back to our actions and let's import it from Mongoose and now it's our turn to develop this function to actually do its thing first we want to set mongoose's strict mode to prevent unknown field queries and we can do that by saying Mongoose that set strict query and that's going to be set to true that's going to just help us ensure that our application is working nicely then we have to figure out if we have a mongodb URI meaning a connection to our database so if there's no process.env dot mongodb URI then we can simply return a console log saying something like bongodb URI is not defined then we need to check if we already have been connected so if is connected return console log using existing database connection and finally if we are not connected and if we do have a mongodb URI we can open up a new try and catch Block in the catch we can simply console lock the error and in the try we can await Mongoose dot connect and the connect accepts the URI of the database we want to connect to and that's going to be our process.env DOT mongodb URI just like so then we want to turn on is connected to true and we want to give a simple console log something like along with DB connected there we go something like this and now what we can do is save this but we have to get access to our mongodb URI of the database we want to connect to and to get it you can head to mongodb.com forward slash Atlas this is going to give you the ability to use their online Cloud database completely for free so either sign in sign up or create a free account in this case I'm gonna connect with Google and once you're in you'll have to build a database so let's go ahead and choose the free version and you can just choose your region I'm going to keep the recommended one and click create here you have to choose the username and password I'm going to choose JS Mastery and then for password do whatever you want but make sure to remember it as you will need it later on and then click create user great so now we have two different users and then you have to go to network access add IP address you can add your current IP address but what you can do as well is ADD zero zero zero zero that's going to look like this which is going to include all IP addresses this is just to ensure that you can freely follow along with the video and have no problems accessing your database great so once that is done you'll go to database and then press connect right here you want to connect using your node.js driver so we simply need to copy this string right here that's the connection string once you copy it you can go to your dot EnV and here you can just paste it but of course we have to give it a name such as mongodb underscore URI is equal to this thing right here and then you have to enter your username which in this case is the username of the user you created and then the corresponding password I just went one two three one two three so once you have it now our application will be able to read it right here and you might need to rerun the application just to be 100 sure that it's going to notice it and after that once you call this connect to DB you should be able to connect to the database which means that we should be able to find or search for but then more importantly create new instances of documents and collections within our database so now that that is done you can go back to your application reload it and click search and then you're gonna get an error saying dot set is not a function and this right here points the dot set which is right here mortgage.set but this indeed is good we can see that this is a real Mongoose function we can call so this makes me think that we're not properly calling this Mongoose function and we can see it here but at the top I have used sever instead of use server so if we fix this now it should be working let's give it another shot by running the Search Command and opening up the console and mongodb indeed got properly connected this is great we're still cons logging the price and we don't have to do that anymore so if we go to the search bar scrape product and then we go to the scrape price or extract price we no longer have to add a console log right here this now means that we are properly connected to our database which then means that we can go back to where we started right here and after we get this great product we can either find it or read the existing in the database or create a new instance of that product in the database so we can start tracking its price so here we can say let product is equal to scraped product and then we can try to find an existing product so const existing product is equal to a weight but now we're stuck what are we calling what kind of product are we trying to find from the database is this some kind of an item or is this some kind of a document or a collection of documents in the database we don't yet know we have only created a database but the way that document oriented databases like mongodb work is you have to create a schema or a model for what you're trying to store in the database so let's hold our thought right here for our bid and let's create our first and only model in our application by creating a new folder in the lib called models and creating a new file called Product dot model dot yes inside of here we can import Mongoose from mongoose and we can say const product schema is equal to new Mongoose dot schema with a capital S and then we pass in an object which is the definition of the schema so each one of our products is going to have a URL which is going to be of a type string it's going to be required so required to true and it's going to be unique to true as well we can only have one product under a specific URL then we want to have a currency of that product which is going to be a type string and required to true we also want to have an image which is going to be of a type string since we're going to store the URL of the image and require the true then we're going to have a title of the product which is going to be type string and required to true we're going to have the current price which is going to be of a type number required true we're going to have the original price which is going to be really similar so original price and that's going to be of a type number required to true then we want to keep track of price history which is really important for our application but we have an array of different prices which are of a type number and required the true and we also keep track of the date when that price was updated so right here we can expand this object because it's going to contain the prices or rather it's going to contain a single price right so an array of prices and it's going to have a date when that price was added so that's going to be a type of date and a default of date dot now and we need to add a comma here after the price history we're going to have a few more Fields such as lowest price which is going to be of a type number but it's not going to be required we're going to also add the highest price which is also going to be type number we need an average price which is also going to be of a type number and we need a discount rate which is also going to be of a type number we need a description which is going to be of a type string we need a category of a type string as well we need a reviews count which is going to be of a type number we need is out of stock which is also going to be type Boolean and by default it can be set to false we need users which is going to be an array where we have an email that's going to have a type is equal to string and required is set to true and the default for users is going to be an empty array so how many users have selected or searched for that product and finally we want to exit this existing block which is the definition and we want to say timestamps is set to true so this is going to keep track of the changes now we have to turn this schema into a model based off of which we'll be able to create documents so we can say cons product is equal to Mongoose Dot models.product or if it doesn't exist yet we want to create one Mongoose dot model of a name product that's going to be based off of the product schema and finally we can export default the product model now that we have created this we can go back to our index and now we can await product dot find of course after we import it from Models product models and we can find one actually and we can find it based off of its URL so URL has to be scrape product dot URL and we can expand this so we can better see it now if an existing product exists then what we want to do is we want to update the price history so we can say const updated price history is equal to an array where we first spread the existing product dot price history and then we update the price by saying price is scrape product dot current price like so let's properly indent this there we go and then we want to modify the product object by saying product is equal to we first spread the scrape product and then we update the values by saying price history is now equal to the updated price history the lowest price is equal to get lowest price which is going to be coming from utils and we pass the price history or the updated price history so make sure to import this get lowest price at the top coming from utils and this price history here it's complaining that it doesn't exist so just for now we can give this updated price history a type of any and we do a similar thing with the highest price or we just call the get highest price again coming from utils and we pass the updated price history and same thing with average price which is equal to get average price coming from utils where we pass in the updated average history and here it's complaining about the average price saying that it does not exist in type and then it started giving us this type of the product but we know that product indeed has the average price if we look into it right here we have lowest highest and then average so later on I'm going to see why we're getting that error for now we know it's good and finally whether we found this product in the database or not we want to create a first instance or the new product whatever it is in the database so we can say cons new product is equal to a weight product dot find one and update so if we did find one then we can again filter it by URL is scraproduct.url then we can put this in multiple lines right here just so it's easier to see what's happening there we go the second parameter is going to be the product or the data we want to update this with so this is the entire product object and then we want to add additional options such as upsert to true and new to true so if it doesn't yet exist it's going to create one in the database great so now let's see what do we have here I'm noticing that I have to close it like this so here we first try to find the url based on the scraped URL we close this right here and then we pass these as the second and the third parameter so here we need to add a comma then we update the product and then we add new if one doesn't exist before great and finally here's a nextgs thing if you don't revalidate the path it's not going to automatically update it you will be stuck in Cache so what you need to do is you need to import something known as revalidate path coming from next cache and then we want to say this page right here which we can Define as revalidate path of template string forward slash products and then forward slash we want to go to the ID of the new product so new product.id this page is going to change because we're now modifying it great so now we are either finding a product in the database or we're creating one if it doesn't exist yet so what do you say that we tested out if we don't have any major errors the next time that we search for our product it should actually get added to the database and how can we know that well we can check the errors if there are any or if not we can check our database so let's go ahead and press search right here mongodb connected it is still searching it seems that the search went through and now if we go to our database reload the page go to our collections and you can see our test products right here and there we go you can see our first product with its description category created ad highest price in all of those great things as well as the price history the reviews count the title and more so this means that the product has been successfully created now if we call it for the second time a new product should not be added so if I click right here and then reload the database right here by clicking refresh we should only see one product still that's because if it changed it should change it right here and now we have the price history of the previous price of that product I hope this makes sense it's a really specific application where we're trying to first get the products that people are interested in and then keep them updated at all times so that we can show the new price once it changes and a huge thing you're going to learn about in this video which you maybe didn't even know is we're gonna run a so-called Cron job this means we'll have to create some kind of an action that runs independently of us running it manually we can execute it at a specific day of the week at a specific minute our interval this is so good and then we can refetch the new prices of those products by automatically scraping the data from Amazon how cool is that it's like having a personal assistant that can go through all of your favorite products and then tell you when is the time to buy cool so now we are actually saving that in the database and now that we have the data we can actually show the product details of that page on our own website so while we're here let's create an action that's going to allow us to fetch the product based off of its ID we can collapse this function and just below create an export async function get product by ID and we want to pass the product ID as its first and only parameter then we open up a new try and catch block we have to connect to our database on every call of these functions we have to connect because you can think of these as Edge functions meaning they run separately this code only gets executed once the function is called so we don't have heavy load on our server another beauty of next.js so these server actions are incredibly useful and once again going back to the course every single functionality of this huge modern stack Overflow clone is done using exactly the same system we create models in this case not one but many and then we create server functions that then act as a bridge between our server-side render front-end application and our server and actually please allow me to give you a behind the scenes of the course where we're explaining some of these Concepts so here as you can see we're dealing with the concept of web hooks as well databases and then exactly how this server side and client-side functionality works but I think this diagram explains it the best and it's the same thing what we're doing right now so first we have server side rendered Pages which do something these server-side pages in this case are our homepage right here that's a server-side page but then on it we have a client component which is search so in this case not ask a question in this case it's a search component this search component then calls a server action call in the database and then that does something in our database such as make a document or find a document it returns it which then gets back to our server-side rendered page and the reason why I was drawing this is because this is all contained within a single next gs13 full stack application if we didn't do that then you would have to have multiple applications you would have to have the front-end app you would have to have the backend app you would have to have the database right and as you can see this gets pretty messy pretty soon so yeah that's the point of using server actions they're pretty cool and that's exactly what we're doing in this video right now so to connect to our database again it's as simple as adding this line as we have already created a function and now we need to fetch the product so we can say const product is equal to a weight product dot find one where the underscore ID is equal to to the product ID if we don't have a product we simply exit out the function or return null and otherwise we return the product in the catch we can simply console log the error this now allows us to go back to our Details page fetch all the details about the stored product and display it in a nice screen just like this so let's do that next let's close all the currently open files we have many then let's go to our app and let's create our second and the last route of the day by creating a new folder called products within it creating another folder called square brackets ID and within that ID creating a new page.tsx and running rafce this is going to create a sample page which is going to be product details now if we save this and go to some kind of a forward slash products forward slash and then enter some random characters you'll be able to see the product details there we go so now first we're going to finalize the home page by showing all of these new trending products that all of our users have been adding and then we're going to implement a connection between clicking this and opening a product page so we can continue where we left off by going to pages going to our home page and then here now we can Loop through all the products real products from the database and then go to their corresponding detail pages so let's do that next within our server actions that are in the lib and then actions and then index DS we have created get product ID and scrape and store product but now we need to create another one which is get all products so we can do a similar thing export async function get all products and then we just render it like this open up a new try and catch block and then connect to DB as we usually would so just another call to the function we have created before now once we connect this call is the simplest of them all we simply say Khan's products is equal to a weight product dot find and we call it like so and then return the products finally if we have errors we can for now just console log the air object and see what we get so now that we have created this function and that we are exporting it we can import it within our page right here at the top we can say import get all products coming from lib actions and we can utilize it right here on the top as if we're calling a regular function but this is a special server function that's making a call to our database but next.js makes it so simple there's no use effects there's no States the only thing you have to do is say async and then fetch it right here by saying const all products is equal to a weight get all products and that's it you have access to it so now we can Loop over instead of this demo fake array we can Loop over all products which are now coming from the database we can hover over it and it says that all products is possibly undefined so we can simply add a question mark right here thank you typescript for saving us so now we save this and if we go back to our current application it looks like something broke so I'm just gonna press Ctrl C and then Y and then re-run our application and we get the same error again which means that we are stuck in some kind of a loop maximum call stack size exceeded we never want to see this type of error so let's see what mistake did I do uh it looks like it's happening right here and it's not really clear what this error is referring to so we're fetching the products we're getting all products from here this is a separate function connecting the database running the products dot find and that's it we have to figure out when did this error start appearing so if I comment this out and if I comment out all of the products right here and save it is the error going to go away it well interesting so now once again if I bring it back we again have the error okay this is interesting so this gives us some data to work with and I'm really glad it happened because my goal is to include as many errors as possible throughout the process of teaching you how to develop this application so maximum call stack exceeded let's figure it out together usually this happens when you have infinite Loops but in this case oh I think I know what it is but it's weird that it resulted in this error um before our product was just a string right we have something like a test right here and then the product was simply a product which worked because we can simply render a string right here but now a product is much more complex it contains more aspects so instead of doing just product here we can do a product.title which is then going to make more sense and hopefully render our Apple MacBook Air so this is now good again it was weird we got an error that is usually happening for infinite Loops when we just didn't put a dot title there I'm guessing that's because mongodb has a special way of creating its documents in the database so this is not just a typical object it contains some more complex properties which caused nextgs to just black out so this is now good but we don't want to Simply return a title we want to return a card that has more information about each individual product so let's go into our components and let's create a new file called Product card dot TSX we can run rafce import that product right here by saying product card and it can simply be a self-closing component like so the only thing we have to do is pass in a key equal to product dot underscore ID and a product itself there we go so this is it and now we can import the product card and now we show a one single product card make sure to put this question mark right here just to ensure that we don't have any errors and looking at the product it's saying that right now this card doesn't accept any such props but if we go into the product card we can now Define what we accept within that card we can say that we get a product and this product is going to be of a type props so right here we can Define interface of props and the most important and the only thing is going to be a product of a type product which is coming from our types so now it's no longer going to complain right here we can accept all of those product props and we can start displaying them on a product card which is right here on the right side so instead of simply saying product card let's actually turn this entire thing into a next gs13 link component and the link of course has to have an href this is our Bridge To The Details page here we can say go to forward slash products forward slash and then product dot underscore ID we can also give it a class name equal to product Dash card within it we can render a Dev and this div can render an image the image is the next gs13 image tag and we can give it a source equal to product dot image and then I'll tag equal to let's do product dot I believe it's title and if I were to type name here you can see how typescript lets me know that hey it's actually title let's give it a width of about 200 and a height of 200 as well as a class name equal to product Dash card underscore IMG so if we save this you can now see this wonderful looking image let's style the diva bit to make it a bit smaller by giving it a class name equal to product Dash card underscore IMG Dash container and now it's going to be contained really nicely we can also dive into this container a bit by going to the search and pasting the name and I believe we don't need to add this BG white 100 we can just leave it white as it is as most Amazon products require completely wide backgrounds so this is going to make it more nicely blend in so now below this div containing the image we can also create another div that's going to contain the rest of the content so let's create a class name equal to let's do flex and flex call as we're going to show the content in one line and then a gap of three then we can render an H3 that's going to render the product dot title and let's save it there we go looking good but we can style it a bit by giving it a class name equal to product Dash title there we go next we can give another div for more descriptive content so a class name of flex and justify between then we can render a P tag within it this P tag is going to render the product dot category and in all cases it's just going to say category for now and we can also give this B tag a class name equal to text Dash black opacity dash 50 and text Dash LG as well as capitalize for now it's simply going to say category but if you figure out how to extract the real category from Amazon then you can just fuse it right here next below the speed tag we're going to have another this P tag is going to have two different span elements the first span is going to render the product question mark dot currency and the second one is going to render the current price if we save this you can see it's looking a bit better we can also do a class name equal to text Dash black text Dash LG and fond Dash semi-bold so now this is looking better and again on mobile it's looking good but if we expand it it's just going to be one card in a row looking wonderful as well and with that believe it or not we're done with our product card and once we click the card we go right here in another URL we visit products and then the ID of that specific product which brings us right here to this new product Details page we have created but now the question is how do we get access to the ID of this product from within this component or page well we get it through params so here we can Define params and this is done for us by next.js we can say props and that's going to be of a type props it's going to contain params of ID is of a type string there we go we have declared a type and now we can get the ID by immediately destructuring it from params right here and we can display it within this div so if we do that now we can see the ID of our product which means that we are ready to fetch all of the product details based off of that ID see it ends with 974-723 and if we go here 974-723 we are ready to make a call to our database fetch all of these values and then nicely display them just like this so let's do that next I'm going to collapse this and we are ready to make a call do you remember that get product by ID function that we created not that long ago well now we can put it to use by saying const product is equal to a weight of course we have to turn this into an async function since we're using a weight and then we can say get product by ID by importing this from lib actions and we can simply pass over the ID then if we don't have a product we can simply redirect to forward slash in this redirect is coming directly from next navigation so we can import it from there in this case we indeed do have a product so all of the details about that product should be in there how can we know well let's try to render the product.title now that we have the product and there we go you can see it here which means that the rest of the stuff is here as well so let's go ahead and create this great product details section let's first start by adding a class name to this outer div a class name equal to product Dash container if we save it that's going to already space it up a bit and within there we can remove the product title and we can create a new Dev within this div we can place another div and within that div we can place a self-closing image tag here we can give it a source equal to product dot image and we can of course import image from next forward slash image the image also needs to have an ALT which is going to be product title it needs to have a width equal to 580 pixels and also it needs to have a height let's do something like 400 in the class name of MX for margin X of Auto if we save this we should be able to see a nice looking image appear now let's add class names to Outer divs the first outer Dev can have a class name equal to flex gap of 28 to provide some spacing on extra large devices Flex row but usually Flex call as we want all the elements to appear in a column and then this div can have a class name equal to product Dash image if we save this you can see it just puts it in this nice looking card and now we can go below this div containing the image and we can create a new container div for our content so here we can have a class name equal to flex-1 flex and flex Dash coal within it we can continue creating the structure of our application so we'll need to have a structure for this stop part and then a line and then the bottom part right there so first we're just positioning everything right here on the screen and on the large screen this is going to look like this we're going to see some products on the left some on the bottom but on smaller devices these are going to go on the right and on mobile devices they're again going to go on the bottom so to create that structure we have to have a div and that div is going to have a class name equal to flex justify Dash between items Dash start gap of 5 Flex of wrap and padding bottom of six within it we can create one final container div that's going to have a class name equal to flex Flex Dash call and a gap of three within there we can finally render a P tag and this P tag is going to render the product dot title if we save it and go back you can see it right here let's make it look a bit better by giving it a class name equal to text Dash 28 pixels to make it larger text secondary and font Dash semi bold and already this is looking more like a title and less than a paragraph now below that of course we want to render a link that's going to allow us to visit the product so let's do it right here below the P tag it's going to be a link component which we have to import from next forward slash link we can give it an href which is going to be product dot URL we wanted to open in a new tab so we can say Target is going to be equal to underscore blank and we can give it a class name equal to text Dash base text Dash black and opacity of 50. finally within it we can do visit product if we save it we're going to have a nice looking kind of a button right great now we want to go below this div and you want to start creating the second section which are these Hearts right here and other three icons so let's create a div that's going to wrap them let's give it a class name equal to flex items Dash Center and the gap of three also we're gonna give it a div and that div is going to have a class name equal to product Dash hearts and within it we want to render an icon which is going to be in form of an image that's going to have a source equal to forward slash assets forward slash icons forward slash red Dash heart dot SVG an altag of heart a width of 20 a height of 20 as well so if we save this you should be able to see this little heart and right next to it we can render a P tag with a class name equal to text Dash base font semi-bold and text Dash that's going to be d46f77 this is this nice pinkish color that's going to render I believe its product dot reviews or maybe it's number of reviews yeah this shows nothing but what if I go number of reviews no it's not that let's try to figure out what do we have in the product and here we didn't Define the type of this product so we can say product of type and then here we can get the product from our add forward slash types and now it's going to immediately complain see number of reviews doesn't exist but if we go into the product we can see that it should have let's see we're looking for reviews count exactly and this is how again typescript makes your life easier especially on large applications and now we should see something like 100 okay great let's go a bit below the product hearts in another div create a second div that's going to have a class name equal to p-2 BG white 200 and the rounded of 10. this is going to create this nice border or just the circle for now and within it we can render an image that's going to have a source equal to forward slash assets forward slash icons forward slash bookmark dot SVG we can give it an ALT tag to bookmark a width of 20 and a height of 20. that's going to make it look like this if you want to feel free to implement that functionality to maybe like it save it to local storage or do anything else that would be pretty cool and whenever you do make any updates the applications that you're watching on this channel definitely make sure to tag me on LinkedIn X or any kind of other platform I'll be there and I'm excited to see the changes you've implemented finally let's do a similar thing for share so I'm going to duplicate this right here below and instead of bookmark it can be share and share right here and that's going to look good wonderful now we can go one two three divs down and we can create one final div now we're diving into the class name of product Dash info so here we're really starting to do the gist of the product what is it for what is it about and most importantly the prices so let's hop on right in into the product info let's create one more div to help us with the layout and that div is going to have a class name equal to flex Flex Dash call and a gap of two within it we can render the product price so let's do a P tag that's going to render product dot currency and then we want to render format number which is coming from utils and then pass the product.current price into it and close it all properly that's going to give us 939 dollars we can put this a bit below and we can style it by giving it a class name equal to text-34 pixels text Dash secondary and font Dash bold if we save this that's big we can duplicate this below the bottom one can be a bit smaller like 21 text Dash black opacity dash 50 and we can do line through okay so this is the crossed out price and this is going to be price currency and then original price so in case we have a discount that discount is going to show up right here crossed out there we go in this case is the same price now we want to go one div down and then create one more structure for the number of stars that this product has again something you can implement we can write different products based on the stars and the number of reviews here we want to give this div a class name equal to Flex Flex Dash coal and GAP all four again to vertically show what we have another div that's going to have a class name equal to flex and a gap of three and within it a final div that's going to contain the image and this div is going to have a class name equal to product Dash Stars right within it we can show our image all if you can guess it stars so here we can do a source equal to forward slash assets forward slash icons four slash star dot SVG give it an ALT equal to Star a width let's do about 16 and a height of 16 as well and that's going to give us this nice Star right below it we can render the number of stars so let's do a P tag and we can render product dot stars there we go that's going to be nothing for now so we can do some kind of a default number like let's do 25. there we go we can style the speed tag by giving it a class name equal to text Dash SM text Dash primary Dash Orange and fond Dash semi-bold and this is going to just make it look more uniform finally we can go one div down we can create product reviews so let's create another div that's going to have a class name of product reviews and within it we can render a new self-closing image tag and this tag is going to have a source equal to four slash assets forward slash icons forward slash comment dot SVG here we can see alt tag of comment the width is 16 the height is 16 and as before we want to render a P tag below it that's going to render the product dot reviews count and then say reviews if we save it that's looking good but we can style the speed tag a bit by giving it a class name of text SM text Dash secondary and font Dash semi-bold there we go that's looking a bit better finally we can go below this p and Below two more divs and create another P tag within that P tag we can create a span element that's going to say something like 93 percent and span of buyers have recommended this again this is something we're hard coding right now but it would be really cool if you can fetch this and parse it and scrape it directly from Amazon and then implement it right here so let's do a class name of text Dash SM text Dash black and opacity of 50. and now we can modify this span by giving it a class name equal to text Dash primary Dash green and fond Dash semi bold this is just going to make it stand out a bit more finally we can go below two more divs and now we want to focus on the most important aspect of our application which are the price cards so we're gonna have four similar cards which most likely means we're gonna have to create another reusable component right but first let's create a div that those cards are going to sit within that div is going to have a class name equal to margin y of 7 to divide it a bit from top and bottom Flex Dash column so they appear in a column and a gap of five to give them some space after that we want to give it a div that's going to have a class name equal to flex Gap Dash 5 and flex Dash wrap and by the way guys once you start writing a lot of UI like this I can often speed it up and as you notice right here I've been going really fast through all of this UI so let me know if this is too quick for you to actually code out how do you follow along with these videos do you have to pause it and then watch it do you just slow me down do you speed me up is this the right speed would you like me to speed up more or slow down feel free to let me know down in the comments your feedback is truly appreciated and based off of that I'll note how to Pace myself a bit better for better education great so with that said within here we can render our first price info card so let's create a new reusable component called price info card dot TSX we can run rafce and we can import this price info card right here within this div it's going to be a self-closing component and to it we'll have to pass a couple of properties of course after importing it the first property we can pass can be title and title is going to be current price we can pass an icon SRC equal to forward slash assets forward slash icons four slash price Dash tag dot SVG we can pass the value which is going to be the actual dollar amount or any kind of currency amount so that's a template string of product dot currency and then after that format number and then we pass the product dot current price so just like that we can expand this a bit further so we can see it and then finally we want to pass the Border collar since we're going to have multiple in this case we can do something like hash B6 dbff that is this color that you can see right here it's barely noticeable but we can also use it for this icon that you see right there but more in that later so for now we're passing all the necessary values and we can go into that price info card and actually make use of the props we are passing so let's immediately get them things such as the title icon SRC value border color and we can Define all of these of an interface props which we can declare right here by saying interface props and then say we want to get the title of a type string we want to get an icon SRC of string value of string and Border color off string and now we can start creating this price card so you can see just price info card I'm just going to collapse this a bit so you can see it a bit better there we go and we want to give this div a class name equal to let's make a dynamic template string of price Dash info underscore card and then we can say border Dash L Dash and then we can pass the Border color like so but it has to be wrapped in square brackets for the color to take effect what we can do then is instead of simply saying price card we can render a P tag and this P tag is going to render the title so in this case this is going to be current price but we can style it further by giving it a class name of text Dash base and text Dash black Dash 100. okay and below that we can render a div that div is going to have a class name equal to flex and a gap of 1 to provide some space and we can render a self-closing image tag with a source equal to Icon SRC alt equals the title of the card width of 24 and a height of 24 as well this is going to give us an image which we of course have to import first from next image there we go we can see it here and finally the most important part which is a P tag that's going to render the actual value but of course let's make that a bit bigger by giving it a class name of text-2xl so much larger font bold and text Dash secondary there we go so now this is how it should be and of course we now want to reuse this multiple times so if we go down I can now duplicate this one two three more times and save it and would you look at that we immediately have four different cards but now we actually want to change the values so in the second one we want to provide a chart SVG and it's going to say average price and also product dot we want to get the average price as well same thing for the third one we don't want to do price tag we want to do Arrow up and this is going to be the highest price so product dot highest price and finally we have the lowest price so this is the arrow down and here we can do product.lowest price and as you can see all four of these icons have changed let's try to change the Border color for example for the last one let's do b e f f C5 and if we click it nothing changes so let's see why the card is not taking into effect the color so let's see we're doing border Dash and then what if I simply copied this value into here usually Tailwind does work like that you simply say border something and then you pass the cooler and as you can see now it works but if I try to do it dynamically it doesn't actually take effect of the oh now it did okay that's interesting so I had to first pass it manually what if I now change it for this one for example FCC it's not gonna do it yeah so Tailwind sometimes cannot take Dynamic values we really have to work hard to make it happen it is possible but in this case we can just eliminate the Border collar and keep all four borders as they are let's save it and let's eliminate the Border collar from all the props right here and we have a wonderful looking interface now the actual value here is the same for all of these different properties because we didn't actually recall our function maybe the price has changed up to this point but we need to recall the same link on our home page to update the database or do we that's the thing it wouldn't make sense right that you have to manually recall it if you had to do that why not just go to Amazon directly and check the price there the point of this is that soon enough in this video you're going to implement Quran jobs which means that things are going to happen periodically after some time which is going to provide a lot of additional functionality to our application great so now we have these below that we want to show a track button once you click it we're going to have some kind of a modal to alert this of the price and then let's just quickly finalize the rest of the things right here so the modal is going to go two divs down which is going to be a separate component but more in that later and then we want to go two more divs down and want to focus on the final of the divs which is going to contain the product description buy button and all the other products so for that let's create a new div that div is going to have a class name equal to flex Flex call and a gap of 16. and just to know what we're working with let's again do a border of L and then let's do let's do black if I save it and go back you cannot see anything let's try border 2 border red if I save this you can see some kind of order but I need to add 500 right to make it actually red there we go so now you'll be able to see how this div changes as we create new elements within it the second element is going to be a div with a class name equal to flex Flex Dash coal and a gap of five still nothing because it doesn't have any height but as soon as we add an H1 within it that renders the product description you'll see how nicely the elements will fall within it this H1 is going to have a class name equal to text Dash to excel and text Dash secondary as well as fond Dash semi-bold there we go although it wouldn't be ideal if we call this an H1 because it's not the most important piece of information on the screen the product name is so let's just do an H3 below that we can render a div that's going to have a class name equal to flex Flex Dash coal and a gap of four and within it we want to open a dynamic block of code and say product question mark dot description and then question mark dot split and want to split it by a new line character which looks like this so we're going to show all the different rows of a description in this case it looks like it didn't pick up too much it actually pick up uh just the breadcrumbs but for some other products it can pick up actual descriptions we can play with this later on and scrape it in a better way finally we have a buy now button which is this so let's create it below the description we can show our buy now button so let's go back right here go to divs below and right here we can render a button this button is going to Simply render an image this image is going to have a source equal to forward slash assets forward slash icons forward slashback.svg and then I'll tag of check it's going to have a width of 22 as well as a height of 22 and if we save it you should be able to see something right here but it's currently white so it's really hard to notice so what we need to do is add some color to this button by giving it a class name equal to BTN W Dash fit MX Auto for margin X items that are Center justify Dash Center gap of three and a Min width of 200 pixels once we do that we should have a regular looking button and we can also add a link right here within it and this link is going to have an href and it's going to point to four slash for now we can also give it a class name equal to text Dash base and text Dash White and there we can say something like buy now we could Implement a functionality to even link to multiple websites advertising the same product this is to make our app even better and then turn it to a paid SAS product finally the last thing we have to do is just go below the button and Below one more div and want to Loop over some more products and to be able to show these products we have to figure out what are similar products to the product that we're currently viewing so for that we'll have to go into our actions index.ts we have our scrape and store product get product by ID and then get all products but now we'll want to also do something similar to get all products so let's simply duplicate it below and let's call it get similar products and this function needs to accept a product ID of a type string then we need to fetch the current product by saying const current product is equal to a weight product dot find by ID and we pass in the product ID if we don't have a current product we return null and finally we need to fetch similar products by saying can't similar products is equal to a weight product.find where we find all of the products where the ID is dollar sign not equal to to the current product ID and will limit it to something like three by saying dot limit of three and now we can return the similar products that are not going to include the existing one we can go back and we can use it right here by declaring it at the top of this page so on top of getting this product we can also try to get const similar products is equal to a weight get similar products to which we simply pass the ID and of course we have to import get similar products from lib actions once we do that right here we can check if similar products Exist by saying similar products dot length is greater than zero and we can add a question mark right here just to be sure that they actually exist if that is the case we can say and and then render a div right now it's going to complain that similar products is possibly undefined so first we need to check if similar products exists in the first place if it does then we can check if it's greater than zero and then if it is we can show those similar products so we can wrap everything in a div that has a class name equal to padding y of 14 to provide some space on top and bottom Flex Flex Dash call Gap of 2 and W of full for full width we can put a P tag in there and say similar products and of course we want to style this a bit by giving it a class name equal to section Dash text and now we should be able to see it somewhere here on the bottom although I cannot yet notice it it seems like we're not getting any similar products back and of course this makes sense because this is the only product that we currently have but nonetheless let's quickly finalize the similar products section because it's so simple let me tell you why we create this div and that div is going to have a class name equal to flex Flex Dash rap gap of 10 margin top of 7 and a w of fool and the only thing we have to do in there is just render similar products dot map where we get each individual product and what do we return well we return the product card that we have already used on the home page by importing it from components product card and passing the key equal to product dot underscore ID and passing the product equal to product and now once we do have some products in the future they will be shown right here at the bottom for now what we can do is remove this red border so we can remove it from here that's line 157 on my end border 2 border red 500 and with that said we have a nice looking product Details page that looks like this where we have the description and we have all the most important details about this product and we can even visit it right here to get to the location where we originally found it but of course this is not the whole point of the application right we don't want to Simply sit here we want to actually track whether the prices are changing and for that reason as I said multiple times so far we're going to implement cron jobs which are essentially functions that are going to be executed within a specific time frame allowing us to refetch and refetch new data on demand or on schedule but before we do that let's just add another product so we can test if everything is working well with our application I'm going to go to Amazon maybe today's deals and we can select something from here such as these Jabra Elite 7 earbuds here we even have a huge discount so let me select it and paste the link right here and click search something happened we don't see anything in the console which means that it's good I hope there we go and now I'm guessing that another product appear right here and it did we can see that we have these great earbuds and if we click them it's something breaks and this is coming from the format number because it cannot reach properties of undefined so it looks like whatever we're passing into the format number is getting right here as undefined so let's see where are we calling the format within the product Details page just by running a control F it seems like it's not getting cold oh no there we go it's being called a couple of times so let's see yeah one of these values is undefined and we need it to be a number so a cool thing we can do here is instead of Simply providing a zero in each one of these cases we can go to the format number and just set the default value so we can say right here num of type number is equal to zero and this is going to provide it a default value so it is a number after all and here you can see we're getting the average price of zero but that's totally okay we can get the averages later on once we can actually average something out for now what we have is the current price of 79 which is the only thing we are interested in and I just noticed I didn't change the labels on all of these other ones so let's see the first one is the current price then the second one is going to be average the third one is going to be the highest and the fourth one is going to be the lowest so if I fix this now everything should be good and indeed the lowest is the current which means that it's good time to buy and the highest was 180 and we can keep tracking that to see how it's working and notice we have similar products in the bottom and now we can move between those products how cool is that so now we know we're properly scraping all the data and let's implement the second most interesting part of our application after of course all the great scraping which is going to be cron jobs so we actually want to recall this from time to time to figure out if the price has actually changed and based off of that maybe send an information to our email so we know that it's a good time to buy so our cron jobs are actually going to be sending emails which is pretty cool so let's go ahead and implement this model first this model is right here we can turn it into a self-closing component and just create it as a new component called model dot TSX run rafce and then import it right here great now we can close all of the currently opened files besides the modal and we can focus on implementing this great looking model which is going to be the last step before we are ready to actually do the Cron job so let's put this to the side let's go on one of these products and let's implement this model right here to implement the modal we can first start with a button that's going to trigger it so let's wrap that button in a react fragment and create it right here by saying button and this is going to be a button of a type is equal to button and it's going to have a class name of BTN and within here we can simply say track now later on we're going to give it a non-click property that's actually going to make it work but for now we just have this track button and below it we want to use something known as a dialog and this dialog is going to come from something known as headless UI which is a completely unstyled fully accessible UI components and here we can find a dialog there we go a modal and we simply want to use it we click it it opens up so you want to install npm install headless UI react so you can run it right here and then here it shows you how to use it we'll have to import use State and the dialog at the top and feel free to refer to these docs as well since we're using use State we definitely need to turn this into a use client component so let's do use client right here and then we can copy all the code from here we can first copy the let is open and then the entire dialog that's right here which we can paste below the button now we can indent it properly and if we save this we can implement the on click functionality on this button so what we want to do is just give this button an on click which is going to say open model and want to create a function const open modal is equal to a simple arrow function that simply sets is open to true and we can also create another one that's going to be called I think you can guess it const close model set is open to false so now if we do this and go back you can see that if we click on track something opens up right here at the bottom but that's not yet a full active modal we want to show it on top of the screen nicely so to be able to do that we want to wrap it in something known as a transition that's going to give it a slow animation of opening so you want to copy this entire dialog and put it within this transition property or element which is also coming from headless UI react so you can import it there you can give it a property of appear it's going to show when it is open and it's going to appear as a fragment this fragment is coming from react so you can import it at the top now if you save this still it's not going to show up on top that's because we have to say dialog as div it's going to show as a div it doesn't have to have the open property because transition has it but it will have on close and we can call Simply close model right here close modal which is a function we created and it's going to have a class name equal to dialog container if we save this you can now see that everything gets darkened and we have some content overlaying the rest of the content so now we want to wrap everything in a div and as a matter of fact we want to remove most of this dialog panel content because we're going to recreate it by ourselves so we're going to start from an empty slate within our dialog we're going to have a div and this div is going to have a class name equal to Min Dash hash screen paddingx of 4 in text Center this is going to allow us to nicely display our transition children right here so you can open up a new block saying transition dot child and within it we're going to render the dialog dot overlay this is going to help with the actual animation so we can give it a class name equal to fixed in set 0. so if we save this now still not a lot happens if we click track but if we add some additional properties to this child such as as is equal to fragment we can do the enter equal to ease out duration of 300 we can also do enter from equal to opacity zero this is just how headless UI does animations enter 2 opacity 100 leave ease in duration 200 leave from opacity 0 and leave to opacity 1. so now if we do this and click track well we still don't see it but soon enough it's going to slowly start to animate in exactly as it does on the finished site of course once the modal is actually done so now we have this dialog overlay and we can go below the transition child and create a second transition child which is going to act as the actual model so let's go here and let's create a transition dot child we can also set it as as it's going to be a fragment we can enter of ease out duration 300 we can enter from opacity 0 and scale 95 so it's going to be a bit smaller but then we're going to enter two full opacity fully visible and scale 100 once we leave we want to ease in duration 200 and want to leave from opacity 100 scale 100 to leave opacity 0 scale 95. and within here we can render a Dev and that Dev is going to have a class name equal to dialog Dash content and if you type test within it we should be able to already see it so if I click track there we go it slowly appeared and slowly went away so now let's style the actual model let's add a few divs inside to do the layout of the model and we can do that by giving it a class name equal to flex and flex call we can do another one within it this one is going to be div with a class name equal to flex and then justify between and finally we want to have one last one which is going to wrap our image so this one is going to have a class name equal to p-3 border border Dash gray-200 and rounded of 10 and within it we can render our self-closing image tag and this is simply going to be for the logo so we can say source forward slash assets forward slash icons forward slash logo dot SVG alt is going to be logo width is going to be something like 28 and height is going to be 28 as well and we can import image coming from next image if we do this and save you can see our price logo which is good but now why is this not centered in the middle of the screen well with headless UI there's a little trick that you can use to Center the modal contents so above this transition child right here you can do a span element this span element for now can render just a string of test but what matters more is what it's going to have within it it's going to have a class name equal to inline patch Block H dash screen so full height and align middle and also a area of hidden is set to True okay let me test it one more time so I'm going to click test and now it actually works I don't think we need this test within it I think we can just put the automatic close so it's just a span that has the Align middle within it and if I do that it is nicely centered so now let's start adding stuff into it so right here next to this image where we have the logo below the dev containing it we want to add an image that's about to close this model so we can add a source equal to forward slash assets forward slash icons forward slash x-close.svg with an ALT tag of close a width of about 24 a height of 24 a class name equal to cursor Dash pointer so we know it's clickable and an on click property of close modal so now we can actually close it we can open it we can also click outside to close it it's looking good now below this div we want to show an H4 and this H4 is going to say something like let's copy it from the finished site stay updated with product pricing alerts right in your inbox something like this and again you can visit the deployed link it's going to be price wise Dash 10.versel.app below this H4 we're gonna also have a P tag and this P tag can say something like never miss a bargain again with her timely alerts we can paste that content within it as well finally below this div containing the P tag we're going to have a form element that's going to have a class name equal to before we add that let's go back to our current website we can give it a flex Flex Dash coal and Mt of 5 so divide it a bit from the content and we can give it a label this label is going to be an html4 email so here we'll be able to enter our email and let's give it a class name of text SM fun Dash medium text Dash gray-700 and there we can simply say email address so if we save this you can see email address and of course we have to add an input below so let's first create an input container by wrapping it in a div with a class name equal to dialog Dash input underscore container and it's going to have a self-closing image for search so we can say something like source is equal to forward slash assets forward slash icons forward slash mail dot SVG it's going to be an out of male with a width of about 18 pixels and a height of 18 pixels as well if we save it you can see it appear right there and then we need to add an actual input which is going to be required it's going to be of a type is equal to email of an ID equal to email it's going to have a placeholder you can put something like enter your email address and a class name equal to dialog Dash input let's see how does that look like okay this is great and below this div we can create a button that's going to submit it all so it's going to be a button with a type it's equal to submit and it's going to have a class name equal to dialog Dash BTN finally it can say track okay this is looking good everything besides these text elements so we can go up the H4 can have a class name equal to dialog Dash head underscore text and the P tag can have a class name equal to text SM text Dash gray-600 any margin top of two if we save it we have a great looking modal right here and the last thing is to implement the form in the functionality to actually be able to do something with this email so what we can do is create a couple of use State fields we already have is open we're going to need one for the loading state which we can call is submitting set is submitting and the start set to false and we need one for the input so we can say use state and we can call that email set email at the start set to an empty string and finally we'll need a const handle submit function which is going to be an async function just like so so now we can actually call these within our elements let's go down let's first deal with our form the form is going to have an on click or rather on submit is going to be handle submit so we can call this function but now our input is going to have a value equal to email and on change we simply want to get an event and we want to set email to be equal to e-target Value which is the value that the user has typed into this input so now we can actually update this and the key question is what happens once we click track and this brings us all the way to the submit if we are currently submitting we don't want to say track so we want to check that if is submitting we can say submitting else we can say track once we click it this is going to trigger the handle submit function which is right here so how is that functionality going to look like to actually send a real email and then keep track of the updates let's do it one step at a time first thing is we're getting a submit event right here so we can say event is of a type form event coming from react specifically it's an HTML form element and we can Define that as its own type like here once we do that we can say e dot prevent default as we don't want to reload the page and set is submitting to true and then we'll want to call a function that's going to add user email to product so what does this mean it means we want to update our user about a specific product and to this function we'll have to pass the product ID and the email of the user okay and this is going to be an asynchronous function so we have to add a weight in there finally after we call it we want to set is submitting to false or not submitting any more we want to reset the email as well to empty and finally we want to call the close modal so now if we do this if we enter a valid email address such as ASD gmail.com and click track it's just going to close but now it's our turn to actually Implement one I believe final action in our course in our application which is add user email to product so let's go to our lib actions and Below scrape and store products and Below all of these other ones we can create a final export async function add user email to product which accepts a product ID of a type string as well as a user email of a type string and in here we can open up a try and catch block as we usually do in the catch we're going to Simply console log the error but in here we want to actually send out our first email so let's say send our first email which is exactly what we can do next inside of here we want to check for which product are we trying to send the email so we need to get the product by saying cost product is equal to a weight product dot find by ID and we pass the product ID then we check if there is no product we simply exit out of the function then we need to check for the user if the user already exists on the list of the users that are tracking that product so we can say const user exists is equal to product dot users dot sum and then user of a type user which has to be imported from types and if user.email is triple equal to user email if that is the case the user exists so then what we can say if no user exists then we can simply run product dot users dot push and we pass the email where it's equal to the user email we save that product by saying a weight product dot save and then we generate the email contents by saying const email content is equal to generate email body or pass the product and then this is going to be of a type welcome finally we have to create this function that's going to generate the email body so let's create a new folder within the lib which is going to be called node mailer and within it we can create a new index.ts within it we can first import something known as node mailer coming from node mailer and of course this is a package that we have to install by running npm install node mailer which we're going to use for sending emails from our next GS application then we want to export const notification type so just notification is equal to and we can have four different ones it can be welcome of a type welcome it can be a change off stock as well which is going to be change of stock it can be lowest price so the change in the lowest price which is going to be lowest price and then threshold met this is when threshold is met when the price drops below a specific level so these are just variables that are going to indicate four different types of emails that we can send and then we can create a function export const generate email body is equal to a function that accepts two things it accepts the product which is the email product info as well as the type which is going to be the notification type right here and that's coming from ad for slash types now that we have that we can start preparing the content which we can send via email so first we can get a shortened Title by saying const shortened title is equal to if product.title.length is greater than 20 characters then we can say product dot title dot substring of 0 to 20 and then da da da because it is too long else we can simply render the product.title we can also declare a subject of the email as an empty string and then the body of the email as an empty string as well and then we can open up a new Switch Clause where we get a different type of the email we want to send it can be one of these four and if the type is notification dot welcome which is the first one we're sending then we want to declare a subject right here is equal to something for example welcome to price tracker and body is equal to a template string where we also add something in there and this can be HTML something like welcome to price tracker and so on finally we need to break and we need to repeat this four different times four four different emails for the change of stock lowest price and so on and actually I generated all of these using cha gbt so right below you'll be able to find this in entire generate email body function in the GitHub just down below you can copy it and simply override the one that we currently have everything else is going to be the same but here you'll just notice that we have the case for the change of stock for the lowest price for the threshold mat and then also for the notification welcome where we send the title and then we send some generated HTML that talks a bit about the product this is it and for this one threshold met we'll also need to Define this threshold percentage which at the top we can Define as export const threshold met of about 40. so if it drops 40 down then we can do it and also threshold percentage it looks like I misspelled it this is how it's going to look like great so now we have our generate email body function and we're importing node mailer let's just see if it's installed it seems like it cannot find it but it was installed indeed node mailer okay we'll see what that's about really soon for now we can go back and we can actually import the generate email body function from node mailer this is going to provide us with the email contents based on the type of the email such as welcome and then most importantly we have to run a weight send email where we pass in the email content and then in the second parameter inside of an array we pass the user email just like so and now we can get into the node mailer file and create a new function export const send email is equal to an async function like so and we can get the things that we pass in such as the email content of a type email content which is coming from add forward slash types as well as a send to which is going to be a array of strings so we can say string array like so now inside of here we can declare some email options so we can say const mail options is equal to an object where we have from so who are we sending from we're going to get this from environment variables soon enough we have the two who are we sending to and we know that already it's going to be an array of send two then we have the HTML which is going to be the email content dot body and then we have the subject which is going to be the email content dot subject so we're missing only one single thing who are we sending it from and for that we have to define the method of transfer so right here above the send email we can say const transporter is equal to node mailer dot create transport and we pass in an object we have to define a pool which we're going to say is equal to true and if you want to learn more about these options you can go to the node mailer documentation and here you can learn more about the SMTP email transfer so it's a protocol for sending emails and then here you can Define different options the pool configuration and more the port host auth and the auth method and we're going to need all of these things to make this work I try to find the simplest service to use and believe it or not the service of Hotmail is much simpler to use for such transfers than Gmail Gmail has a lot of security privileges that you have to go through then we have to have the port of two five two five we have to have the auth and then here you'll have to pass your user as well as your pass which I'm going to teach you how to create really soon and then we can Define the number of Max connections which is for now going to be one so now we have almost everything ready to send our email besides who are we sending it from and the auth of our node mailer SMTP account so let's make that happen here immediately in node mailer documentation we have the using Gmail article that says that even though Gmail is the fastest way to get started with sending emails it is by no means a preferable solution unless you're using the oauth to Authentication Gmail expects the user to be an actual user not a robot so it's really hard to make that work so for that reason we can just set up node mailer using Hotmail so here you can just figure out how to do that there are many different articles but of course I'm gonna teach you how to do it in this video the only thing we need to do is get these credentials from auth user and password and you can get to those by creating an Outlook account I know I know who uses Outlook but in this case we'll need to get it to be able to automate the process of sending emails and by the way no hard feelings if you are an Outlook user so let's go ahead and create a free account you can choose a new email I'm gonna do JS Mastery at outlook.com okay that already exist somebody's trying to steal our identity I'm gonna try JavaScript Mastery in this case and then you can create a password once you log in I believe this is it you'll have a new Outlook account which you definitely didn't sign up for when watching this video but now you can actually use it right here within the auth of the Hotmail so here in the auth and the from you can add your exact email so this is going to be your email account so if we go to my profile I believe we'll be able to see it there we go JavaScript Mastery at outlook.com so we can add that here and here and then we have to add the password and of course this is going to be hidden so here we need to do a process.env DOT and then let's do email underscore password now of course I'm not going to show you my password but what you can do is go to EnV which is right here and add an email underscore password is equal to to the password you use to create your account no it's not one two three one two three one two three in this case so add it and we'll be right back there we go now we have everything we need to establish or initialize a transporter and we can use the transporter when paired with mail options to actually make an email transport so right here below the mail options we can say transporter dot send email where we pass in the mail options we created and here we pass a callback function that has the error of a type any and the info of a type any and then here if there's an error we can simply console log it by saying return console.log error and else we can simply do a console log email sent and then with the response or in this case the info so this is great and this should actually work so we are ready to give it a shot you might need to reload your terminal just for the changes in the dot EnV to take effect so press Ctrl C and then Y and then rerun it and now we can actually call the send email by importing it from node mailer and calling it within our add user email to product this should now initialize the sequence of sending a welcome email so we can collapse this function and we can actually give it a shot because this add user email to product should be called right here within our modal so we can uncomment it and we can import it coming from lib actions and the product ID has to be passed as a prop to our model so here we can get the product ID which is going to be of a type props and then we can Define the interface of props is going to contain the product ID of a type string and we of course need to pass that when calling the model so we can see where we're calling the modal which is within the page here and we can say product ID is equal to just ID just like so there we go and now it's not going to complain because it's getting exactly what it wants at least I hope there we go we're good and now we're calling the add email right here when clicking this thing so what do you say that we give it a shot we can go back to our application right here we can reload it just to ensure everything is good you can enter your email address I'm going to do JavaScript mastery00 gmail.com and you can click track it's going to submit it and close the model once we do that you can check if there are any errors in the console everything seems to be good here and in the network tab there is nothing there so everything seems to have went okay but we do have an error right here transporter.send email is not a function and this is happening at send email so let's see we go to our add user email to product and then we go into our send email and it's saying that transporter.send email is not a function we're defining the transporter here by saying nodemailer.create transport but as we saw it is complaining that the node mailer does not exist even though we installed it in packet Json so let's check it out we have node mailer right here it is possible that next.js is getting stricter with its rules and node mailer is strictly a back-end package so doing some reading right here it is apparent that we can only use this on the server right so what we might want to do is Define this entire file as use server so what we can do then is if you save it you'll see that we get a warning or an error here saying that only async functions are allowed here so we need to turn this generate email body to an async function and we can remove the return here because we know what async functions return they return a promise so now whenever we call this generate email body such as right here we have to do an await beforehand and now the email content is good and now if we save this the error should go away because we have all async functions such as the generate email body and the async sent email but the threshold percentage still is not async I'm going to Simply put it as a variable within the generate email body function so right here and it can be just a constant con threshold percentage right here now if we save this let's see do we have any other warnings or errors a user or file can only export async functions found object so yeah we have this object but we don't necessarily need to export it it can just be a regular object and there we go we're back on track now what if we try it one more time what if I try entering a JavaScript Mastery email and open up the console right here I'll try to give it a shot and run track and no error yet do we have something here no errors in the console no errors here either network is looking fine and this is looking fine as well okay so we didn't get that error and if we reopen the file right here node mailer yeah it's still giving the warning right here for type but it is possible that the function went through so the next thing we can do is expect that the email is going to arrive to our mailbox so this means that we have no errors but unfortunately we also didn't get an email so let's do some further digging first we can try to resolve this smaller issue of a type error and that can be solved by installing a Dev dependency so we can go to our terminal and run npm install dash dash save Dash Dev and we can do at types forward slash node mailer so if we're using typescript it's good to know our node and next.js application no that will be using these specific types for node mailer so this is going to get added within your package.json as a Dev dependency right here once it gets there our application is going to know what this is and it's not going to complain which is great now one thing I noticed is that here if we use a sent email it's going to say that property sent email does not exist on transporter sent message it says send mail and again this is another huge huge benefit of typescript as soon as we install those built-in types as you can see we can notice that this does not exist it's did you mean send mail so right here we can just do send mail and now hopefully this will go through so let's go ahead and just type our email one more time I'm going to do JavaScript Mastery and I'm going to click track but unfortunately I still didn't get any email nor did we get any information in the console so one thing that I thought of right now is that we used the previous email that we had this when this was wrong and then we added the new user right here but then the second time it's not going to add it because it's already there so what we need to do is now just for test purposes we need to send it to another email usually it's gonna work for everybody so what we can do is right now just go here and enter an email that you haven't entered so far in this case I'm gonna do contact jsmastery.pro I'm going to open up the console and I'm going to click track if we do this now I'm going to scroll all the way down to the bottom and you can see from JavaScript Mastery at outlook.com to contact jsmastery.pro this is great so far this is looking good and now the last thing we have to do is wait for an email foreign have gotten the email right here within our inbox from JavaScript Mastery at outlook.com now one thing I want to mention is that it was under spam so you might want to check your spam folder and move it to your inbox but here's how the email looks like welcome to price wise You're Now tracking Apple 2022 MacBook Air and then here's an example of how you'll receive updates and then here we say you know the price has lowered or the product is now back in stock don't miss the chance and buy it so this is pretty cool it's a real application that sends you emails so with that said we can now put this to the side close our email provider go back to the application and implement the last part of the application which is ensuring that these emails are happening periodically and the bright data scraping happens periodically as well so that we can be alerted on time so let's do that next we have this send email updated we have our actions everything here is looking good but now that we have added a user email to product indicating their interest how do we figure out when to send it or how to do any of that stuff let's close all of the currently opened files and let's start fresh to get started with implementing our cron functionalities believe it or not all the code associated with that will be within one file or to be more precise within a single API route So within our app folder we can create a new API folder and create a new route called cron and within it we can create a new file called route.ts and all of the logic is going to be within here which is pretty exciting as this part of the code is going to utilize everything we've done so far it's going to scrape the product it's going to do that periodically it's going to modify and call our database and do all sorts of other things but all of that is going to happen within this single API route so to implement it we can say export async function get as it's going to be a get request and then we can open up a new try and catch block within the catch we can simply throw a new error and we can say something like failed to get old products or error in get and then console log that error if we are in the try though we first need to connect to our database as we usually do from our server actions so we can say connect to DB import that from add forward slash libmoncus and then we can try to fetch all of our products by saying cons products is equal to a weight product dot find and then we can pass just an empty object means find all of them and we can import product from lib product models then we can say if there is no products then we can simply throw a new error saying no products found but now we start with the first step of our Cron job which is one scrape latest product details and update DB there we go I can code better than I can type apparently but we got through that sentence so let's scrape all the products how do we do that well we can say const updated products is equal to a weight and we're going to create a promise dot all that's because within it we're going to call multiple asynchronous actions at the same time yes we're going to access and update all the products in our database at the same time so we can say products dot map where we get an asynchronous function callback function that's going to get the current product and then within here we want to scrape that individual product so we're not going to do this on a per product basis rather we're in a map over all of our existing products so let's say const scrape product is equal to a weight and we're going to call the function we have already created called scrape Amazon product coming from lib scraper so you need to import it and pass the currentproduct.url I'm going to expand this a bit further so you can see everything in one line there we go then we can do a simple check if there is no scrape product we can throw a new error no product found else we want to update that product's history so we can say const updated price history is equal to an array and now if you remember correctly we're already dealing with price history before so we can search for Price history and you can see that we're calling it right here between this lib file I believe there we go product and we have update price history so we want to do exactly the same thing as we're doing here updated price history and then we also want to update the product in our database as well so all of these things from line 44 and my end to 26 from updated price history to updating the product and then to updating the product in the database we can copy that get back here and just put it below this if check so we can say const we can call it updated price history no need for a type right here we can get the current product in this case we called it current product dot price history and then update the new scrape product that current price then we want to redeclare the product object by saying Khan's product is equal to we spread the current scrape product and then we update all the prices so in this case we need to import all of these functions from libutils by double clicking them and then pressing Ctrl or command space this is the fastest way to do it and now we have this new product which we need to use to update this in the database so here we have one extra curly brace so we can remove it we can indent this properly and we say const updated product is equal to a weight product that find one and update based off of the URL we pass the new product and we don't have to create them from scratch here because we know they already exist and then we have to figure out what are we closing here do we really want to exit out of the map right here well not yet so we're still right here below this updated product and we're diving into the Second Step which is two check each product's status and send email accordingly okay I know this is getting a bit complicated we are essentially just scraping again the products we already have in our database updating the database and now we want to check the product status has the price lowered is maybe the product back in stock and so on so let's declare a new const email notif as a notification type is equal to get email notif type and then we want to pass the scrape product and the current product so we can know what has changed and then this get email notif it's gonna come from a utils it's the one that we commented out before so now we can uncomment it and now we have the notification we have threshold percentage and we can ensure that nothing here is red so here it's saying the lowest price does not exist on type new title storing options let's see what this is about so here we have our notification which we have to import at the top yes that was the thing before we didn't have these now we do so now maybe instead of importing these as we converted our function into a server action so we can no longer export these maybe we just get in there and copy this notification right here so we can paste it right here at the top of this call we no longer need to import it and then we have the threshold percentage which is in the node mailer so let's go into the node mailer file and then we have defined it I believe within this function right here so we can simply copy it go back remove the import and then put it here as well so now we can easily use all of these within the function and nobody is complaining so now we can go back within our route and we can import get email nodeive type I'm going to expand this a bit further and you can see it complains about the scrape product which is happening right here it's saying that the argument type of this is not assignable to parameter of type product that's because it says that we're missing the average price and is out of stock so if we look into the return of our script Amazon product function and scroll down you can see that indeed we misspelled is out of stock we have a capital f here which we have to fix and what is it with the average price yes this was supposed to be average price not just average so if we fix this I believe we're going to be good to go yep once again thank you typescript for noticing this this can take you hours and hours to notice if you weren't using typescript now that we have this get email notification we're gonna know which kind of notification we have to send so what we can do then is say if email notification type exists and this is important if updated product dot users dot length is greater than zero meaning if there's somebody to update then we want to get the current product info by saying cons product info is equal to an object where we have the title equal to updatedproduct.title and URL equal to updatedproduct.url then we want to construct the email content by saying const email content is equal to generate email body again using another function which we have created passing the product info and the email notification type finally we want to get an array of user emails to which that we need to send the emails to so const user emails is equal to updatedproduct.users.map where we get each individual user and we do a check user and then we can Define user as a type in this case we can just use any and then simply return all user emails now we have an array of all user emails we need to send emails to so what do we do well we call await send email which is a function that we have to import from lib node mailer and then we Define the email content we want to send and user emails we want to send it to and with that it's so easy and the email content once again thank you typescript returns a promise because we made it into an async function so we have to add an await right here for this to work pretty cool stuff so now we're actually sending an email we're gonna go one div below and then simply return the updated product as well with that we want to go all the way above the catch and return a next response Json where we simply pass a message is set to ok and we can pass the data set equal to updated products in this next response is coming from next server so this is it I understand that this was a bit tougher that we had to incorporate all the functions we have built so far in this video into this single Cron job and that's usually what cron jobs do not any new logic but the logic you have already created by following a specific period so once again we connect to the database we find all products we re-scrape all the products by doing scrape Amazon product we update their price history and the current prices then we check if a notification has to be sent maybe they're back in stock maybe the price dropped below a threshold anything like that we find all users that subscribed to the changes in those products and then we simply send emails to all of them this is it so now we have our cron route and we can actually deploy our application which is then going to expose our API route online so we can call it periodically I'm going to show you how to do that as well so the next thing we have to do is simply deploy the application so I'm going to close all of the currently open files collapse the folders and go to our terminal so right here go to view and then terminal and we can stop it from running and close the other terminal which we won't need anymore and clear it so how do you deploy modern Nexus 13 applications with scraping and crons well you deploy them with versel versel of course being the creator of nextgs of course it's going to be the simplest and best to deploy right with them so go to versel.com click Start deploying and then you'll be greeted with all the projects you have created before once you log into your dashboard as you can see I have a lot of projects right here we have rebuilt our entire JS Mastery Pro platform right here but then also we have many of the team projects and cohort projects from our JSM masterclass experience all of these projects are production ready and are deployed on for cell as they're using the latest and greatest of next gs13 Technologies so the entire JS Mastery Pro website alongside the masterclass page which you can visit in case you want to learn more about it is built using nextgs and deployed on her cell so to deploy your application you can go to add new and then go to project but before it appears right here we'll have to push the code to GitHub so simply go to github.com forward slash new and right here you can say price wise and then you can make it either public or private and then click create a repository now you can put this side by side with your current code editor and then we can start copying commands one by one the first command is git init but before you do that please add the dot local to your dot EnV this is going to ensure that it doesn't get published on the internet then you can run git init git add Dot get commit Dash M first commit git branch and Main get remote add origin and then finally git push U origin Master this is the only thing it takes to publish all of your code right here on GitHub in a matter of seconds and you can see it is there now that it is we can go to versl and you can see just now the latest version of pricewise is there so you can click import now inside of here you can play with the name a bit and then you have to add your environment variables so what you can do is just go back to your dot env.local copy everything from there and when I say copy everything I literally mean it just press Ctrl or command a and copy it go back and then simply paste it right here versel's magic is automatically going to copy all the keys and the values separately once you do that you can press deploy and wait for the magic to happen now usually deployments don't go right on the first try so we might need to come back and fix something in the code which is totally okay it's just important that we get this first deployment out of the way so let's wait for the build and then we can fix a couple of typescript Errors if there are any and unfortunately I was right we do have some build errors so right here we see that module not found can resolve supports colors okay that's interesting another error happened while pre-rendering the API cron page okay and then this is it and here it broke while reading the price so let's go ahead and fix those together the support scholar one is a bit weird but it happens sometimes one of our packages requires some other packages to be in there so doing a quick Google search and just figuring out nextges 13 can't resolve supports color the first answer is to just install it so we're gonna do that right now just to go through with the deployment that's one change we have to make and then another one is figuring out what is the price error in the Quran route so right here let's search for DOT price here we have the price history not really price so let's just reread the error it's saying cannot pre-rendered because it cannot read properties of undefined reading price okay and we don't really get a line number right here so let's just search for price and let's just close it right here so there's just one instance of price like this but if we go forward we can see price history lowest highest and more but not really just price right and here we just declare it as a key so this is not really a place where it should break regarding this price let's go back a bit and check it out so here let's see one thing that I noticed is right here under update product we're seeing scrape product.url but we need to get the product that URL so this is just a tiny mistake on my end but besides that there's one more important thing we have to do which is also a great lesson on its own alongside every single API route next.js allows you to export a couple of different functions that are going to modify the way that this function or API route opens so we can say export const Max duration and that Max duration here is going to be set to 300. this is not milliseconds this is seconds so that's a couple of minutes right here I think even GitHub copilot told me it's five minutes so it cannot go over that we can also set export const Dynamic is set to force dynamic and in case you want to read more about different execution modes you can simply Google different file conventions and Route segment configs and here you can see Dynamic Auto Force Dynamic and more and then you can read what other options can you pass to it here you can read more about different options for example Force Dynamic is going to force Dynamic rendering and uncache data fetching of layout or Page by disabling all caching of fetch requests and always revalidating okay pretty cool and then we want to also set a revalidate to zero that's the last option export const revalidate is equal to zero hopefully these couple of fixes and the installation of our support Scholars is going to make it work so let's simply run git add git commit Dash m let's say fix build errors and let's simply run git push now we don't have to go through the entire process of resell simply you can visit your repository or your dashboard go to the last project and go to deployments and you're going to notice that the second deployment is now running and we can wait to see if it's going to be an error or if it's going to be successful and there we go we are ready so now if you go to the latest deployment and you click visit you can see that we are now live and we have our latest two products because this is connected to the same database we were connected to before just to ensure that everything works alright let's try another Amazon product I'm gonna grab another one of Amazon's deals and let's go for this Philips 3000 Series airfryer it's also at a great discount so I'm gonna copy it go to our now deployed price wise right here and then enter the product link and click search after searching it's going to add it right here at the bottom you can Implement a redirect but we can also simply click right here immediately a new model opens up in case you want to be alerted about these product updates but even right now you can see it's a great deal because it is discounted from 180 to 79 and then here we have the description the buy link and similar products but of course we don't want to click track yet because even though we did implement the cron function or the API route which we exposed that we can call we didn't yet Implement a Cron job or a timeline when it can be ran but now I want to show you a guide that versel themselves wrote on how to set up Chrome jobs on versa and as I said before cron jobs are time-based scheduling tools used to automate repetitive tasks by using a specific syntax called a cron expression you can Define the frequency and timing of each task this helps improve efficiency and ensures that important processes are performed consistently and versl supports cron jobs for serverless and Edge functions and here you can learn how to implement it although it's easy to set up it's difficult testing their cron functions as they only run once a day in production and as an alternative we found kron-job.org it's completely free and you can easily adjust the Quran schedule so no need to learn how to compute it and they also have logs so the only thing you have to do is create an account and it looks like they still use the older version of material UI I would remember this anytime I'm sure they're going to switch the Tailwind really soon and after creating an account you simply have to verify your email address and once you do that you can sign in once you're in you'll be able to see this xero enabled cron jobs so let's simply create our first one we can first give it a title such as price wise we can give it a URL and let's go to our current version of the application but not this specific deployment you want to go all the way to versel to your projects pricewise JSM and then you want to visit here and not any specific deployment this is going to give you a clean URL for all the updates on your latest branch so now take that and paste it right here but we want to append forward slash API forward slash cron so this is the URL or the endpoint where we had that cron function we can click enable job and then the execution timeline for now can be every two minutes you can do this just for testing in case you want to play with it but usually you can do it maybe every day at a specific time so maybe every day before you wake up at let's say 7am this looks good to me and then that's when it's going to run for testing in case you want to play with it you can also do every two minutes then you can also schedule an exploration for now I'm gonna do it on the first of October but this is first October of 2024 which is in about a year but that's okay with me and believe it or not that's it you can now click create or you can click test run so let's click test run and you can see do you want to start a test run for the current job settings including any unsaved changes we'll execute your request immediately and display the results so let's click Start test run this is going to wait for the response and we got an internal error of 500. one thing that you can do to debug this is go to our versel which is right here and then go to logs so right here you should be able to see different locks and why the errors are happening so we can see that the API cron failed with a status of 500. and if you open up the second one you'll actually be able to see the error and we get the same one air get cannot read properties of undefined reading price and once again we don't have any more useful information about the line number or anything like that the only thing we know is that it cannot read properties of undefined reading price so this is the last thing we'll have to debug it is possible that we have a typo somewhere in our Chrome route that wouldn't be hard to imagine as this is a long file where we had a lot of different scrape products updated products and more so with that in mind in the GitHub just down below there's going to be the updated route.ts file which you can simply copy and then override right here doing that will most likely eliminate that little typo we had and hopefully the price is not going to be undefined so let's go ahead and just add that change by saying git add dot get commit Dash M let's say fix error and then git push finally let's wait until this is pushed to main which we can know by going to deployments and then waiting for deployment to finish and the ones our deployment is live we can revisit our live deployed website and we can run another track request and let's try with another email address just to be sure I'll try with this long JSM masterclass experience at gmail.com one to see if it works and click track this is going to submit the request and then we can go back to our versel go to logs and we can track to see if we get a 500 or if it works so here we can switch this to live and hopefully we're going to see a Quran request come in or rather we can call it directly through our current job and I'm going to click test run one more time and let's start a test run now we can go back to our logs and hopefully wait for a successful log there we go API cron mongodb connected and is it going to be a 200 or a 500 it looks like API KRON gave us a 200. if we go back right here to get cron there we go it succeeded so it was a typo right there but thankfully it went through we can see more information about the product here and then message okay this is what matters the most so now that we know that it is working we can actually create this Cron job and believe it or not this is it it's going to automatically execute it at 7am tomorrow or at least that's how I have set it up and then you can go to status Pages or even dashboard and here you'll be able to see all of your events that have ran based on a specific cron schedule in case you want to test it a bit you can also go here edit it and then do something like every one minute in that case it should just happen immediately in a minute and then we'll be able to see it in the event dashboard so let's wait for that and see if it works and as you can see cron jobs are being executed as they should and we're getting emails so just so it doesn't happen every two minutes I'm gonna go to cron jobs edit and then I'm gonna put it every day at 7 00 am there we go and I'm gonna click save with that said we've came to the end of this amazing video the application has now been fully built you can track any kind of an Amazon product immediately and then get its current price average price highest and lowest price as well as read more details and see similar products I of course all of this is being powered by bright data that allows us to scrape the data and then we can track those scrapings using cron jobs so it's been quite an exciting project although it looks simple there's so much stuff happening on the back end and now you know all of the concepts needed to make things work so once again huge thanks to Bright data not only for sponsoring this video but also for making such a phenomenal tool allowing us to scrape anything anywhere without any blockers and if you like this video you'll surely like what we do on jsmastery.pro in case you want to become a master nextgs 13 developer take our Flagship nextges 13 course where you build an entire modern stack or flow clone application or if you're really serious about your Development Career check out the JSM masterclass experience with that said thank you so much for watching this video and have a wonderful day

Channel: JavaScript Mastery

Views: 182,578

Keywords: javascript, javascript mastery, js mastery, master javascript, web scraping, web scraping tutorial, web scraper, web scraping amazon product, web scraping javascript, web scraping project, web scraping amazon, javascript mastery react js, nextjs, nextjs 13, nextjs 13 scraping, react scraper, react scraper website, javascript scraping, bright data proxy

Id: lh9XVGv6BHs

Length: 241min 41sec (14501 seconds)

Published: Sat Sep 30 2023