Full Stack AI Semantic Search with Next.js, Pinecone, Langchain, & ChatGPT - Full TypeScript Course

in this video we're going to build a full stack AI enabled semantic Search application using next.js Pinecone Vector database Lang chain and chat GPT a few days ago I released this code base along with the Twitter thread that did very well at over 450 Stars I had a lot of people reaching out to me saying that they really appreciated this code base it helped them out a lot but a lot of people were wanting a tutorial on how to actually build this thing and how how it worked so that's what I'm doing here today now what is semantic search well I plugged the question in of what semantic search is in the context of AI to chat GPT and it gave a pretty great answer it says semantic search in the context of AI refers to a search technique that aims to understand the intent and meaning behind a user's search query rather than relying solely on keyword matching it goes beyond traditional language-based search methods by incorporating natural language processing and machine learning techniques to comprehend the context or relationships and Concepts associated with a query if we got here to the bottom it says overall semantic search enhances the search experience by providing more meaningful results that align with the user's intent making it a valuable tool for information retrieval and knowledge Discovery now what is this going to end up actually meeting in practice well it's going to be very similar to something like chat gbt except we're going to be able to incorporate proprietary or public or private data sets or maybe data sets that aren't yet indexed by GPT what else you can do is you could actually take data in a database and expose it to something like this and enable more of a natural language query over that data as opposed to requiring it to be in a strict schema like you would need for querying a traditional database so there's a lot of applications that you can do and therefore this is becoming a very very popular technique and in the context of me being a developer a lot of developer companies developer Advocates and teams are actually incorporating this in their docs so if you want to have a developer documentation site that is improved you might enable semantic search so instead of the user trying to go through and find the answer they can just ask the answer regardless of what it is and that's kind of the application that we're going to be building today we're going to take some developer documentation we're going to take those text files they could be text files they could be marked on files or they could be PDFs in our case and we're going to be able to index those in the Pinecone database as vectors and then query this so that's kind of going to be what we're doing now what do you need in order to build with this you're going to need a Pinecone API key so you're going to have to go to Pine Cone and sign up and then you're also going to need to go to open Ai and you're going to have to have an API key so here I have an account with openai and I also have an account with Pinecone and that's all you need in terms of services but for your actual like local environment you're going to need to have node.js installed and that's kind of about it so with all that being said I want to call out one great YouTube channel that I am essentially porting over a portion of the code base that was originally built for node.js and I'm now applying it changing it up a little bit and building it for next.js developers digest I will link to the channel below it actually has a lot of really great um JavaScript AI tutorials that I also recommend checking out and subscribing to as well so with that being said I want to go ahead and get started and we're going to go ahead and start writing this code and we're going to start off by just creating a new next.js project so here I am and a terminal and I'm going to say npx create next app and I will call this pine cone next semantic search or something like that and for the questions I'm just going to take the defaults we want to use typescript eslint Tailwind we're going to say no to the SRC directory yes to the app router and know for customizing the default import alias and we're going to go ahead and change into that new directory we'll open this up in our code in our text editor and we're going to go first to our actually type TS config and we're going to go ahead and set no implicit any to false just for this tutorial to get through um not having to have everything typed so we're going to use any a couple times and therefore we need to go ahead and set that and we also need to go ahead and install a couple of packages so what I'm going to do is go here back to my terminal and here we're going to go ahead and install our dependencies using npm yarn pnpm or whatever you would like and we're going to need three one is going to be Pinecone database slash Pinecone one is going to be Lang chain and if you choose to have PDFs as your data then PDF parse so we'll go ahead and install those and from there while those are installing we're going to jump back to our code base and we're going to create a new file called Dot env.local and this is going to be where we store all of our environment variables and we're going to need three environment environment variables one is going to be our open AI API key one is going to be our pine cone API key and then one is going to be our pine cone environment so we can start populating these just by going into first open AI we'll go to our view API Keys here and I will create a new secret key that I'm going to use for this tutorial and I'll copy this to my clipboard I will drop it here next we'll get our Pinecone API key so I'll go to Pinecone I've got API keys I'm going to create a new one for this I will go ahead and copy that as well drop it here and then for our Pinecone environment this is just going to be whatever environment we're in and this is going to be us East for gcp for me all right so that's all we need for our environment variables and we can go ahead and close that um one thing we're going to set also is a couple of pieces of configuration so I'll create a file called config.ts and here we're going to set a couple of Minor Details that we're going to be needing later on and these are going to be the index name and this is going to be the index for your Pinecone database so I might call this test tutorial index or something like that because I'm going to be deleting all these API keys and in this index when we're done and then the timeout is going to be how much time we let our application wait until the index is initialized because when you create an index on Pinecone it takes some time so for this tutorial I might say something like like 80 000 milliseconds which is like 80 seconds or something like that so this can be anywhere between 60 seconds to maybe three to four minutes but I think that 80 80 Seconds should be good to go so we'll go ahead and close that and the next thing we're going to do before we actually write our UI or we build out any of our routes is we're going to need to go ahead and create the functionality for talking to both Pinecone as well as open Ai and and using our lane chain sdks and stuff like that as well so to do that we might go ahead and create a file called utils or whatever you would like this to be called and this is going to hold a couple of different functions one is going to be for querying a Pinecone once we've stored data there one is going to be for creating the Pinecone index and is one is going to be for uploading data to the Pinecone index okay so before we write any of those functions we need a couple of imports so the first one is going to be the open Ai embeddings and the open AI embeddings is going to allow us to call openai to embed text into vectors that we can then store and use on Pinecone we're going to have this recursive character text splitter which allows us to chunk large amounts of text that we can then use in our app we're going to import the open AI API from link chain slash llm slash open AI we're going to import the load QA stuff chain which I'll talk about more in just a moment but this is basically like a generic question and answer type of API that you can just stuff a large amount of text in and then ask questions about it we're going to import this document from Lane chain slash document and then we're going to import our config and we're going to just need the timeout from there all right so I guess the first function we might create is going to be for creating the Pinecone index so we'll have this function called create Pinecone index it takes a Pinecone client an index name which is going to be essentially we could have also imported it from here and then the dimensions for our Vector database all right so the first thing that we're going to do is just go ahead and console.log the name of the index and this will be the index name that is passed in to the function next we'll go ahead and get a list of the existing indexes that are available and what we basically want to do is just check to see if the index already exists and if it does not we're going to create it so in the next line of code that's what we're going to do here we're going to say if the existing indexes already include that index name we're going to skip this code but if not we want to do something so what we're going to do is we're going to go ahead and say we're creating an index with the index name of whatever we've Chosen and then we'll just say await client dot create index and then we'll go ahead and log out when whether or not this is successful and then we're going to go ahead and wait for this initialization now there is no API that I know of to actually call this to wait for it to resolve therefore we're kind of using that guesstimate timeout that I talked about earlier in our case it's like 80 Seconds and if that is not um the case like if we already have uh an index for there we can just go ahead and log out that the index already exists so we can say like if the index doesn't exist we'll create it and then we're going to wait 80 seconds or whatever length of time for it to be created and if not we are going to just exit and we're going to go ahead and console.log at this index already exists so whenever we're ready to initialize this we can just call this function this could be you know invoked and whatever way you'd want we're going to just invoke it from our UI by having like a function that allows us to kind of invoke this from the client so once we've created the Pinecone index what we need to do is actually upload data to the Pinecone index and what we need to do is create a function for that so we're going to create a function called update Pinecone and this is going to take in the first two arguments the same as the last function the client in the index name but the third argument is going to be the documents that we want to store in Pinecone and we're going to start off by just logging out the index again next we're going to go ahead and log that out we're going to go ahead and actually retrieve that and then log it out here next we want to go ahead and map over all of these documents that came in to our function when we call this and this is where we're going to do like all this work we're going to basically be processing all of these and doing some formatting and then uploading these to the Pinecone database next we'll go ahead and just log out that we're processing the document and we'll just log out some metadata about the document we're getting get a couple of variables off of the document one is the text path and this is literally the path on your local file system so this would be for me like slash my name slash documents blah blah blah or whatever wherever this thing existed text is going to be the actual text that is going to be part of that document that we're going to be using in our database or storing next we're going to go ahead and take that text and split it in order to do that we need to create an instance of text splitter and the chunk size that we're going to be setting is a thousand a thousand based on what I've seen in other tutorials and other documentation to be honest so this is kind of like a good recommended recommended size and now we're just going to log out that we're splitting this text and here we're going to go ahead and actually do that text splitting by calling the instance of the text splitter and we're going to say await text splitter.create documents and we're going to pass in that text and we're going to go ahead and just log out how many chunks that we have and then next we're going to go ahead and call the open AI embeddings endpoint and this is going to be an actual like API call so we're going to say cons embeddings array is equal to open AI embeddings.embed documents and we're going to map over those chunks and we're going to be using the chunk Dot Page content and we're going to find any instance of a new line and replace it with just a space and then after those embeddings are completed we now have this array that we can use for Pinecone whereas we could not have with just the the original text itself so we'll go ahead and log out that we're creating the whoops the length vectors array with ID values and metadata so now that we have that we need to actually take this data and upload it to pine cones we have it formatted and some uh to some extent and we want to go ahead and upload it so what we're going to do is we're going to set a batch size which is in this case 100 which is from what I read and understand the recommended an efficient recommended size for uploading the Pinecone and then we're going to create an empty array that we're going to be populating in the next Loop so we'll create a for Loop we'll say four the length of the chunks we're going to just map over them we'll then take the chunk using the index of the uh of the item coming through this mapping and we'll get that value out of the chunks array we'll then go ahead and create the vector and this is an object that has an ID of values and a metadata values the idea is going to be a combination of the text path and the index we're going to set the values as the embeddings array and then we're going to pass in some additional metadata and then we're going to basically push this value into that array that we created just a second ago so we have that batch array we're going to go ahead and add this Vector to that array along with all the existing values so this would be almost like a push and then we're going to check to see if we've reached that batch size so if we've reached 100 items so we're going to say if backshot length is equal to 100 or we're at the end of the array we're going to go ahead and upsert this meaning we're going to upload this to um Pinecone and that's about it if this is completed then we've now uploaded everything and we're done with this function so this is going to be the function that we call to upload the data to Pinecone and then finally we've created the pinecret index we've uploaded data to it we need now a function to query it and that's going to be the last function that we create and this function we'll call query pine cone Vector store and query llm and that's exactly describing like what we're going to do we're creating both so we're going to go ahead and retrieve the index and we're going to go ahead and get the query that's passed in which is the question and we're going to go ahead and create an embedding out of that so we're using in the open AI embeddings to query both the documents as well as the query and we've already used the other the documents embedding in the last function and then we're going to call index.query to talk to the Pinecone and we're going to pass in this object where we're saying the top K like the top number is 10 we're passing in the query embedding and then this will kind of come back with a results dot matches array and we can say log that out so we're going to say we found query spots that matches that length and hopefully this will be 10 because that will be kind of the the number we're we're maxing out at and hopefully we have that number and then we'll say if that came back with anything like if there is a link to that array meaning we found some matches what we want to do is then call our llm which is open AI so we'll create a new instance of openai we're going to then use this load QA stuff chain passing in the llm and we're going to go ahead and extract and concatenate the page content from those documents so we're going to map over all of that and we're going to join it with a space now I want to jump out of this really quickly to kind of show you this QA stuff chain documentation because this kind of threw me off a little bit I was like what does that even mean well there's all types of different chains and Lang chain and one of them is this stuff documents chain and this is the one that we're using here and it says this chain is the most straightforward among the three it simply injects all input documents into the prompt is context and Returns the answer to the question it's suitable for question and answer tasks over a small number of documents and I would also suggest to maybe go through some of these other chains to kind of see how they work and what you might be able to do with them there's quite a bit that you can do there so we've concatenated the page content here now we want to actually make a call using the data that we have so we're going to say chain.call the input documents we're going to call a new instance of document passing in then concatenated page content and then the question is going to be the question that came in as an argument and then finally we're going to log out and return that text so we'll go ahead and log it out and then we'll return it and then finally we'll just have an else statement here that says there are no matches so we're not going to even call gpt3 because the if statement started up here if we have a response.length and then if not we're just going to log out saying we don't so that's it I think that that's all we need for our for all the utilities for talking to open Ai and Pinecone so go ahead and close that and we need a couple of routes now for our API so we're going to create two new folders one for that's where we're going to call read and maybe we'll create a new folder called API to kind of like hold all these other folders so we'll have slash API slash read we'll have another folder called API slash setup and the setup folder is going to be where we call that those functions for creating the index and uploading data to that and then the read is going to be us querying that and we separate those because obviously they they're two separate types of operations so for each of these we'll create a file called route.ts for both setup and read and we'll start with the read route so the thing that we're going to always need for most API routes in next.js or in this case I think they call them route handlers now is this next request and next response or maybe at least one or the other we're going to then import the Pinecone client and this is what we're going to use to make to create an instance of the Pinecone client and then we're going to import the query for querying the Pinecone database so we're going to have the query pine cone Vector store and query llm function from our utilities and then we're going to go ahead and import the index index name I guess and the index name is what we're going to be passing into the function next we'll go ahead and create a function called post and this is going to allow post requests coming into this API route we're going to get the body that's coming out of the request and this is just going to be kind of the query that's coming in we're going to create a new instance of the pine cone client by calling clients.inmit we'll create a client and then we're going to call the function that we imported query pine cone Vector store and query llm passing in the client the index name and the body and then we will go ahead and return that using the next response.json so we'll get like a nice object coming back with a data property and you might even you know throw a try catch blocks block here just in case this returns nothing or something like that so that's the read and this is going to be the thing that you're going to be using to query all of the different questions you're asking and then the other is going to be for actually setting all this this stuff up and creating it so what we're going to import is we're only going to import next response because we don't really have a requests that we're going to be you know interacting with it's only going to be us returning a response we're going to import a couple of loaders and these are how you kind of load values and data to start interacting with the Lang chain apis and other apis so we have the text loader which does what it sounds like we're going to be using it to to deal with text we're going to have the PDF loader which allows you to upload PDFs and then we're going to have the direct reloader which allows us to work with directories we're going to have a couple of imports for my our utils the create Pinecone index and the update Pinecone functions and then we're going to also import that index name so those are all the Imports that we have we're going to then go ahead and create this function called post and this is going to be what we use to interact with the Pinecone functions and and stuff like that and what we want to do now is we want to First go ahead and get all the documents that we're going to be storing and have those ready because if you remember that function that we called earlier where we're calling update Pinecone takes in those documents so we're going to go ahead and create a new instance of direct reloader which allows us to pass in the different paths that we want to work with and in this case we're passing in slash documents and therefore we're going to need to create like a slash documents folder in just a moment and here we're going to like try to match so if we have a DOT text file we're going to call textloader if we have a DOT markdown file we're going to call text loader as well if you have a PDF file we want to use the PDF loader and there are all different types of loaders that you might also be working with Beyond this so we're going to go ahead and create a variable called docs that's going to call loader.load we're going to set our base Vector Dimensions here and we're going to be passing these Vector Dimensions into our function so this could be changed to whatever you would like we're going to create a new instance of the pine cone client and initialize it and we're going to now have our try catch block where we're going to first create the Pinecone index and then we're going to then upload the documents to that and then once that is complete we're going to return a next response with some data that's just a string saying successfully created index and loaded data into blind code so we'll go ahead and close that that file as well the only thing we now need to do we need to do two things I guess we need to create our user interface our UI and then we also need to get our documents that we want to upload so for the UI we can go to page.tsx I'm going to go ahead and delete delete most of this and we're going to start at the top of the file we're going to need to say use client because we're going to be using using use state and react in this use state is going to be how we set up and manage local state we're going to set a few local variables we're going to have a variable called query that's going to allow the user to type in the query that they would like set query updates that query the result is the result coming back from our API call to query the actual database we'll have a function called set result to update that and then we'll just have a loading and set loading to enable us to show whether or not we're loading in the UI we're gonna have two types of API calls one for calling the read and one column for the setup so for the setup we'll have a function called create index and embeddings that we'll call the slash API setup function and this could actually have been a get request so this could have been API slash setup here could have been a get but I think I had ported over a previous post so I'm still using post it doesn't really matter next we're going to call the um yeah we're going to call the API we're going to wait for the Json response we're going to you know get that response we're going to log it out so that's pretty straightforward we don't really need to pass in anything to this function because it doesn't require any arguments the next function is going to be the one that we use to actually query Pinecone with our query so we're going to have a function called send query we first check to make sure the query exists if it does if it does not we return if it does we continue we go ahead and reset a couple of other state variables like the result and the loading spinner and we set loading the true there and then we're going to go ahead and call our API passing in the query as the body and then once that result comes back we set that in the local state and we reset the loading variable which we will reflect in our UI in just a moment next we'll go down to our UI and here we have some I would tell when styles that we are going to be using and I think I'm going to go ahead and just delete the um one for a Min H screen because it's going to kind of mess up the UI that we're going to be working with by default and from there we should be able to just take what's already there existing this isn't going to be the most beautiful like UI you've ever seen in your life it's going to be actually pretty basic but I think that this tutorial is more around just getting the job done as opposed to building up the most uh the best styling and stuff like that next we'll go ahead and create an input and this is going to be for the user to type we will have an on change Handler to call set query passing in the event dot target.value which is basically going to be just whatever's in the input we'll also have a button and the button is going to be the first button is actually going to be for calling send query and here we're just passing in ask AI this could be you know changed wherever you would like we have a little bit of a styling on both of those as well using class name for Tailwind we're going to have a couple of different state pieces of state that we're going to look for and we're going to show you why based on that so if the loading variable is true we're just going to show that we're asking the AI if the result exists we're going to show the result and then finally we have to have a button for creating the actual embeds and for creating the index so we'll have a button called create index and embeddings and we will have that one called the create index and embeddings function which is right here and I believe that's all we should need for the UI and I think that's actually all the code that we need the next thing that we're going to need is actually some training data so I'm going to go ahead and create a folder here actually not not in the API directory but in the main directory we will create a folder called documents all right so we have our documents folder there and we need to go ahead and create some text files now you can get these from obviously wherever you want this could be this is actually going to be the main thing that you would consider when building an application specific for your use case but for me and uh in readme which is the documentation that we use for lens protocol there's a way to just export our documentation so I'm going to export those docs this is kind of the the docs for docs.lens.xyz there's all there's all types of stuff what I basically want to do is allow developers to instead of coming here and having to go through all this uh let them just ask questions about our docs so that's the general idea there so that's what I've exported here and that's what we're going to be using so I'm going to go ahead and open up my finder here I'm going to go to downloads go ahead and unzip this all right so we have all this stuff here I'm going to go to V 1.1 I'm gonna go ahead and copy all of these folders here and then I'll go to this application that we just created and I'm going to go ahead and paste all this stuff here so we have a bunch of markdown docs now in our documents folder and this is going to be I think all we really need in order to go ahead and run the app so we'll go ahead and test this thing out by going to our terminal and I'm going to run npm run Dev and it looks like we have the app running on low close 3000 we'll open this up see what happens all right so nothing is breaking yet that's great I will go ahead and open my terminal and we also want to keep an eye on I'm sorry um I'm opening up my developer console we also want to keep an eye on the terminal so I might kind of do that and we will go ahead and click create index and embeddings and it says creating my tech test tutorial index so that's great if we go to Pinecone console and I go to my indexes I should be able to go ahead and refresh and see that we have this new my test tutorial index create being created so that looks like it's happening successfully and we're going to wait the 80 Seconds to see once that's completed if we can actually store all this information in our Pinecone database all right so now we're seeing some additional output here and it looks like our data is now being split into chunks it's being embedded and it's being stored in Pinecone so this is going to map over all of that data that we had in that documents folder and go ahead and upload it and once this is done we can go back into our app and we should be able to type in our question and then click ask AI all right so the upload was successful we get this response from our API saying successfully created index and loaded data into Pinecone now you might be asking what is lens protocol because that's the documentation that we we are using so let's just ask AI what is lens protocol and we're going to test out our query now we'll click ask AI we get some feedback here querying pine cone we got 10 matches from Pinecone now we're asking this question what is lens protocol to open Ai and we're using that data that came back from Pinecone and it says lens protocol is a web 3 social graph on polygon proof-of-stake blockchain it's designed to empower creators to own the links between themselves and their Community blah blah blah so it looks like it's working we might ask now like how can I use lens as a developer and we'll go ahead and ask our next question and see what happens here it says as a developer you can use the lens SDK to quickly and easily build social applications the SDK abstracts away the need to write a lot of lower level graphql boilerplates um you can use the lens SDK for react react native or other JavaScript Frameworks you can also deploy the protocol yourself and these are all like pretty precise interest so that's that's really great that's kind of it that's the end of this tutorial really we we built everything from scratch it seems to be working uh you can then now go into our utils and you might want to polish up or modify how we're interacting with openai with maybe more custom prompts or you might do some you know differentiating things that might make sense for your application but the core of what we wanted to accomplish is already done so that's kind of it so thanks for checking this out if you want to see the code base it's linked in the comments below be sure to like And subscribe and share if you enjoyed this video and come back and soon and I'll hopefully have something else for you thanks

Channel: Nader Dabit

Views: 2,768

Keywords:

Id: 6_mfYPPcZ60

Length: 38min 7sec (2287 seconds)

Published: Sat Jun 03 2023