OpenAI Embeddings and Vector Databases Crash Course

embeddings and Vector databases are essential if you're building any type of AI product in this video I'll go over what they are and how to use them with open Ai and their apis I'll cover this in three parts I'll explain the theory then the use and finally integration after watching this video you'll be able to create long-term memory for a chat GPT or perform semantic searches based on a huge database of PDFs connected directly to an AI let's start there are two terms I'll cover embeddings and Vector databases they work together I'll begin with embeddings what are embeddings to put it simply an embedding is data like words that have been converted into an array of numbers known as a vector that contains patterns of relationships the combination of these numbers that make up the vector act as a multi-dimensional map to measure similarity for a simple example let me describe a 2d graph the words dog and puppy are often used in similar situations so in a word embedding they would be represented by vectors that are close together well this is a simple 2D example of a single dimension in reality the vector has hundreds of Dimensions that cover the rich multi-dimensional complex relationship between words images can also be turned into vectors too and it's how Google does similar image searches the image sections are broken down into arrays of numbers allowing you to find patterns of similarity for those with closely resembling vectors once an embedding is created it can be stored in a database a database full of these is considered a vector database and can be used in several ways including searching where results are ranked by relevance to a query string or clustering where text strings are grouped by similarity and recommendations where items with related text strings are recommended also classification where text strings are classified by their most similar label for the purpose of this video I'm going to just cover searching since it would be the most commonly used there are many practical ways to do this but open AI has provided a great AI model to specifically create embeddings it does not however provide a way to store them which we will be using a cloud database for later in the video now let's start making an embedding by accessing open AI on the Google page I'm going to browse open AI I'm going to head over to the open AI website where I can create a new account or log into an existing one which is what I'm gonna do it's free if you want to sign up I'm going to log in using my Google credentials and I'll be taken to a few options here between chat gbt Dali and other apis I'm going to head over to their API page and this will take me to the dashboard here I want to start off having a look at the documentation on the embeddings which you can find just over here and I'll also link this in the description but what we're going to start off with is creating some API requests for embeddings themselves in order to do that we're going to head over to the API references page here under embeddings is all the information to create one and it's quite simple it's as easy as doing a single post request with some inputs and getting a response back we could write code to do this or do it inside of a terminal but the easiest way is inside of some GUI software and this is called Postman it's an API platform and it's also the sponsor for today's video Postman is a piece of software and also a web app that's entirely free that allows you to do all sorts of API requests it's quite easy to get running I'm using Windows so I'm going to download the windows version of the application here I've launched it the first thing I'm going to do is create a workspace for my API queries this I'm going to call open AI Vector database there are a few different types of workspaces between personal private but I'm going to select the team one because this allows me to share out my workspace to other people in the future if I wanted to for this new workspace I'm going to open up a new tab kind of like Chrome here I've got a few different options but the main one here is the type of request which can be a get post push I'm going to do a post request and here I'm going to get the URL that we're going to be using for the embeddings which is just down here and it's the api.openai forward slash version one forward slash embeddings URL I'm going to paste this here in the URL section and what's useful is that Postman is telling me that I do need authorization to use this endpoint for an API I also have the instructions on how to generate one so I'm going to head over to this open AI API Keys page and on this page I can select it to create a new secret key I'm going to label this embeddings and then I'm going to generate it make sure you keep this key private I'm going to copy and paste this into Postman and I'll place it here in the API key section be aware Postman allows any type of authorization we're using the bearer token which is how open AI authenticates with all this setup we can now create our first embedding which is the easiest part on the open AI website under create embeddings all the information to create a request is available it only really needs two things the model as well as the input for the model in this example we'll be using open ai's text embedding using ada002 which is also the cheapest version and for the input it can be any type of text so here on Postman I'm gonna head over to create this body request under the body I'm going to select to pass in some raw information and this will be in the form of a Json object I'll add in some curly brackets and here I'm going to pass in the string for the model as well as the string for the input in true programming fashion I'll have this input say hello world and that's it all that's left is to send this post request to open Ai and its API and here's the response we've created our first embedding quite easy and this is also the vector for the embedding down here and if you preview it it quite a lot of numbers let me now show a few different types of examples of embeddings that could be created there are single word embeddings these are things like dog or cat where the embedding is a single word and the embedding Vector is generated when it's sent to open AI these would be used in situations where you might want to perform a search you can do multi-word embeddings too so I could for example have a small sentence like open AI vectors and embeddings are easy and this creates a more nuanced embedding but for the most part it looks the same to us humans the strength of embeddings is where you chunk large bits of information together such as paragraphs or entire sections of documents to create an embedding that can be then later drawn upon when you search a database and for some useful information Ada version 2 allows for a maximum input of 8 000 tokens more or less which is around 30 000 letters or characters and to give you an idea of how much that is a page from a contract or a legal document can be up to about 3 000 characters on a single page this means you could probably embed about 10 pages in a single request to give you an idea of what this looks like a let me copy paste this entire page from this NDA contract then I'm going to jump into Postman and paste this here into the input removing any line breaks or additional spaces this gives you an idea of just how much is being currently embedded and this embed request takes just about as much time as a single word now that we can create embeddings we need to store them somewhere openai doesn't provide databases so we'll need to create our own and a database full of embeddings is often referred to as a vector database I'm going to use a provider called a single store they provide a real-time unified distributed SQL database which also is quite easy to use since it's in the cloud and on top of that they allow for you to incorporate Vector databases straight in there so what we'll do is set up a database and we'll start storing our embeddings and then also start searching through them first you'll want to set up an account it's free and comes with additional credit as well as the ability to set up unlimited databases I already have an account so I'm just going to sign in via Google on the main dashboard the first thing you'll want to do is create a workspace and in here we'll soon create a database for your workspace you can call it anything in mind I'm going to call it open AI Vector database I'm going to select the cloud platform for AWS but you can also select a Google or Microsoft Azure platform and for the region I would recommend just picking the one that's closest to you but I'm just going to leave it to the default on us West next I'll select the button next on this page I can set how fast the workspace is how many CPUs how much RAM it has but for this tutorial I'm not going to do much I'm just going to keep it at the bare minimum I'll leave the advanced settings as the default and select next this workstation will set up in the background and once complete I'm going to create a database on it let me close these info panels here we'll have the workspace on the left hand side and on the right side I'm going to select create a database I'm going to name this database open AI database and then select to create it once it's created it'll show up here in the interface and I can select to view it it doesn't have any tables or data just now so what we'll do next is head over to the SQL editor where we can run some simple commands to create a table and then input some data make sure that on the top you have selected the database you're currently using so in this case I'm going to select the database open AI database and now I'm going to write a simple SQL query it'll be create table if not exist and I'll have the name of the table as my table actually I'll do my Vector table here in Brackets I'm then going to put in the different types of columns I'm going to have the first will be the text the original text as a text type the next will be the vector and the vector will be a blob type you can have different types such as integers and numbers and decimals and you can even have a larger table with things like an ID and other attributes like a URL but I'm keeping this nice and simple for this demo let's run this command and this has created the table here we can see a logged below I can head into the database and select it and view the table additionally I can view the different columns as well as the data type additionally I can head over to sample.2 have a look at the rows which don't exist yet let's create our first row and insert a vector into this database what I'm going to do is open up the SQL editor in here I'm going to write some new syntax to insert a row this is insert into the table name which is a my Vector table then in Brackets text and Vector being the two attributes we want to fill and then the values for those which in the this case will be the ones that we have in Postman so I'm going to open up a postman here I'm going to copy out of this input hello world and I'm going to paste this here into the values next I'll put in the embedding here I'm going to pass in Json underscore array underscore pack which is necessary to turn it into this blob structure and in here I'm going to pass in the string of the array this is in Postman here under embedding and it's this huge block of numbers I'll select to copy all of them and then here in the interface I'll paste them back in and that's it now I simply have to run the command and this should enter this embedding into the vector database which it just has here we can see the results one row added and if I head over to the open AI database under my Vector table I'll be able to see under sample data this a new row has been added now it's time to add some more data because we can't really have a database with just a single row what I'm going to do is is head back to postman search through the history of all the posts we've done so far and pull out some of these examples to place in as a rose inside of the database so I've got this one here for open AI vectors and embeddings are easy I'll copy the input as well as the embedding straight into single store wow it always surprises me how many dimensions there are to this embedding now I'll enter this and insert this row heading back to the database I'll be able to view the sample data and here it is let me try this one more time with a larger example I've got this one here which was the document which had quite a few different characters in here and the process is more or less the same copying out the text then copying the embedding vector and then finally inserting the row into single store let me show you the three rows that now exist that are quite simple but enough for us to start performing searches searching a vector database for embeddings is actually quite simple the first step is to identify what you want to search for so for example we might want to search for anything related to open AI next we have to create an embedding for our search term in this case we would create an embedding for the word open Ai and then finally we would perform a search in the database against the existing embeddings this would return a list with the closest similarity being at the top heading over to the SQL editor I'm going to write this query by writing the following select text which is one of the columns then I'm going to pass in dot underscore product passing in the next column the vector column and adding in Json underscore array underscore pack I'll leave this empty for now but this is where I'll add the array and then convert this to a score that we can use for ranking results next I'll grab this from my Vector table and I'll order it by the score in descending order this will be limited to just Five results now opening up Postman I'm going to need to create an amending to search so the embedding term I want to use is open AI this will create the vector here which I'm going to copy out and then paste back into the SQL query the SQL query will use this as a reference and use it as part of the scoring system so let me run this and show you the results here's our successful search of the vector database against the term open Ai and all the other vectors that are similar to it the scores are ratings essentially how highly ranked it is is how similar it is to the content already in the database I could create another search term embedding here called hello Earth and use the vector that I receive as a search and have a try to see if this changes up the results pasting this into the query here on the vector database I'll find that now hello world is the top ranking result built with a much higher score than all the others and this is fundamentally how Vector searching Works in this part of the video I'm going to create an actual function using JavaScript on node.js to interact with embeddings first let me create a brand new folder called openai vectors and embeddings next I'm going to do a fetched request to the open AI API so here inside of the file index.js I'm going to write in a small header here to connect up to open AI in this header I'm going to make the content type as application Json and I'm also going to pass in the authorization which will have the key here as a barrier token similar to what I did inside of Postman the safe and secure way to do this is using an environmental key but for this example here I'm just going to pass in my token straight away to make a life easier for myself this of course is something you want to keep secure and after this video I'll be deleting this key now Now to create the function I'm going to create a async function and I'm going to call this create embedding it's only going to pass in one item which will be the text and this is the text that we're going to embed and I'm going to use this to pass to an API now I'm not going to use axios or anything like that I'm just doing a regular fetch command here so I'm going to pass in let response equals await fetch I'm going to pass in the URL for open ai's version 1 of embeddings which is just a forward slash V1 forward slash embeddings then I'm going to pass in a couple of parameters this will be a post request so I'll set the method here as a post I'll pass in the headers that we just defined earlier and then finally I'll pass in the body which I'm going to Json stringify and this is the post data there's two items here just like before the model itself which is the text embedding Ada model and the second item is the input which is the text we want to embed next I'll listen to see if there is a response from the server and that response is okay and then I just want to console log it out and return it as part of this function here I'll write response Json so it'll turn it into a Json object and then I'll pull out the data from that I'll console log out the data and then I'll return the data this function is now complete so what I'm going to do is call it to test it out here I'm going to call create embeddings and just pass in hello world next I'll open up the terminal and run node index.js to run this function and just like on Postman this embedding gets returned quite quickly we can see it just over here with the array here on the right hand side and we can plug this into a database next what can we do with this well there's quite a few things you can import a PDF library to read PDFs chunk them and and then store them in a database so that they could be retrieved later and searched through you could do the same for websites or you could do the same for all sorts of things if you've stayed until this point you definitely want to learn more about openai check out my new book teach me open Ai and GPT this is a digital book I've put on sale for 49 that covers everything from how to use the openai API how to do fine tuning and much more in a really visually interesting and fun way it's over 48 pages and it's in the description below these were just the basics if you want to learn more have a look at my video here on integrating openai a database and even a web app together and if you haven't already this is the end of the video so don't forget to hit

Channel: Adrian Twarog

Views: 21,043

Keywords: embeddings, vector database, openai, open ai, ai, openai embedding, embedding openai, embedding, vector openai, openai vector, text embedding, code embedding, openai api, what are embeddings, what is embedding, what is vector database, embedding function, javascript, postman, js, nodejs, programming, create embeddings, search embeddings, pdf embeddings, text embeddings, creates an embedding vector, embedding vector, openai embeddings nodejs, word embeddings, openai embeddings

Id: ySus5ZS0b94

Length: 18min 41sec (1121 seconds)

Published: Fri Jun 30 2023