Fine-tuning OpenAI's GPT 3.5 for LangChain Agents

today we're going to take a look at fine-tuning GP 3.5 this is a very new feature that has been released by open Ai and it gives us the ability to fine-tune our own custom GT 3.5 models so we can feed it a ton of our own conversations that are the ideal conversations we would like our chat Bots or conversational agent to have and with those open AI we'll fine-tune a GT 3.5 model and give that back to us as a sort of custom model that we can then just retrieve from the API given a we'll basically get like a unique ID for each model so that's really cool because with these models I mean there's a few things that we can do to improve them we can do a ton of prompt engineering it takes a long time but usually doesn't get us all the way there we can also do retrieval augmentation again it helps in some cases but not in others and if we add fine tuning into that mix we actually have a lot of what we need in order to get almost any type of behavior that we'd like from our chat box so this is a very important feature and one that is going to become a key part of anyone's talking who is developing AI applications so a big part of fine tuning is always data preparation and collection we're going to touch on that in this video but I'm actually going to leave the majority of that for another video although it is very interesting so I definitely want to go into details of how I'm building these data sets but for now we'll just jump straight into the fine tuning component okay so we're gonna get started with this notebook here we have a few installs here we yeah we go ahead install those we're going to get our date set so again this is one that I've pre-built I would say there wasn't much manual work in this it's kind of all automated to build it essentially yeah you can you can kind of see what we have here these are all actually different but the system message for each one of these is the same but then you have like essentially like a conversation or a small segment of a conversation down here all of these are generated by gpt4 so we can almost think of this fine tuning as a form of knowledge distillation which is where you would use one model's knowledge and to sew that knowledge into another model through some sort of fine tuning approach which is what we're doing here now you can see for example one the records here we have the roll system this is the the template system prompt from Lang chain we have this role user and then we have you can see we have tools in here right so that's the bit that is the most important because what I want to do here is actually fine tune chat or tube T 3.5 to be a better agent and the issue that I have with GT 3.5 as an agent is that it really struggles to use this Json format so you can kind of see it here it just seems to struggle with it a lot GT4 does very well and that's why I've used GT4 to generate this data set GT 3.5 not so much so we've created this data set in order to essentially teach it how to better use that format okay now this is just focusing on almost like a retrieval tool only naturally in a real use case we would want more diverse conversations not just using a single tool we'd want multiple tools and we'd also want conversations that are not necessarily always using tools but for this example I think this is good enough so I'm just going to save this to a local Json lines file we can see over here we can open this and all we will see is this so we have our messages so this is the format we need to follow when we're training so we have a like a dictionary we have messages and then we have a list of meshes that we're going to be training on each one of these represents a single conversation okay we can go across like I said the first system message is the same for all of them again that's maybe something you'd want to modify or maybe you just use your particular format that you're using for a chat bot that you'd like to train so here we have next message role users we have the tools that the assistant can use okay see these are all going to be the same again and then we get to here and this is where things begin to change so here we have the end of the user message which is where we have a question from the user that is followed by the assistant responding with I want to use a vector Search tool and it uses this Json format that we've set up and what we have here is like a search query okay main focus of technical reports this technical report and you can see this is kind of coming from this right all of this is generated by gbt4 so we said okay this is a question based on some some information that we have from a data set uh make a search query which is what we have there and then continue and we get the context and we get later on an answer as well near the end okay here assistant Json action final answer focus of the technical report so on and so on is this right that's our format so then we want to go on to here where we are going to first upload the files that we're going to be using for fine tuning so that's our conversation Json lines file we're going to be using this openai file crate which is obviously a new method in the openai client so you do need to make sure you have updated the client uh the version I'm using here is zero two seven nine okay cool from that you we run this and we'll get a file ID we need to take that so we get it from here go to file ID and we're going to use it to create a fine tuning job okay so this is the actual like the fine tuning of our model and we specify what model we would like to fine tune at the moment this is our only option so we run that we should also sometimes it can take a little bit of time for this file to be available but you'll see if you if it pops up with an error here that means it just isn't available yet so you just need to wait a little bit it doesn't take too long though okay so now we can see we we have this fine-tuning job running and it says finish at null that's because obviously it takes a little bit of time for the model to be fine-tuned so we essentially need to just wait until this is not null anymore and once that is not nowhere we will also get our fine-tuned model ID which we'll need to use later on so we can yeah we'll get the job ID where is that it is here and using that job ID we can just retrieve the information for our fine-tuning job like so okay so we see it still hasn't finished and it won't actually finish for a while now you can also check this so this will just say hey look created fine tuning job and then fine-tuning job actually started okay and yeah they're the events what you can do is you can kind of set this so this will just keep looping through and every 100 seconds it's going to call the API and check if the fine-tuning job has finished you can either do that or you can check your emails so open AI once your fine tuning job has finished we'll also send you an email Okay so this does take a little bit of time I don't I think it was at least sort of 10 20 minutes when I ran it so probably probably more along 20 or so minutes so what I'm going to do is just pause this or solve this and I am just going to come to here which is the model that I fine-tuned before this video so this one is already ready and I can just I'm just going to go ahead and use this rather than waiting now okay once we have our fine-tuned model we'll get our you know we get the ID from here which is what I mentioned before and then we're going to I'm just going to test this okay so I'm actually going to use this fine tune model in a conversational agent through line chain so to do so yeah we need to fine-tune model ID I'm gonna use a like a retrieval chain I'm the the retrieval chain in line chain I find is a little bit slow and overly complicated so I'm going to just load my own custom chain from here like retrieval chain retrieval tool which is what we'll be using as and essentially it just makes less LM cores so retrieval will be faster okay and from there we just want to load our or create outcome stational agent as we typically would so we have a LM here we're just using our fine-tuned model name we yet we're using conversational buffer window memory we have this Vector DB chain which is going to be our tool which we initialize here so this tool is using pine cone so if you're following a line you can just get your environment and your API key from here Okay cool so once we have initialized all of those we can come down to here which is initializing our agent constitutional right description so conversational agent and we just run that cool now we can say okay tell me about llama2 and I actually I should just point out that in order to use this tool here you would need to index your data first so I have another notebook for that I'll leave a link to that at the top of the video so you can run it through that if you if you really want to go through the whole thing so okay let's go with tell me about llama two it's just going to give us an answer but what we want to see is that it's correctly using this Json format which it is doing which is really cool so yeah we see it for the Search tool action it did that I'm printing out more than I need to here I'm printing out the vector because I was debugging earlier uh we have a second like Json or agent action which is to provide the prime Lancer to the user and it is llama2 is a collection of pre-trained and fine-tune large language models so on and so on right okay cool so that is one question let's try another what makes llama 2 so special I think this is pulling from like the same context but we can come up to here again I'm printing out too much but it's fine so alarm 2 is special because it features a collection of pre-trained and fine-tuned language models Optimizer dialogue use cases these are always called so on and so on I think it's supposed to say llama app form open source chat mode models on most benchmarks tested and maybe substitute for closed Source models okay that's good now tell me about the Llama 2 red teaming again what we're looking for is the correct Json format which it handles nicely and yeah we get I think it's the same same good answer again right so all looks good everything's in the format that we would want and yeah that is how we would fine tune a GT 3.5 model on our own data set again like I said a big part of this is actually building data set this is just a very quick sort of demo example on how to do this so what I want to do going forwards is put together more a longer video on fine tuning where we'll kind of dive into actually building the data set which is honestly the more important part of all of this well at least we've seen the different endpoints for the open AI client that we need to use when we're fine tuning so for now I'll leave it there I hope this has been helpful and interesting so thank you very much for watching and I will see you again in the next one bye

Channel: James Briggs

Views: 33,639

Keywords: python, machine learning, artificial intelligence, natural language processing, nlp, openai, openai fine tuning, openai fine tuning example, openai fine tuning api, gpt-3.5-turbo, gpt-3.5-turbo-0613, gpt 3.5 fine tuning, fine tune llm, fine tune gpt 3, fine tune gpt 3 chatbot, langchain fine tune, langchain fine tune model, langchain agent, conversational ai, fine tune chatgpt, train gpt 3, gpt 3.5 training, gpt 3.5 chatbot, custom gpt 3 chatbot, fine tune ai model, james briggs

Id: boHXgQ5eQic

Length: 14min 14sec (854 seconds)

Published: Mon Aug 28 2023