Building a Summarization System with LangChain and GPT-3 - Part 1

okay in this video we're going to look at building a summarization system and summarization is a challenge that has been around for a long time there are lots of issues to do with this that people have faced in the past one of the the most obvious ones is that each person tends to summarize things differently so often you'll find that one person wants a summary One Way someone else wants it a different way and in the past that's really meant that you had to either train separate models or find versions of models and the data sets for doing fine tuning and stuff in the past were very limited too so you had things like CNN news data set or this kind of thing where the summaries were basically based on articles it's a very tight kind of Niche and if you had something that was a little bit out of domain the models wouldn't do that well at all so this changed a lot when the whole sort of instruct tuning started to happen so the the sort of first ideas of training something for instruct tuning and then all obviously the RL HF tuning that has been done for the latest models like chat GPT Etc and turns out that now to do summarization we can really get good results just through prompting the model in the right way and passing in the context that we want to be summarized so let's look at how to do this in Lang chain and I'll give you some tips about it as we go along we'll look at different ways that you can do it in Lang chain and some of the advantages and disadvantages of each of these so you can see here I've just brought in the normal sort of Stack Lang chain open AI we've got tick token as well which is basically just for counting tokens that is going to be used in here so first just setting up a simple summarization chain so here you can see we're going to be bringing in the normal sorts of things we've got our prompt template we've got a large language model chain one of the new things we're bringing in is the text splitter so how we're going to actually split up the text for this and then we're also going to bring bring some different chains in for how we deal with with text that is longer and this raises a number of issues in the past or one of the biggest challenges you had in in summarization was that the models could only do 512 tokens so you had to split everything up now currently as I'm recording this the chat GPT model can go out to 4096 tokens there are rumors of new models coming out with eight thousand and thirty two thousand token spans they're definitely going to change summarization in a big way but at the moment we've got better than we had in the past with 512 but we still can't put in perhaps a whole book or something like that into it so we need some ways to be able to split up the text and this is where the text splitter and the map reduce and you'll see stuffing and you'll see refining different sort of ways of splitting up documents so the document that I'm using here is a very simple document that I found on online quite famous how to book how to win friends and influence people I've just taken the first chapter of that and stuck it in a text file as far as I know this was written close on 100 years ago or something like that so I think it's out of copyright for us to be able to use this so you can see that what this does is it goes through and it splits it up and actually we're ending up with four different texts that we'll be able to put into the model this is the first thing is making our docs and then if we look at the docs we can see what the actual content is like we go through and see and I suggest when you first start this out pick some content that you know so you can get a sense of how good the summary is and which techniques make the summary better which techniques don't help for that so we've got this in now we're going to talk about is there are three different uh ways to do the summarization and really that there are ways to combine these documents that we've split up how do we Summarize each each sort of chunk of these documents and use them so the first one we're going to look at is mapreduce so this is one of the more common ones and these are just taken from the Lang chain dock so I put the link there that you can go through I the idea with mapreduce is that you've got we've got our text split up into four chunks what it would do then is it would do a summary for each chunk so we would end up with four summaries and then it would do a summary of the summaries to end up with a final summary so you've got initial prompt and a summary for each chunk of data then you've got you can use a different prompt or the same prompt to basically combine all these into one overall summary here so the advantages of this is that this can scale to much larger documents and to multiple documents as well than some of the other ones I will show you it also allows because each of the calls is totally separate to each other we can actually do those in parallel so this allows you to do multiple calls at the same time summaries but you will find that this can use up a lot of tokens quite quickly the Cons with this is that that this is definitely a lot more calls to the language model and then also when you're combining you're often going to lose some kind of some information in there unfortunately there's no simple answer for that because as you'll see later on something like the stuff document we the challenge there is that we can't go with long text so anyway let's look at doing a mapreduce summarized chain so here we're basically bringing in types of chains the summarize chain we're loading in this I've also got Tech scrap just to print this out and then basically I just set up a really simple chain I'm trying to use the large language model we're using a temperature set to zero to try and stop it from hallucinating and we're just going to run this through and it will go through and it will end up with a nice paragraph of text that is our final summary here so if we want to look at what the two prompts are remember we've got the prompt summarizing each individual chunk and then we've got the prompt for combining this it turns out in this case those prompts are the same but we can just go through we can look at what those prompts are and see them so we can see that the prompt is nothing fancy here we cannot write a concise summary of the following we pass in the text and then we've got right you concise summary and it will output our summary and then we're just doing the same thing later on with a summary of summaries so one of the things we can do is we can run we can set this same thing up and run it as verbose equals true so we can see the output and we can see actually what's going on so when we look at this let me just scroll up a bit we can see that okay we've got the our prompt write a concise summary of and then we've got this whole chapter was about why this book was written and we can see that it's taking in all the text and it will then make a summary of that chunk and then at the same time it's going to be if it's doing it in parallel it will then do be doing the second chunk and it can do all these chunks in parallel for this and then finally we get to the chunk of where we've got the summaries so in this case four of them and we're combining them into to a final summary and this is where our summary is here and you'll see that there are bits of this that will be clear in these summaries so if you went through a look at it again this is why I recommend you pick some text that you know quite well and try that so this is a map reduce summary technique the next one up is the is stuffing stuffing is what it says it's just trying to stuff it all into a single call to the large language model so this makes sense if you've got a large language model with a big token span so going for something like the 4096 is okay obviously the newer models with even wider token spans would be better but if you're using something where you've only got 512 tokens available this is probably not the one you want to use that much so here we can basically take everything that we've got and we're going to basically run it through so the pros of this are a single call and also when it's the model actually has access to all of raw information at once so it can use all the bits so that things are important and that's key for if we think about the mapreduce thing if we've got something that's slightly important in chunk three and there's parts of it that are slightly important in chunk two it might drop out that information on both of those chunks because individually it wasn't a large piece of information for chunk two or for chunk three but combined they would be useful for a summary of the whole thing so that's one of the disadvantages of the mapreduce here we can basically see how this goes I thought we'd also look at doing some different kinds of summaries so here we've written a prompt template and the prompt template is just going to be write a concise bullet point summary of the following and then we're passing this in so the idea here is now rather than getting a paragraph summary out we want to get factoids and bullet points out that we can use for this so we just to do this we basically make our prompt we pass all this in to just create this and then we pass the prompt template into when we Define the the chain so again we've just got load summarized chain the type is going to be stuff this time we're passing in the bullet point template and then we run this through and sure enough you can see now we've got bullet points of each of the key things this is allows us to do a different kind of thing now we could have done that with mapreduce as well it's not like stuff is the only one for doing that if we come and look here we can do the that our own prompt with the mapreduce one so the difference is that if you remember the mapreducers got two sets of prompts right one for doing the individual chunks and one for doing the combining chunk and that's what you've got here so if you just passed in prompt equals bullet point prompt it you'll get an error so you need to pass in for when you're doing the map reduce you need to pass in the map prompt and the combined prompt and if you've already initialized this you can actually just go through and overwrite them so that's just showing you how you could go through and overwrite them we then would get something back you've seen the mapreduce we're getting more bullet points back so it's obviously decided that there are certain things and we're getting different bullet points back yet again this is why I say try it out with some texts so you get a sense of what the summary is doing and then we can basically just put that all together okay one of the things I want to show you too was that if we go through on this one I'm just going to run this one so that we can look at something in a second okay here we're just passing in our prompt template we're just setting it up just like before the big difference we're going to be doing here is we're going to do return intermediate steps equals true so what is that about you'll see that now when I'm taking the output I'm not just taking the whole output I've now got a dictionary where I can access different types of outputs so I can get here and I'm only returning the outputs here for this I can take the output text which is the final text that we've got here but I can also take the intermediate steps so if you look at this one so this is like our final output text but this one here here is the summary of just the first chunk and then if I want to see the summary of just the second chunk I can come in and look at that and I can see okay so summary of the third chunk no problem we can look at that so the advantage with this one is if you wanted to reuse these this could be useful and ideally you want to keep some of this data so that eventually you can use this data for fine tuning if you're putting this into production you'd want to use a way of storing this data to be able to do your own RL HF in the future for this kind of thing so the intermediate steps is a key thing for this the third type of summarization is the idea of this refined summarization so this is we're not passing in everything at once we're passing in multiple channels but we're not doing it in the same way the map reduce one what we're doing here is where we're passing in first chunk with an initial prompt right so the first chunk just passes is and what we pass into the first step and then we take the output of that and we pass that in so that's the summary of the first Chunk we pass that in with the input of the second chunk so by doing that we're now adding to the summary of the first chunk so it's like we go step one summarize the first chunk step two summarize chunk two plus the summary from chunk one at the same time and then step three summarize the summary from the output of Chunk two which is actually a summary of Chunk two and chunk one with the input of Chunk three so you're refining this over time as you go through this this has advantages so that you can pull more relevant context out challenge with this though is it's very much a sequential thing so you can't do these calls independently so if you've got a long document this can be very slow it could be quite slow to go through and do this but you might find that for your particular task it gets better results so you can see here that we've gone through it's generated a longer summary in there you can go through and try it out just like with before we can basically put in our own prompt and we can play around with the prompt but you'll see that the way the pump is done is slightly different so the first prompt is similar to what we had before right it's just for that chunk of information but the refine template is where it's going to basically be telling your job is to produce a final summary we have provided an existing summary up to a certain point and that's passing in where it was from the previous chunk and then we have the opportunity to refine the existing summary only if needed with some more context below and then we're passing in the next chunk there so by going through that we're building this up over time we can do the whole immediate steps thing as well and you can see that the first output and then you could go through and see what was the summary like at each chunk so this can be useful for just testing am I losing I've got 30 chunks where it gets to chunk 25 has a totally forgotten what was it chunk one and this would allow you to go through and see where what happens to the summary over time for this thing so these sort of give you the three basics of getting started with the summarization system in the next video we will look at going through and adding in a checker to this so that you can actually have the language model check the output of some facts against the input text to see like a reverse thing to make sure that it hasn't hallucinated and that can be really useful for improving the quality of the summaries as well so we'll look at that chain in the next video as always if you have any questions please put them in the comments below if you found this useful please click like And subscribe and if you've got anything that you would like me to cover in some of the videos please also let me know in the comments below thank you for watching I'll talk to you in the next video

Channel: Sam Witteveen

Views: 18,025

Keywords: OpenAI, LangChain, LangChain Tools, summarization, python, machine learning, artificial intelligence, natural language processing, nlp, Huggingface, langchain, langchain ai, langchain in python, gpt 3 open source, gpt-3.5-turbo, gpt 3, prompt engineering, prompt engineering gpt 3, llm course, large language model, llm, gpt index, gpt 3 chatbot, langchain prompt, gpt 3 tutorial, gpt 3 tutorial python, gpt 3.5 python, gpt 3 explained, HuggingFace Hub, AI Model for summarization

Id: LNq_2s_H01Y

Length: 15min 1sec (901 seconds)

Published: Fri Mar 10 2023