Is Falcon LLM the OpenAI Alternative? An Experimental Setup with LangChain

so if you are serious about learning how to work with large language models then this is an essential skill you must be able to work with different models if you are into creating applications with large language models then you know that the current golden standard is to use the models from open Ai and then pair it up with a framework like Lang chain to hook it all up and create your applications and that is something I've been covering on my YouTube channel lots of examples there but there are some downsides to using just the models from openai first of all the API costs money that's one and the second one is that you might not want to share all data that you want to send to these large language models with open AI so these could be sensitive private information or company information that you want to keep secret so I've also been getting tons of questions on how to counter this how to go about that so in this video I'm going to show you how you can work with open source large language models as well and in particular the Falcon model which has been getting a lot of attention and hype lately because it's been outperforming all the other open source models that are currently out there if we look at the leaderboards from hugging face so I've prepared an example for you which we will run through together so there are some instructions and there is some code that you can access but what we are going to do is we are going to test the Falcon and then in particular the 7 billion parameter one so that's not the fully trained one with the 40 billion parameters because that takes forever to ask questions and it takes a really long time to load and I found that the 7 billion one and particularly the instruct version is something you can work really well with using the code that I will provide to you so we are going to do some experiments where we compare it to the text DaVinci 3 model from open Ai and then put it to the test in a task summarization task so this is going to be pretty interesting so you'll learn how to use open source large language models from hugging face then you'll learn how to set them up and compare them to open AI models and as a bonus of this video you'll also learn how to summarize large text using Lang chain and the summarization methods that are in there so this is going to be a very important video because as we progress in the era of AI and other open source models will come out the gap between where open AI currently is with their models and open source models will likely get smaller and smaller and smaller until we potentially and many believe this will be the case open source models will be better than what currently open AI is offering so if you're serious about learning how to work with large language models then I would say this is an essential skill so knowing how to work with various models paid models different apis but also the open source models which are for example here available on Hawking phase and then setting up quick little experiments to compare and validate all the models and the results in order to pick the model that is the best suitable for your specific use case so that is what we will do in this video let let's get into it so in order to follow along you first of all need access to this repository over here the link will be in the description and we will be working from the models directory in here where the example is and also the python file that we will be using now in order to follow along you need a python installation and some basic understanding about Lang chain if you don't have that already I would recommend you watch my previous video on working with Lang chain that will set you up and explain how everything works but if you already understand this then we can dive straight into it and what you can do is first of all like clone this repository and then more specifically go into the models folder within your favorite IDE so your IDE of choice so for me that is vs code I've opened up the file over here and we're in the falconmodel dot Pi so if I load up a python interactive session over here which is something I've also been getting tons of questions about them how to do that link to set that up in vs go mode will be also in the description we can do some basic Imports and then to configure everything the only thing you need if you want to run the open source models from hugging face is a hugging face API token so if we go to huggingface.co then log in create an account it's free if you don't have one already and you go to the settings and then the access tokens you can create a new API key over here so as you can see if one for Lang chain already with read accessibility but you can create a new one say what's it for and then either read or write read would be sufficient generate the token you can show it and then copy it and then the next step that you have to do is set that up in an environment file so within the GitHub repository there is an example already it's an empty one but what you can do is if you turn or change this name into dot end like you see over here and then just change the hugging phase API AI token with the token you've just generated then you're good to go what this basically allows us to do is using the dot End Library to load our environment variable of the key into our variable over here and that will allow us to communicate with hugging face alright so there will also be a requirements.txt for this project to make sure you have all the dependencies and all the PIP libraries basically installed but it's dot end Lang chain and I think that's it then you're good to go alright so with those instructions out of the way we can now get started with the models from hugging face so first what we do is we do the load.f any fine dot end to load our environment variable that we just set and then make sure that the token is accessible over here and then we can continue to the actual interesting part and that is using the hugging face hub from Lang chain so you can see we import that over here and then what we can do is I'll show you how to interact with these models so it starts off with a repository ID and if we come back to hugging phase you can see that each model basically that we're looking at right now has a a repo and you can basically copy that and then we can use that to interact with the model now like I've set the 40 billion one I've tried it but it takes forever to load I don't know if it's even possible to do it through the API or that you have to set it up on some kind of like heavy Surfer but the 7 billion one and like I said the instruct version works really well so we are going to copy that it's already in there but you can basically like put that in here but you can also swap that around with some of the other models that are available on hugging face or in the leaderboards so that's really an interesting part so that's how you can swap them out and then basically we are going to create our large language model through a lang chain object basically like we normally do and we can give some model parameters in there as well so through this we basically specified your repository idea and for now we set a pretty low temperature and we set the max new tokens to 500 meaning that the response that we get from the model will add a maximum be 500 tokens so let's run that and store that within our interactive session you can refer to the hugging face hop documentation here in the Lang chain documentation to get a bit of an understanding of how this works but it's basically the same example that I've just walked you through but this is where the original documentation is from and here you can also see that they're using a Google model in this example so that is how that would work so the next step over here is to create a prompt template and a large language model chain and we're just following along with the sample from Lang chain where the template is hey here's a question and then we fill in our actual prompts that we fill in later and then we say answer this question basically let's think step by step so we create a prompt template and then a large language model chain again if all of these concepts are new to you and you don't understand them I refer back to my previous video on the basics of working with Lang chain and then it will make sense for you I'm assuming you already understand that and then we're going to run this chain so let me first like quickly complete or store this code and then we have a question so we start off with a very interesting very hard question how do I make a sandwich and we are going to check like okay how is the Falcon 7 billion model going to respond to this so we have to question we basically first get the response goes pretty quickly and then we can wrap that that is basically just to make it nice and pretty when we print it and there we go we have our first response so you need to gather the ingredients you'll need bread meat cheese condiments and toppings once you have all your ingredients you need to take a slice of bread okay sounds like a pretty solid instruction on how to create a sandwich this is pretty cool right because this is a completely free model like no credit card required no data sharing with open AI not so interesting question but hey we're just getting started so we already have it up and running with just these few lines of code and to me that is still like so amazing how these open source models if they are made available and if they work really well you can just access them with couple lines of code and you can basically create cool applications with them like completely for free I'm really excited to see how these models will develop over time and when we have like the like the Cross Point where these open source models are really better than the ones from open AI but now let's take it a step further and set up a quick little experiment that we can do so I'm going to download one of the transcripts from my YouTube videos and this one in particular that I did recently on flow wise and through the YouTube loader also from langchain we are going to put in that video URL and then we're going to get the transcript so this is how that works and again that's also covered in one of my previous videos but we can basically load the transcript and here you can see we just have the whole transcript basically of that video which is automatically generated by YouTube so for every video this is publicly available information you can plug in any URL and you can get the transcription based on on the YouTube algorithm and then we are going to split this up into different documents basically because as you might know when you work with large language models and these and these apis you are limited by the amount of tokens that you can send to these apis so we're going to set up a text splitter and basically what this will do is we'll just chunk this whole document this whole transcript up into various documents and we can have a look over here so there's about six documents in total seems to be yes so six splits basically so we took the whole transcript and we chopped it up and now this next part is pretty interesting and this is summarization so this again is something that large language models currently struggle with due to these limitations in the amount of tokens that you can send to the API so Lang chain has a cool little built-in method load summarize chain basically where you can do or create this summarization iteratively so you first like split it up the whole transcript and then create little summarization for each of the splits basically that you've created and then you throw those onto one pile again and then go iteratively basically on that to create your final summarization now and that is exactly what we will be testing this Falcon 7 billion parameter model uh with and then compare it to open AI to see how it performs so we've splited up the documents and now the next step is to load this chain and quick little side tip that you can do if you print the prompt template and also the combine template you can see how Lang chain is handling this so it basically is a prompt to say like write a concise summary of the following then you put in the text and then it will give you the concise summary and by default as you can see this template is basically the same for first creating the initial summarizations and then also when you combine everything and create a summarization of depth it will use the same prompt template but you can change that by using the parameters as you can see it's in the comments over here but you can just add a map prompt over here and then just change that so map prompt equals and then you give your map prompt over here and you can also change the combined prompt so that is how you would do that but for now we're just going to leave it at default and we're just going to run the docs and we set for both to true so it will also show us kind of like what's going on under the hood so here you can see all the chunks basically first write a concise summary of the following and then this is the first introduction sentence that I basically start with and then this is somewhere later in the video and so on and so on and then it basically combines all of that and then finally if we look at the output summary we can see that we have a string of text over here and then we can just wrap that so that if we print it we have that on one line and what we have flow wise is a visual UI Builder that allows users to build large language models apps in minutes that's a good one that's basically how I start the video the tutorial covers setting up a free API key and cloning the flowwise Repository I believe that's also correct because we use not it's not really we use a free Pinecone API key but we also use the open AI API key which is free to set up but it costs money if we want to interact with the API we do clone the flowwise repository so that is correct and then the project is then cloned in a lang chain experiment project is opened in a terminal that's also correct the tutorial concludes with a step-by-step guide on how to integrate chip GPT with company data that's not entirely true that's basically an example of me explaining kind of projects that I'm currently working on for my freelance work but okay it seems to be at least the start is quite accurate and remember this is a totally free model and also text summarization using this method is pretty hard because we are splitting up all the context basically and if the cut or the split is right in the middle of like a talking point then the ending and the beginning of the other chunk doesn't really make sense so if you summarize it like separately it could get messy so it's a pretty interesting task to compare these models on so now let's see what we get if we use open AI so we have another text splitter over here which is kind of redundant because we're using the same but now we're just going to say hey we are going to create a new large language model object basically new llm but now we use openai so let's run this and again if you want to follow along this does require you to put in your open AI API key in the dot end file as well so you need one free to set up but you do get charged once you query it so we set up the model and then we create another chain and then we plug the open AI llm in here everything else works the same that's the nice thing about using Lang chain that's it's unified in a way that you just specify the model and then once you have that in place you have that object you can just like completely copy copy your code that's really awesome so we can again create the output summary by running it and here you can see basically the same thing so now the model will go to action and we will iteratively query the open AI API to get the summaries of all the chunks and then boom it's finished over here we can wrap the text again and then print it okay so what do we have over here so this article discusses flowwise Ai and open source visual UI Builder that allows users to quickly build large language models apps so the first sentence is kind of like the same although this says that it's an article but they do talk about flow wise and they recognize that it's an open source visual UI Builder to quickly create large language models apps so those are the same and then it refers back to like the article which is the video but okay it explains how to set up flowwise connected to data and build a conversational AI so this I would say is a lot better already because this is really what what we do in this video it also explains how to use Lang chain a comprehensive framework for developing applications powered by large language models so it's Lang chain under the hood so yeah in some way I talk about language and how to use flowwise AI to quickly prototype AI projects finally it shows how to use flowwise AI to sell AI service clients as a freelancer okay so this is definitely the winner because really the main message of that video was how you can use flow wise to like quickly prototype AI projects and then I conclude this video by basically saying hey I work as a freelancer I work with clients and this I could see how I could use this as a tool to quickly spin up like demos and prototypes and then go from there basically so overall the open AI model is still the clear winner but if we compare like the the workflow and and what we're getting it's getting close so let's see so the temperature is set pretty low but let's just see if we run it one more time let's also see like how uh deterministic this one is so I load the Falcon model again I will create a summary one more time go over it it is really fast though which is quite interesting it's faster than than open AI for sure right now so what do we have do we have kind of like a similar answer so flowwise visual UI Builder yeah seems to be the same okay so that's good because we set the temperature pretty low so to 0.1 so we would expect no creative Styles here basically and just giving us the results basically so it is good in that sense but I would say at least from this little experiment it's nowhere near the capabilities of open AIS models right now but I will definitely run more experiments with this and also keep in mind that this is the 7 billion parameter model and the one that has been getting all the hype is the 14 billion parameter one so if you know if you know a way how to run this effectively like on your local machine or through some server please let me know in the comments because I do want to experiment with this one alright so you now know how to work with the open source models from hugging phase like I've said very valuable skill to have if you're serious about learning how to work with large language models we've looked at the Falcon 7 billion parameter model we set up a quick little experiment created some text summarization and it's overall to give you an understanding a good idea of how to go about working with different models how you can compare them and it's really up to you and the use case at hand to determine what the best model is is there a budget can you share data with openai how fast does the model have to be these are all questions that you have to ask yourself if you're working on a project with large language models alright and that's it for this video now please leave a like down below and also subscribe to the channel to make sure you stay up to date update on all things going on with large language models and Ai and now if you're interested in using these skills to sell them as a surface to potential clients as a freelancer but you don't know where to start then check out data freelancer first link in the description that's it for now and then I'll see you in the next one foreign thank you

Channel: Dave Ebbelaar

Views: 17,940

Keywords: data science, python, machine learning, vscode, data analytics, data science tips, data science 2023, artificial intelligence, ai, tutorial, how to, falcon model, free, openai alternative, langchain open-source, open source, free llm, huggingface, free openai model, free chatgpt model, free api, no creditcard, open-source llm models, falcon-40b, falcon-7b, falcon llm, falcon llm hugging face

Id: ReX4Kxp7Ty4

Length: 20min 19sec (1219 seconds)

Published: Thu Jun 08 2023