DIY: Ollama Compare LLM Results Web App (Ollama, Python, Bottle, SQLite)

welcome back as you know I am Eli the computer guy and in today's class we are going to be creating Python scripts that allow us to compare the results that come back from different large language models so now you know in the modern world with large language models there are a crap ton of them that you can use some of them are small some of them are large some of them are fast some of them are slow some of them are precise some of them give you we really weird uh results so one of the problems that you might run into is basically the question of which large language model should I use and so what we're going to be doing today is we're going to be using ama if you don't know about AMA ama's cool we just did a class on AMA so basically this is a framework that allows us to run large language models on our local machine so we're going to use olama and we're going to download multiple different large language models onto our local machine and then what we're going to do is we're going to have a python script and we're going to plug in what our request is what our query is and then is going to do a loop and it's going to go to whatever language models we tell it to go to and it's going to pull back the results from those models it's going to show us how long it took to get the results and then we can sit there and we can take a look and we can say okay this is the query that we inputed and then here are the 2 3 4 5 10 results that we got back it's Dynamic we actually just create a list for the llms and then we can determine which llm is giving us the results uh that we want in the time frame uh that we want and so basically that is what we're going to be building in today's class let's go through a few demonstrations so you understand what we're trying to accomplish in today's class the first demonstration script is basically a commandline script and what we're going to do is we are going to have this text file so this is a text file that has information about our business we have hours of operations here and we're a restaurant so we have side items fries and salad we have sandwiches we have Entre uh and we have drinks down here so basically the idea is that we can ask these llms questions about information in this text file and it'll give us responses uh I will say it will give us responses whether or not they're correct responses that's a different story and so basically what I can do is I can click on the Run button and this pops up and it now says how can I help you and so I can plug in imagine if this is a you know a restaurant and I'm trying to figure out if they have vegan options you know what vegan options do you have so I can simply ask a question like that what vegan options do you have then I can hit enter and now it's going to be processing through if we go back and we take a look at info.txt so this is the thing uh so what vegan items we have fries we have salads we have a veggie sandwich we have a portabello sandwich we have a vegan lasagna and we have soda and water so those are the vegan options that are possible not necessarily what we're going to get okay uh so we have from model llama 2 7B the 7 billion parameter uh response the vegan options are the veggie sandwich for $9 and the vegan salad for $20 um and that's uh that's all we get for the 7B version if we go down we go through the 13 llama 2 13B version and we get the veggie sandwich the portello wrap and the vegan lasagna so those are here we also have this time so it shows that the first request took 7 Seconds the second request for 13B took 15.17 seconds model pi oh my golly this just gives us a response like look at this look at this like uh you know llama 27b that's the response llama 213b that's a response Pi so this is Microsoft's model response sure here are some vegan options based on the menu items provided fries and soda or milkshake that's not vegan salad and water or milk that's not vegan veggie sandwich okay portella wrap remember to check with your server and then it just gives you so much more so what the hell I don't even know what this is this is not even a joke uh and anyways and that come back came back in about 10 seconds uh then we have the mistl model down here response of vegan options of the establishment are sides fries and salad sandwiches veggie sandwich and port Bell wrap Entre vegan lasagna and drinks water so mistl is probably the best response here and we get this back in 7 Seconds now the important thing again one of the things you have to be thinking about if you're going to be coding right is how are you going to parse the response it's not simply good enough that the llm gives you a response but how are you going to turn that response into something to actually give the uh the end user and that's the other interesting thing with this is you can take a look and you can try to figure out you know how how would you code uh to parse this particular me and so this this is the command line version now if we go over uh here uh basically this is a web app version so it's the same type of thing only we have a web app and a SQL light back end so if I run this so I'm using bottle you know if you follow me you know I like bottle uh let me see here we go to 127.0.0.1 and now we have this web app so these are the previous queries that I done uh and they're they're basically they're they're together for the query you know what days are you open at noon mistal Pi llama 2 llama 27b and it gives you all this information but this actually gives it to you in like this web app format so I can come here and I can say um how much is a burger and fries again think about that that's one of the big things with llm that's supposed to be this natural language processing that I can just plug in whatever uh question I want and you know it'll it'll give a response one of the interesting things with this is realizing how bad the llm responses actually are I may have to do a bit of a little bit of a Time speed um jump cut here just because this does take a while so this is actually processing through four different uh requests so it goes to mistol it goes to Pi it goes to llama and it goes to llama to 7B and so all of that together you know takes about 30 or 40 seconds okay so there we go now we have our answer from mistol a burger and fries from this establishment cost $1 since a burger is priced at $10 and a side of fries fries is $1 right uh based on the information provided a burger fries typically costs around $1 this includes a cost of the sandwich at $10 plus any sides like fries and an additional cost of $5 I don't again where does that even come from llama 2 13B a burger and fries would cost $10 for the burger $1 for the fries for for a total of 11 there we go and llama 2 7ba burger and fries at this restaurant cost $11 and again this gives us our little time codes here and so this is for if we're going to be asking a question about again a type of data file text file the other thing that we can do though is uh basically we're going to close out of this and the final demonstration is basically where we can just ask a random question again if you want to play with this and you don't want to focus on actually pulling in information out of a file because again it's quirky it gets a little disputing after a while if you try to pull information out of a file so this this uh particular script allows us to just simply ask a question uh so we can run this it goes to uh again 127.0.0.1 let me just open this up okay so basically this is the same type of app it's just not reading from a text file like before and I can say um I don't know how do you make ice cream I don't know some random question like that and then we can hit submit and it'll give us a whole long explanation from all of these different models on how you do something such as uh create ice cream which is kind of interesting I do find that the llms seem to be better when you don't feed them information like if you just ask them like random questions like how to make a sandwich or whatever they actually do pretty good if you ask llms to tell you a story about a snail and a sailboat they will give you that story licky split they do a great job of that um if you ask an llm whether a particular store is open at 7 o' on Fridays um it's not so pretty anyways for this how do you make ice cream Moll to make ice cream combined ingredients as milk sugar blasé Blas blasé so these are all within pre- tags I just want to show you so you can know how to basically how to try to uh parse the information when you get it um so this this doesn't have to look like this I just have them in pre- tags I'll show you that in the uh in the actual code examples uh pie here I don't have personal beliefs or thoughts but I can provide a general explanation about oh oops that's how do you jump oops ah I lost it okay there you go uh to make ice cream mix together heavy cream milk sugar your desired flavorings pour the mixture in uh llama 213b to make ice cream it actually gives you ingredients here it gives you full instructions and 7B uh it also gives you the ingredients but more in that line and so again this is going to be one of those important things for you to be thinking about with these large language models again one of the most frustrating thing I I I'm about to lose my mind I was just at a Meetup last night man people all just like ai ai ai but they don't really talk about like the actual product the actual technology and it gets to be so frustrating one of the problems that we have right now is again everybody is talking about llms llms are X Y or Z one of the important things you need to be thinking about as a professional isn't just simply the concept of the llm but again what what does the response look like when it comes back how easy is it for you to parse the response when it comes back does it give you all of the information that that you need again one of the big things to be thinking about with decision systems or knowledge systems is that it only needs to give you exactly whatever it is that you need right so if it only gives you a very short answer and that's all you want getting short answers might be the way to go again how long it takes to to process a request right so mistal takes 7 Seconds Pi takes 3 seconds llama 213b takes 17 seconds llama 27b takes 6 seconds again if you are going to be putting this on a system that will have load right if you have aund users a minute on your system or 100 concurrent users which one of these uh which one of these different models is going to work better for your particular situation and so basically the class today I'm going to show you how to write this code so you can you know dump in whatever model you want onto your own system and then you know see see what results you get oh boy so it's my favorite part of the class I always love getting into the architecture it's the real nut meat of these classes I think it's like one it's one of those things that's interesting and Technical but not too technical that you you trip over your words when you're talking about it right so anyways for this particular system what I'm going to be showing you today everything is local and so one of the valuable things about that is you are not going to be making API calls so you don't have to worry about costs you don't have to worry about security issues you don't have to worry about networking issues or anything like that everything that we're going to be doing today is going to be running local on our local machine now it is important to understand whenever you run this is that I am using an M2 uh M2 Max I guess a MacBook Pro that's about one year old in in my time frame from from 2023 uh it does have 32 gigs of RAM it has that M2 Max processor so on and so forth so one of the things that you just have to kind of keep in mind is with these different models depending on the resources on your computer you may or may not be able to run different models so the smaller model should run fine llama 27b should run fine for you Pi Phi should run fine for you some of the other models should run fine for you but if you run into problems where bogging down your computer just might be too slow to do this so if you have a system with 8 gigs of RAM God help you maybe something will work who knows uh but you're going to want at least 16 gigs of RAM most likely uh to do most of what I'm going to be showing you today so anyway so with this uh we are going to have our machine right and so I have a Mac but again you can have Windows or Linux or whatever else it should work fine uh we are going to uh install the olama framework like I showed you before so ama.com you simply go to ama.com click the download and install and install and it's really really simple from then from there uh you're going to need to go to the command line and basically you're going to need what's called to do what's called the pole command and so what the poll command is going to do is that is what will pull down the different models for you so not all the models are not loaded by default because they actually do get to be pretty big so you will decide specifically Which models you want on your machine and so when you're going to be running this make sure you pull the models down first or it will most likely fail out past that then we're going to make need to make sure that ama is running right so AMA is a service that should be running on your machine if AMA is not running the scripts will just simply not work so if you if you try to run your script and it basically completely fails out most likely AMA is not running and so you've just got to have Ama run uh you do the AMA serve command that will start it if it's not already started uh and if you just restart your machine it should auto start by default uh past then uh we're going to be using python uh because we always use python in these classes and there is an olama uh module in Python so we're just going to import the AMA module into Python and that will allow us to be able to interact with python uh with that that's all you're going to really need to do um to do that that first uh demonstration that I showed you where you're just doing the command line thing uh basically all you need is Python and AMA and it'll run fine if you want to do the web app though uh in order to do the web app you're going to need to use bottle uh or whatever you so bottle is the web app framework that's what allow allows us to actually use python as a web server so that's what presents the page to us and then we are also going to be using SQL light uh so SQL light is a relational database and basically what will happen then is we put in our query uh the query then goes to all of the different models we're making the request from when the uh the the the the response comes back we're going to take that query we're going to take what model it is and we're going to take the response and we're going to dump it into the sqlite database so that will be our data store so that when we go to that web page when we go to 127.0.0.1 that's what will then present a nice little report because we'll simply be able to do a select all statement from the table that contains this information pull that in and then basically Loop through and spit that out so this is essentially what the architecture looks for looks like for the system today you need AMA you need AMA to be running you need python we always use Python you need the olama model we're going to be using bottle and we're going to be using sqlite again these can be swapped out as you see fit if you know enough to swap those out by all means swap them out if you don't know enough to swap them out don't swap them out the reason I use these is because they're incredibly easy for for new people to use so now the first thing that we need to do before we get into the code is we need to make sure that ama is installed that ama is running and that we have pulled down the models that we care about so I'm using this Mac and it's probably hard for you to see here but up here in the little taskbar or whatever uh there there's a little llama right so if you see a little cute llama appear and you're using a Mac uh then you know that o llama is running again with Windows or whatever else you should see an icon like that if that is not up there you can use the command o Lama and simply serve and that will start the oama ser Serv um as we can see there is an error because it's already in use uh so we're not going to worry about that today uh from that we can see what um what models are already installed onto our system so if I do olama list this will show me the models and so you can see here I have a llama 2 13B 70b and don't try to do 70b unless you got a really good system I've got 7B 7B chat that's a different version llama to the latest we have mistal we have mixl and we have Pi so this shows us the different models that are on the machine when they were modified or basically when they were installed and the size of these things again depending on your system especially if you bought a Mac you might have to worry about the storage on the Mac look I love my Mac I love a lot of things about the Apple ecosystem the default storage size is not one of them uh and so like with mixl here you can see that takes up to 26 or that takes 20 6 gigs of storage on your machine uh llama 270b that takes 38 gigs of storage and so if you add all of these up you know you're already getting up to about 60 gigs uh of dis space that's taken up simply for these models so that's one of the things just to kind of keep in mind with this again especially if you have a Mac make sure you don't bog down your entire Machine by by downloading all of these different models uh let me just clear the screen if I want to pull down a model so again so so there's a whole bunch of different models if we go over H if we go over tolama we can take a look at this so again we can just go to ama.com that's all that is we can go over here to models and when you take a look they have a metric crap ton of different models and again this is this is one of the reasons why today's project is is pretty important is because it can be very difficult to understand which model you should use uh for whatever system you're going to be putting into a production environment and if you go through you can just see there's a a crap T ton of them like Pi so Pi is small uh Pi 2 a 2.7 billion parameter language model by Microsoft research that demonstrates outstanding reasoning and language understanding uh we have Orca general purpose model ranging from 3 billion to 70 billion suitable for entry-level Hardware dolphin mistol again you've got you've got llama to Chinese again if you're dealing with Chinese clients or whatever tiny dolphin you just have so much a tiny llama 1 billion model and three trillion tokens open chat so on and so forth so anyways basically if you you want to pull down a model all you do is use the command olama once olama is already installed install it it's very simple olama then you simply say pull you want to pull and uh I'll do llama 2 colon 7B so that's one of the models uh then I hit enter it'll pull the Manifest it's already actually as I've showed you before this is actually already installed onto the system so all it was doing was verifying that everything was correct and so now I have llama to on the system so these are the things that you need to be thinking about with o llama make sure that it's installed make sure it's running and then pull the models uh that you're going to be using so now let's take a look at the first code example so this will run just on the command line there's no web browser there's no SQL uh database or anything like that this is the simplest version of what we're going to be creating today uh so with this what we're going to do is we're going to need to import the oama mod module because that's how we do uh we're going to import OS and we're going to import the time module uh then right here what we're going to be doing is we're going to be pulling in the file for info.txt so I've created this info.txt file again let's say for a restaurant uh hours of operation it gives the hours of operation and it gives the menu items uh so you could uh statically create this like I did just simply type out a whole bunch of information and ask questions about it also do realize you can actually create this dynamically so you could have information about your business stored into a database again like sqlite or any other kind of a relational database and then whenever you fired the script you can actually do a query against the database pull all the values out of the database dynamically write them to a text file and then once you have that text file you could then pull all of that information into your python script something along those lines so that's one of the things to consider but for this this class we're just going to have a static file here again with ours of operations and menu items and that type of thing uh so basically current directory equals uh so all of this right here is basically that file is in the exact same folder as this script and so remember F when you read the files and when you write to files by default it will always go back to your root directory so whatever your user root directory is is we don't want to deal with that we actually want it to be in the directory where this script is so that's why we go through this whole mess so we get the current directory as we've done before what the file name is and then we do OS join the current directory with a file name and that gives us our file path to the info.txt with open file path r for r for read as file and then data is going to equal file. read so we're going to we're going to open the file and then we're going to read from the file and that will now our data variable uh then we're going to come down here this is where the query is we'll go into that in a second first thing we're going to do is just to clear off the screen we're going to do os. system clear so os. system clear will delete everything that is on the screen I then create a list of our models so depending on again how many times you want this to run you could have two models in here you could could could have 20 models in here uh the cool part about doing a list like this is it's D Dynamic so basically simply run through however many models you have uh you just plug in the name of the model so llama 2 colon 7B that's for the 7B version llama 2 13B Pi or mistal so what this is going to do is it's going to end up looping we'll do 7B and it'll be 13B and then it'll do PI and they do mistal that's one of the values about having this function up here we can Loop up to the function get the results Loop up to the function get the results so on and so forth uh we're going to come down here we're going to do a while true so what this does is we ask a question we're going to get the responses back and then it will allow us to ask a question again so instead of having to constantly rerun the script a while true Loop we just keep this script going for us a query equals so what question are we going to be asking input function how can I help you uh os. system so when I hit that so I enter how can I the the question that I want it's going to clear the screen it's then going to print out question and the query so that's going to show up on the screen and so for llm in model so we this is the model list so for each llm in the model answer equals ask query llm and data so query is our question llm is the model we want to use and data is that text file we are then going to go up to the ask function right so and here's the ask function so to find ask query llm data start time equals time.time so I want to see how long it takes each one of these different models in order to produce a uh a response and so we're going to have this is going to be the start time query equals answer this user's query so question again this is an F string based on the information in this data and we plug in the data be as succinct as possible just answer the specific question with no other information this right here is where you get to have the fund of prompt engineering something that should be simpler than it is coming up with decent prompts is really a pain in the butt anyways uh response equals. chat model so the model is llm so query goes here data goes here llm goes here for the model message equals roll user content query right so new query equals the old query Plus data that goes into here response is going to equal response at message at content so we don't want everything we just want the content value that comes back and then time elapse equals time. time time so the current time minus the start time that will tell us how many seconds hopefully seconds have passed we are then going to return both the response so the response and we're going to uh return the elapse time and so now we're going to what we're going to do here is we're going to then uh print the model so what model we used the response answer zero so answer response is at the zero index and time is at the one index and that will print out on the screen it's pretty simple so again if I I do this it says you know how can I help you and I can say are you open at 700 p.m. on Friday now it's going to be going through so this is the first one model llama 27b yes we are open at 700 p.m on Friday so if we go over take a look at info.txt we can see on Friday we open from 12 uh yeah from 12:00 p.m. to 80000 p.m. uh so that answer is in fact correct uh so that was 7B 13B response no what I am not open at 700 p.m. on Friday my hours of operation are 12:00 p.m. to 8:00 p.m. on Fridays so it literally says I'm not open at 7 p.m. but I am open at 7 p.m. right again and this is the kind of weird stuff you get back Pi look at this look at what pi gives you I don't understand what pi is doing Pi yes I am open p.m. the hours of operation for this day are 12 to 1200 to 8:00 p.m. additionally we have a variety of menu items including sides such as fries and salad I do have to say can I be honest with you having previously been a business owner with employees I do kind of appreciate Pi for this piie is literally upselling the customer simply from the response it's like yes we're open would you like some fries with that I I got to say from a technical standpoint pie annoys the crap out of me but at having been a previous boss that's written written people's checks there's a little part of me that's kind of It kind of appreciates that it's already trying to upsell people even from a command prompt uh and then that that took 3.5 seconds to come back and then we have mistal response based on the provided data I am not open 7:00 p.m. on Friday hours of operation for that day are listed at 12:00 p.m. to 8:00 p.m. and that took 4 two seconds and so again this is a thing that really shows you in the barest way possible basically how different the responses can be from the exact same query uh from the exact same information given to it just bit depending on the different models how vary the response and also things like how long it actually takes you to get that response so this is the first code sample uh from this we'll go over the web apps now that you know how to create the command line version of this let's create a a nicer web app version so we're going to be using bottle again you can swap out with Dango flask or whatever you want to use uh we're going to do from bottle import route Post run and request function we're going to import SQL light three so we have that database back end import OS oama and time uh then we're going to go here and this is the whole thing basically to be able to import the info.txt file again with the uh the hours of operation and the items that kind of menu thing uh so you know the current directory the file name we're going to join that together to a file path we're going to open the file path as file we're then going to read all of that information into the data variable so the the value for the data variable will now be whatever is in the info.txt uh we then have a class for database so this is kind of a copy and paste I've been liking this database class I've been using for a while uh and basically this just simply allows me to to very easily uh you know redo whatever steps I need need to do in order to interact with a database uh so it's class of database uh we're going to define the path function and so this is the path to the database so we're going to have a database called query I want this to be in the current directory and so query with current directory we're going to join for the file path and we're going to return the file path whenever we need the file path to the database that we're going to be connecting to we're going to have a function for create the database so the file path equals at database. path that function I just showed you we have the connection for SQL light as I've shown you in different classes and then we have a create table SQL St statement so create table if not exists result so we're going to create a table called result if it doesn't already exist we're going to have an ID which is going to be our primary key we're going to have the model what llm are you using that's going to be text the query the question ask text response as text and then time I just have that as text to make easier so this is the elapse time how long it took uh cursor. execute the SQL statement of create table and then you have to commit always in sqlite make sure you commit or it hasn't happened if you don't commit it didn't happen if you like it then you better put a commit on it as they say and then we're going to close the connection uh here we have the statement for select so we're going to select information out of the database select all uh from result order by ID descending this will pull all the results out we'll deal with that later and then here's where we have the insert so insert into uh the result table model query response so this is where we're going to be inserting using the uh the SQL function here's where we have the ask uh function we're going to deal with that in a second and then here is where we have the index function we'll also deal with that in a second so these are decorators here the at Route decorator for root so when you go to 127.0.0.1 or whatever your IP address is or whatever your domain name is it'll automatically drop you into the index function and for a post so we use the post decorator for this and so when we're going to post a value we need that decorator so with that little question at the top when we ask a question we need to use post then down here we have the database class. dbcreate so in the database class we're going to run the DB create function and so if the table does not exist we are going to create that table run host at 0.0.0.0 so any IP address associated with this computer will be able to provide uh the website to us again we're just going to be going to 127.0.0.1 Port 80 now if you're using a different operating system so I'm using a Mac OS I can use port 80 without any problems so Port 80 is like the default port for web browsing if you're using a different type of computer operating system with different security you might have to use port 8080 so that's just let me to keep in mind so you would do 127.0.0.1 colon 888080 to get to that 8080 Port if you have 80 it's the default so you can just use 127.0.0.1 debug equals true so again if there's any problems I do want to see what those problems are we go here to the index so query equals request. form forms get the query value so down here we're going to create a form so form action is post question text area uh column 50 Row one name query that's what we're going to be pulling from um right and so basically that is where we're going to get this value uh model so again we can we can put in however many models we want so it'll dynamically run llama 27b 13B pi and mistal if query does not equal none so that's one of the big things with uh uh with these kind of queries like if you have a if you have a blank request you don't necessarily want that to run it's going to put crap into your database and that type of thing so basically if essentially what this means is if there is a query then run the query if not don't so four version in model and so basically this is for the llm so each one of these in the model right response equals ask so the they response function query version data so we're going to go up here and we're going to look at this and so for here I actually have this is query llm data it's the the importance is the position not necessarily the name so this is the question that we're asking this is the model that we're using and this is the data file uh start time equals time.time again the query and the data as we did before uh AMA chat the model equals llm we're going to get the response back as we did before message content and we're going to get the elapse time and then we're going to return what the response is uh from the elapse time and so this is going to Loop through all these different models then we're going to do database. DB insert so the version the llm the query response zero is the response and response one is the lapse time for the response that will get dumped into the database and then print the response out on the command line on the screen just so we can see what's going on record set equals database. DB select so B basically we're going to go to the database we're going to select all of the information out of the database so if we come up here that DB select again select all from result order by ID that's what we're going to be doing we're going to pull all of those values in for record set we're then going to create the form so this is the form as I showed you before we're then going to be building we're going to be building the for the the the results so previous equals nothing set new equals nothing set old equals nothing for record in record set set new equals record 2 so basically this is the query so one of the things that we're going to be doing is we want to uh group all of the responses together under one query that's what we're going to be doing here so if set new does not equal set old previous equals previous we're going to create a div a big black div around everything and then set old equals set new then what we're going to do is we're going to print we're going to add all the previous values then we're going to print everything out so model equals record one uh response equals record three uh time equals record 4 and then we're going to print out the form and the previous and we're going to return the entire page now let me just kind of show you something real quick so let me run this so I'm going to run this hopefully it doesn't fail and I'm going to go to 127.0.0.1 as we did before and so basically this is where we have the groupings so here is the query are you open on Sunday model mistal response based gives me this information model Pi response model llama 13B response llama uh 7B response with the times and all that and you can see how they're all grouped they've got this nice uh black div around everything so I can see okay are you open on Sunday these are the answers I got for that how much is a burger and fries these are the answers I got for that what day are you open here are the answers and it's all it's all within like that same black box that's why I'm doing this mess here with the set new uh equals record 2 so what the query is if set new does not equal set old basically what we're doing here is if everything has the same query it's going to be in the same div box and then as soon as a new query comes up it will create a new div box for for us to create right so this is one div box and then this is the next div box and this is the next div box and that's basically what we're doing here with this whole set new set old and set new record basically trying to compartmentalize those so they're easier to take a look at again if we take a we run this right so it's running right now so I can say um I don't know uh do you have uh chicken all right then I can hit run now it's going to go through again it's going to take a couple of seconds because it's actually going it has to test to run off of all of the different models that you have so it'll take uh I don't know what it'll take 30 30 or 40 seconds I don't know do a little dance here oh right there we go we got the response so do you have chicken mistal yes based on the given data chicken is available during the hours uh based on the information uh you can have chicken based on the hours uh no we do not have chicken based on the data provided the answer to the query is yes we have chicken uh so three yeses a couple of NOS one of the things uh though to show you here is you will notice that like here like the text goes off the page one of the things that I did with this is again one of the things you have to be thinking about is how you're going to parse the data so if you're using this just as a as a testing routine just to see what the results are going to be you may want to know how the the the data is actually formatted and so that's why I have here I'm using these pre- tags so record three is the response and I'm putting the response within the pre-tx what pre- tags is is however the text is formatted in asky text that's how it should show up on the HTML page and so that's why I did that here if you want that to look prettier uh you can remove these pre- tags and it will look a lot nicer I left those there basically so you can see how the text is actually formatted so you can f try to figure out again for your particular purposes how you'd want to be dealing with that response if you're using this in a different application so the final project is basically giving you the ability to ask open-ended questions of olama so uh so originally I started this project because I thought it was going to be really cool you're going to be able to ask questions you know about a document and it was going to give you these great results well it did give us results it's not necessarily great results and so I decided to add this final project here to allow you to ask more open-ended questions it does not have to revolve around that info.txt file if you want a story about a snail and a sailboat this web application will give it to you uh so for this we're going to import bottle as before sqlite OS AMA and time we're going to have the database class and this is all going to look basically the same the only difference here is our database is going to be question. DB so before it was query. DB this is question. DB so both of these scripts are in the exact same folder so if you basically if you have the name here the same in both scripts you're going to run into a bit of a disaster possibly so I just change the question. DB there everything else is basically the same we have the ask function so but with the ask function we're simply going to add the query and the llm there's no other information there's no data answer this user's question what the query is be as succinct as possible just answer the specific question we're going to then put in the query we're then going to get back the response which is the content and then we're going to get back the elapse time we go down to index and basically again query equals the value coming in from the form these are the different models if you know as long as query isn't blank uh we're going to ask you know query llm then database insert llm query response zero and response one record set we're going to select from the database here is our form as we had before previous set new set old record and record set and basically this is going to print everything out on the screen so if I run this right I can go over here I can go to 127.0.0.1 and you can see again this is a historical context so we can see the other questions I've asked how do you make ice cream we get different responses how do you jump how do you make a ham sandwich right how do you make an Italian sub so I can say uh what is a sausage made of they say you never you never want to know what's in the sausage well if you tell me I don't want to know what's in the sausage then I really want to know what's in the sausage okay what is a sausage made of mistl a sausage is typically made of ground meat bound together in a case and that may contain various additives Blas Blas oops Blas Blas Blas and it goes to the side because again we still have it in that pre sausage is typically made from llama 2 sausage is made of meat and Seasonings so it gives us this different information took us 5 Seconds here 3 seconds here 7 Seconds here and 2 seconds here and so again this allows us to ask more of those kind of cool open-ended questions um which you know feels a little bit more fulfilling than the rather poor answers you get when you ask a question about a specific file so there you go there's a class for how you can create these projects that allow you to ask one query and then compare the responses from a number of different llms what's great about this is It's Dynamic so those models are basically within a list so you can add more models to the list you can remove models from the list and basically just wait for it to process again do realize some of the models depending on the the the power of your computer may take a little while so I'm not sure you want to put 10 models in that list but as they say you do you uh you're going to download and install ol llama you're going to pull the models that you need and you're going to make sure that o is actually running on your system before you do any of this uh the first example that I showed you was simply command line so we use the input function we basically take the query we take the information from the data file and then we ask questions from these different llms uh the last two again that showed you how to create a web app and so basically again with the first one we have that info.txt information where asking questions about that and getting the responses that we get uh and that shows up on a nice little web form the final one is basically the exact same thing only we don't worry about the info.txt file to allow you to answer more again open-ended questions and the real power for this is for you to get a better idea of the quality of results that you're going to be getting back and for you to be thinking about how you're going to be parsing the information again with these llms one of the things that I hate is people talking about AI Eli what do you think about AI I get confused and then I realize people are talking about llms these are not these are not AI trigger Wen these aren't AI right they're large language models they're great they're wonderful they've got a lot of uses this is not artificial intelligence but why it's important for me to to to delineate that is because if you think it's AI you're going to expect the system to work in a way that is not going to work right if you think about it as a parsing system if you think about it as a natural language processing system right you'll interact with this technology in a certain way if you think it's intelligent right you're going to you're going to try to interact with it entirely differently and so what I want to show you here with these results is basically to give you that idea of these llms do act differently the results that the llms give you can be all over the board how you do the prompt engineering right that that will that will Define like the types of results that you're going to be getting back and so again this is much more oh Airy fairy configuration right if you're used to like Linux configurations you're used to Linux configurations with Linux configuration this equals that this equals that this equals that uncomment this if you want this to work comment this out if you don't want this to work right that's how we're used to using configuration files what I want you to be thinking about these llms is it's like dude man do man it's like it's like Airy fairy configurations where you kind of sort of tell the computer like you know the kind of results you'd really appreciate hey computer would you mind giving me the results that I want cool dude right like that's a different way of administering a system that's a different way of interacting with a system and so understanding that how you're interacting with the system and what you have to give the system and the results that you're getting back is this kind of this new paradigm that's what I want you to take away uh from this class to be thinking about again how are you going to be implementing this functionality into your systems when we think about AI with AI so many people focus on this idea that AI is the product right well AI will solve everything that's great I've got an llm llms won't solve much l are a piece of functionality so again how do you put this piece of functionality into a much larger platform that's the kind of thing I want you folks thinking about uh again with all this code and everything I do not believe anything that I've done it values intellectual property rights if for some reason it does uh everything is going on to GitHub and the other places that I uh share the this type of code and you are free to use it you're free to copy and paste it give me attribution don't give me attribution I really don't care again it's 100 lines of code doesn't matter to me uh but you can use all of this uh to to play with it uh as you see fit uh so anyways that's about all I've got for today as always I enjoy teaching this class and look forward to see you in the next one

Channel: Eli the Computer Guy

Views: 2,200

Keywords: Eli, the, Computer, Guy, Repair, Networking, Tech, IT, Startup, Arduino, iot

Id: OohNQTLBIt0

Length: 46min 41sec (2801 seconds)

Published: Thu Apr 04 2024