GPT Prompt Strategy: Brainstorm, Search, Hypothesize, and Refine - THIS is the FUTURE!!

one question that I get a lot is Dave you have made literally hundreds of videos and you know you've written a few books but the books are out of date and I want to use a chatbot to talk with you or you know otherwise use all all your material and so if you saw this video yesterday Matt wolf produced this video where he basically made a Mr Beast chatbot by downloading I think it was like 11 or 12 uh transcripts which is a really phenomenal idea but also I've been talking about mgpt and other ways of uh basically organizing arbitrarily large uh data sets so I went to one of my old repos where I had my YouTube chapter generator so uh if you've seen some of my older videos I I generated this thing where it would you know just download and and download the transcript from a video and make a uh set of YouTube chapters this is done automatically there's other services that do this now even YouTube will sometimes generate chapters for you so this project is kind of you know whatever um but if you want to see how it's done this is how it's done and this was back with gpt3 so with gp4 it's probably infinitely easier but in the service of doing this I wrote a script that literally it's just called download all transcripts and all it does is it goes to my channel and you can you can change it so that it will go to any channel you just change the channel ID here um and it will download literally every transcript um so here you can see that I have 270 plus transcripts from all of my videos this is obviously a potentially highly valuable uh Corpus of training data now one thing is that I think people are coming to terms with the fact that fine tuning is not useful for uh knowledge representation we're now in the age of retrieval augmented generation and basically mgpt is just a retrieval system it's not a particularly sophisticated memory system and one thing that I want to address is like yes I can be very highly critical of scientific papers I'm notoriously difficult to impress I'm sorry if this frustrates some people um another thing that I wanted to address is that some members of the audience are more scientifically minded and some members of the audience are more engineering and coding minded where they just want practical results so sometimes I'll be talking about things from a more scientific lens and sometimes I'll be talking from a more pragmatic lens so my criticism of mgpt is is more from the pragmatic lens um and yes I'm aware that there are AB testing and all sorts of more scientifically rigorous things that I can do and I apologize again if that frustrates you anyways moving on so the uh basically we've got this huge Corpus of valuable data here so if you want to understand how my mind works and all that stuff you can download these transcripts it's all free um and do whatever you want with it um just don't you know do anything creepy uh not that anyone would I don't think anyways so now yesterday I created a video about uh latent space activation um so a few recent Concepts that I've been working on I made another video talking about sparse priming representations so if you haven't seen that a sparse priming representation is a way of activating certain aspects of a large language model without having the full bulk of it and so this this banks on the fact that large language models are associative just like human memory is however large language models have read literally hundreds of lifetimes worth of text so you only need to remind them of a few key facts or features or assertions about stuff even things that are outside of its training uh distribution and it can then adopt and use new Concepts you can teach it new terms you can teach it new events all sorts of stuff using sparse priming representations and you can compress relatively large things I actually took um my systems thinking book that I'm working on and gave it to Claude And I asked Claude to create a sparse priming representation of my systems thinking book and it was able to compress it down to about a page and then unpack most of it now this is basically a form of semantic compression or you might say uh from a mathematical perspective it's like a semantic PCA principal uh component analysis um but anyways the idea here is that it is a form of lossy verbal compression um but because llms have the ability to confabulate which means that they can infer or impute uh or kind of guess um sparse priming representations are super useful uh and then yesterday I talked about lat and space Activation so lat and space activation is kind of the broader concept of what's going on behind the scenes inside of the model and then uh also yesterday I mentioned uh something that I was working on which is what I call the Basher Loop or bshr so bshr is the point of today's video and this stands for brainstorm search hypothesize and refine so I'm married a librarian um so this basically constitutes dinner conversation for us but in the field of information and uh librar science uh people have information needs this is why you go to Google this is why you go to Wikipedia or even you search Reddit or YouTube now there are tools out there like perplexity like the chatbot that Matt wolf built um but there's one thing a few things missing from all of these tools and that is information literacy and information for strategies so I am here to teach you about that and we are also here to um introduce the Basher Loop uh repo which is brand new um but basically the ultimate goal here is to create a Basher Loop uh Search tool chatbot that will allow you to search literally all of my videos um so there's a lot of steps to do we're not going to finish it today but I wanted to take this time to one introduce the repo introduce the concepts and show you the work that I've already done so so uh when you first get to the repo it's just Dave shap bshr Loop or Basher loop as I as I say it out loud um so the Basher Loop welcome to the uh you know uh the repo uh it allows you to search for AR arbitrarily large information domains um this is what it means the overview so the overview of the loop first brainstorm it accepts a query from the user it brainstorms a list of of of questions which I showed you yesterday and I'll show you the prompt that I used to do it uh today in just a little bit then you use uh search functionality whether it's you know apis or vector database or whatever the queries that you start with are then basically just piped to whatever your information tap is or if you have multiple information TS you can use that then hypothesize so you take the information that is retrieved from your data source whether it is wikkipedia or a vector database or just a general internet search um and then you have the llm read all of it and form a hypothesis based on that information this hypothesis is an answer to the potential question it says okay based on everything that you're seeing this is the answer and if you want to try this this is basically what uh perplexity does but it does it as a single step where it just searches and summarizes it doesn't really form a hypothesis uh nor does it actually write good queries which is my chief complaint and why I canceled my perplexity subscription um and then finally refine so this is refine or recurse or whatever but the idea is you don't just do one information search you like think about the last time that you had a really challenging problem what do you do you do a Google search you open a bunch of tabs you read a few things you read a few more things and then you learn about the space and then you go write a better information query you write a better search query so this is the loop this is why it is a loop is because finding and using the correct information is not not a singl step process when you go to the library whether it's the University library or the local city library you what do you do you go you talk to a librar and they ask you a bunch of questions about what you're trying to achieve and then you uh you know you go look at a book and you take some notes and it's like okay cool I liked this book but now that you're informed you go from a naive query to an informed query real quick Dave here with a request so You' probably noticed that I'm cranking out videos and I'm really getting my content dialed into what is adding the most value to you guys and so I'm really doubling down on subscriber growth and just really serving the community now the reason that I've been able to increase the quality of my videos and the frequency is because I've stepped away from Consulting more than half my income came from Consulting so I'm really kind of hurting for cash right now however I have been largely supported by a Grassroots movement so my request is Hop on over to my patreon page and sign up the lowest tier is $5 a month and that gets you access to Discord which I'll show you in just a moment but I also have a premium Discord tier which gives you access to some private channels some live streams and QA sessions and that sort of thing so if you could help me out and you'll get access to this community uh and it's a great Community we've got almost 600 people right now and we talk about all kinds of stuff not just AI we talk about the future of humanity we have artistic uh interest we talk about economics we talk about existential coping post- labor economics all of the above and then finally I also have some premium channels for the high tier subscribers so thanks for watching and uh yeah thank you you go from a naive query to an informed query so in this is where I'll go down and I'm not going to read all of these to you verbatim but I'm just pointing out that this information is here in the repost so that you can get familiar because I'm assuming that most of you are not familiar with librarian information science so here's some critical things that I can teach you so that you can make informed queries so first of all what we're talking about is information foraging information foraging is when you go into an unknown or incomplete information domain where you have an information need but you don't necessarily even know what questions to ask so you have to go in search of information by basically starting out guessing information literacy is one of the primary things that I complain about both with uh these tools and humans in general um so information literacy is basically understanding uh how to engage with sources of information um in order order to find correct information or validate the information that you have and this is something that I talked about yesterday and we'll also show you in just a moment where the way that I build information literacy into this tool is by telling the llm during the brainstorm phase to generate counterfactual questions basically it's trying to prove itself wrong and it's trying to find all sides of Any Given information problem which will result in a better uh recall um with recall is a very specific term from library science where basically are you getting all of the information that is relevant that is Salient to your query um so next up is satisficing so satisficing is a portmanto of um sufficing and satisfying so when you satisfy an information query the idea is that you have found you have gotten to the level of good enough it's not perfect it's not complete but you have the information that you imminently need and you're ready to move on so one example that you can use for satisficing is when you're looking for a restaurant to go to there are that you might have in your mind the perfect restaurant but the perfect restaurant might not be available it might be too far uh it might just not even exist and so what you do is you look for something that is good enough and so the idea is that within information foraging within search you need to satisfice from time to time which means that the information the exact information that you need might not be available a definitive answer to your question might not even be possible if you watch the video that I made yesterday one of the things that I asked was what is the exact Coastline of Britain and the answer that it gave me was well that's really hard to determine but here's how you could go about it and so that the answer to the information query that I had the information need was not a definitive answer but rather uh this is how you approach that information problem so in that case the what what it needed to do was outside of the scope of information foraging um information needs so I've referred to this before but basically all problems are a form of information need so an information need is I have some sort of problem solution query whatever and I need information in order to be able to tackle this whether you're a CEO making making strategic decisions about AI whether you're uh you know a broke college student whose car is making funny noises whether you're in a relationship and you want to know like how do I navigate this fight that we just just had literally everything you do in life isn't has information needs um and this is why libraries are super underrated as well as Librarians um but of course I am biased because I married a librarian now um this is a term that I invent at least as far as I know that I invented so there's what's called a naive query so a naive query is something that everyone does where you go to Google and you you don't even know the right questions to ask so you just kind of guess you just launch your best guess out into the ether and then Google gives you back some stuff that it you know Google has a lot of really good algorithms so it can kind of infer what you're actually looking for um and and the reason that it can do this is because lots of other people before you have have launched those naive queries and Google Tracks where they ultimately end up and it says okay based on what you're looking for based on what you put in I'm predicting that this is actually the page that you're looking for so you launch a naive query because you don't know what you don't know once you know a little bit more then you can write an informed query because like let's say for instance you start with you know the one of the questions that I asked yesterday which was uh who were the important Senators during the uh peak of Roman power right you might not know that there is a thing called Pax Romana so part of the part of what you would find during that query is you know the Pax Romana or as someone pointed out in the comments Rome was a kingdom first then a republic then an Empire so depends on how you define like you know Empire versus Senate you know where how do you define the peak of Roman power and so you can then integrate all of those bits of information to ask better questions like uh you know was Rome more powerful as a republic was Rome more powerful as an Empire now of course large language models already know most of these things um which is why the Basher Loop can actually work in a vacuum for things particularly historical information that have been discussed to death but but as other people pointed out in the comments there's lots of data out there that is just not in the training distribution like your personal data like my YouTube transcripts like corporate data um so once you have your naive query you you you know a little bit more then you can dial into an information sent which is like okay here are some Clues to what I'm actually looking for and then you can write an informed query and then I already mentioned uh precision versus recall I'm not going to read that one to you and finally there's some use cases um so I've kind of alluded to all these you could apply the Basher Loop to the business data Lake to an internet search to a University Library system or a personal archive in the case of my transcripts so I've got all the transcripts saved here you can go through them all there's obviously quite a few uh yeah so there's that and then finally or not finally but next is contributing so I've got uh there's there's three ways that you can contribute or participate and the order is listed so please make sure you check the contributing uh file before jumping in first is join the discussions um this is where everyone should start um don't just jump in immediately submitting issues and PLL requests um if they're not if they don't follow the guidelines here they'll be closed and rejected um but yeah so check out the contributing markdown file um if you're not familiar with this the discussions tab is up here so you can jump in there's no discussions yet cuz I just created the repo there's issues here uh again no issues have been created and there's no PLL uh there's no poll requests yet either all right so then I've created demo 01 so this one will be um before we get all the way into uh you know building our own local repository our own local database on my YouTube transcripts um first we're just going to use an existing information Source specifically Wikipedia so I've got the pro the the project defined here um I showed a very naive uh like initial test yesterday I've got the code copied here it doesn't work yet so don't you know don't lower your expectations we are still very early in the process but what I did want to show you was that demo1 is here it's got its own read me and I also the the work that I did do is that I already wrote the system prompts so let me let's go through these one at a time so this is the system prompt for for 01 the brainstorming one now what I had yesterday was just this bit so you're a search query generator yada yada basically it generates a list of questions um in a Json object that can be used to do Google searches or Wikipedia searches or whatever uh and so this is the first part of the process that will be used to generate uh the the brainstorm step of the Basher Loop so brainstorm a list of questions um but then what I added was refine queries so this in this integrates that idea of whatever information you give the chat GPT API um if you don't give it any previous queries it'll be like okay whatever but if you do it will integrate it if these materials are present you are to generate informed queries more specific search queries that aim to zero in on the correct information domain Do Not Duplicate previously asked questions use the notes and other information to write better questions basically so this one prompt will be will be part of the Basher Loop so whether it's the first time you're running it or subsequent times it will generate recur uh recursively better uh search queries so that's the first bit um so I wrote that and then going back to uh system O2 so the second part is hypothesize obviously the the search part that's to be handled by an API or chroma DB or whatever so you don't need a system prompt for that because uh basically the that that's that's an algorithmic thing that is just a script um so the next step that is handled by the llm is the hypothesis generator so like like the like the brainstorming the query generator you're an information needs hypothesis generator you'll be given a main information need or user query as well as a variety of materials such as search results previous hypotheses and notes whatever information you receive your output should be a revised refined or an improved hypothesis in this case the hypothesis is a comprehensive answer to the user question etc etc etc basically I'm just giving in instructions that whatever I give you generate a hypothesis that answers the user query as comprehensively as possible I tell it to worry about citations because we'll hold that out as metadata um and so that's some that's a problem that we need to work out um okay and then finally the satisficing check so I'm guessing that this is probably the first time that most of you have heard about satisficing um so here's the instruction that I wrote In order to produce this and I'll explain why um but the the very short answer is I'm going to give you a bunch of information and then you judge you being the llm uh GPT you judge whether or not the information need has been satisfied and then I tell it like you're to make this judgment by virtue of several factors amount and quality of searches specific specificity and comprehensiveness of the hypothesis and the notes about the information domain and foraging if present so basically we can include notes from the search part to say hey we've exhausted all the search like we're not finding anything else and you can see that we've run you know 30 queries and this is the best information we've got um so you know several things to keep in mind the user's information need might not be answerable so in the in the case of you know asking what is the exact Coastline of Britain um you might want to see a specific number um or it might be partially answerable which is like you know it's generally estimated to be about I think it was like 12,000 kilm or something like that um or given the information uh available for the problem basically some things are outside of the scope of information needs for instance if you need to uh to actually do something in the physical world right so like you might go to the library and say like I need to file my taxes and it's like well I'm a librarian I can't file your taxes but I can give you all the information that you will need to be able to file your taxes so tool use and acting upon the world is outside of the scope of the Basher Loop but the idea is if the information is available we will give you all the information that we can get we will basically act like a librarian and be your information coner to get you the information that you need that you can then go take and you know give to another AI or a chatbot or a robot or your best friend or do it yourself or whatever anyways finally the output format is a Json object so there's two things that I put in the Json object one is feedback which is basically it unpacking what its perception is and the reason that I do this first is because as language models kind of talk through things it'll be able to triangulate and make a judgment or a decision a little bit better because as you might have seen on a I explains videos if you have a language model just output an answer like if if you give it a whole list of stuff and then just ask it to say true or false it's going to be kind of random but as I demonstrated in uh yesterday's video with sequential reasoning um large language models because you get more latent space activation with sequential activation this is also a form of sequential activation wherein it talks through the evidence that you've presented and then it will make a better judgment as to whether or not the user uh information need has been satisfied that's the tldr of all of this and I think that brings us to a the conclusion for today so that's where we're at this is where we're starting um I'm probably going to be working on this for a little bit this project isn't going to live too terribly long um it's probably going to be a three or four part series we'll see um but yeah so thanks for tuning in I hope you got a lot out of this like subscribe etc etc you know the drill [Music] cheers

Channel: David Shapiro

Views: 23,773

Keywords: ai, artificial intelligence, python, agi, gpt3, gpt 3, gpt-3, artificial cognition, psychology, philosophy, neuroscience, cognitive neuroscience, futurism, humanity, ethics, alignment, control problem

Id: sJLB2EIQUvQ

Length: 23min 6sec (1386 seconds)

Published: Wed Oct 25 2023