Using ChatGPT As a Copilot For Your Mind

when I have an article idea I'll often start with just this like really messy document full of quotes and sentences and little like things that might go into it and then I'll be like I don't even know where to start with this this is crazy and then I will just be like can you put this into an outline and I'll just paste the entire document into chat gvt and it'll often find an outline and like the the outlines it comes up with are like really basic but sometimes I think what it's one of the things it's really good at is like pointing out the obvious solution that you missed because you're too like close to the problem hello and welcome to the cognitive Revolution where we interview Visionary researchers entrepreneurs and Builders working on the frontier of artificial intelligence each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work life and Society in the coming years I'm Nathan lens joined by my co-host Eric torberg hello and welcome back to the cognitive re Revolution today we're sharing an episode of the new podcast how do you use chat GPT how do you use chat GPT is hosted by Dan shipper founder and CEO of every a daily newsletter that promises the best business writing on the internet in just his first few episodes he's had guests on the show including sahill lindia Nat Eliason lonus Lee and today Yours Truly this conversation is both extremely practical and a real exchange of ideas coming into it I had used chat GPT mostly for unfamiliar tasks where I really needed help orienting myself and getting started and of course I've got great value from a wide range of different use cases but to be honest I hadn't found chat GPT super helpful for my own writing process so I was really interested to learn more about the methods that Dan has developed to use chat GPT as a thought partner and a writing assistant learning from him inspired me to do more of this for myself toward the end of the episode Dan asked asks me what I am most excited about next and I mentioned the new Mamba architecture and state space models more generally which I honestly can't stop thinking and talking about we'll have a big episode on this coming very soon and I'm glad to report that I did use some of Dan's recommendations to help develop the strategy the devices and the overall structure for that episode in a way that I did find legitimately very helpful one note before we get started there are a few points in this episode where we each shared our screen to show off content visually and while I think you will be fine with just the audio version if you want to see the visuals you can check out the YouTube version of this episode of course there's always lots more to learn so if you like this sort of content I encourage you to check out how do you use chat GPT with Dan shipper welcome to the show thank you Dan great to be here I'm excited for this I'm excited too for people who don't know you are the founder of weark you are the host of the excellent podcast cognitive Revolution and you are a gp4 red teamer so you were responsible or one of the people on a team of people who were trying to figure out how to make gb4 do bad stuff before it was released which you had a really interesting tweet Thread about I don't know I think a couple week ago or two weeks ago something like that so very excited to have you I think you you'll have a lot of insights that I'm excited to share with everyone I think one of the things in thinking about your work that stands out and thinking about cognitive revolution in particular the podcast that you run is I think you have this idea that one of the values of AI is in helping us to offload cognitive work so just like in the way that machines we in the industrial revolution we offloaded like manual physical labor AI will augment or offload a lot of cognitive labor from humans and I wanted you to just talk about that um tell me more about what that means and then tell me is that a good thing and where is it a good thing well that's a big question I would say I talk about AI doing work and helping us in a couple of different modes for starters we will probably spend most of our time today in what I call co-pilot mode which is the chat GPT experience of yeah you have you are as a human going through your life and going through your work and encountering situations where especially as you get used to it you realize oh AI can help me here so you make a conscious decision in real time to switch over to interacting with AI for a second or a minute or whatever to get the help that you need and then you proceed but you are the agent right in that situation going around and you know pursuing your goals in contrast the other mode that I think is also really interesting is delegation mode and that is where you are truly offloading a task and I always say the goal of Delegation mode is to get the output to the point where it is consistent enough that you don't have to review every single output and if you can get there then you can start to really shift work to AI in a way that you no longer have to do it and that can that could be useful in different kinds of ways right the the co-pilot mode is about helping you be better that's your classic symbiosis or intelligence augmentation and then the delegation mode is more like we can save a ton of time and money on things that used to be a pain in our butts or we can scale things that are not currently scalable and there's a lot of that in the world right I think almost everybody has things where they would say you know if you just ask the question is there stuff that you could be doing that would be really valuable to have done but you just don't have time to do it there's a lot of that that can be quite transformative in the middle and and what's kind of missing right now still is between co-pilot mode where you're getting this kind of realtime help and deciding how to work get into whatever you're doing and delegation on the other end in between is ad hoc delegation yeah where it's I'm going along but I wanted ideally I would like to delegate more and bigger subtasks to the AI on the Fly and that's where we're not quite there yet the agents broadly can't do much in the way of a significant task so it's you're still shoehorned into one of two scenarios where you're engaging with it in real time and getting help or you're going through the process of doing a setup and doing a validation setting up a workflow to where you can truly delegate and it's that in between that I think is probably that Gap gets closed over the next year as agents quote unquote begin to work and then we can start to delegate bigger chunks of work on the Fly the next question was is it good I don't know if I have a great answer to that I think it's largely good it's I think it's as it's good as long as it's a it's good as long as humans stay in control of the overall Dynamic and I'm definitely defitely one who considers everything to be in play for the future both on the positive side I don't think it's crazy to think of a post scarcity world and on the negative side to quote Sam mman I wouldn't rule out lights out for all of us I think we are definitely playing with a force here that has the potential to be totally transformative in good and bad and probably a combination of ways I'm thrilled by how much more productive I can be and that's some of the stuff that we'll get into in more detail I am thrilled by the prospect of having infinite access to expertise and especially for people who have far less means than I do to have that kind of access to expertise I am a pretty privileged person who can go to the doctor without really thinking twice about taking the time off from work or what that's going to cost me or whatever obviously a lot of people don't have that luxury I think there is a real way in which AI can cover a lot of those gaps not fully yet but already significantly and obviously more and more over time I think that kind of stuff is going to be potentially disruptive it may be the source of a lot of political debates and challenges but anyway yeah there's there's so much upside but I think there is very real risk and it's very easy to hold those two perspectives at the same time to be just thrilled by the the capability but also to be always always keeping in mind a sort of healthy fear I love that I think that's such a rare perspective and as humans we just tend to collapse on one either it's horrible or it's great and then we have these camps and I think like obviously the wise perspective is there's going to be some really amazing stuff about this and there are dangers like when technology changes society and change it'll change our brains like we will adapt to this and in the same way that it is adapting to us that will change things and we'll need to like deal with the dangers that it presents I think that's a very wise perspective and I asked that question is it a good thing that cognitive work will be offloaded because I think that there's good and bad but one of the things that I feel is the fear scenario is like is quite dominant for a lot of people and and I think the people who are like anti- fear or presenting a hopeful view are a little but they're a little bit too like rose-colored glasses and I think finding real ways and real use cases for how offloading some of this cognitive work actually helps people is just like a really important part of creating a world where AI is a Force for good or Force for creativity rather than a world where it just replaces people or it creates dangers or there there's I don't know all the bad scenarios and one of the things that I've felt going back to your kind of co-pilot mode versus delegation mode point one of the things that I felt is that AI reveals to me how much drudgery there is even in highly valuable highly creative knowledge work and that we sort of like lie to ourselves about the amount of drudgery because that work is so romantic compared to I don't know I don't know working in a factory maybe or just any other kind of job and it's it's easy to to look at a lawyer and be like a lawyer's job is full of drudgery or whatever but I write I run a business I have a YouTube show now I have a podcast there's a lot of stuff that's like just pure drudgery of that and I find it really interesting because using chat gbt using AI tools more broadly it has made me aware of how many repetitive or like just overall kind of brain dead things I have to do just to write something smart on the internet um on every and once it's visible I use AI for it and then I don't have to think about it as much anymore and I I think that's a really cool thing totally for me coding comes to mind most there when you talk about the drudgery of high value and again pretty privileged work to be doing but I'm not a full-time coder have been for a couple short stretches in life but more often I've been somebody who's dipped in and out of it and it is a real pain in the butt to have to Google everything obviously different people have different strengths and weaknesses I do not remember syntax super well sometimes if it's been a while I'm like wait a second am I is this am I remembering JavaScript or am I remembering python like what exactly is going on here yeah and so to be able to just have the thing type out even relatively simple stuff for me is a multiple X speed up often in terms of productivity improvements often an improvement in just strict quality too compared to what I would have done on my own yeah and makes it so much easier to get into the mode in the first place there's this kind of I wouldn't even call this drudgery but it's gearing up just like somehow getting my people talk about in bird watching getting your eyes on really focusing on what are you seeing and trying to like get that detector right there's like a similar thing at least for me in terms of getting into code mode and it also just streamlines that tremendously because next thing you know it's writing the code and I'm reading the code reading the code is a lot easier than writing the code so I do find yeah just uh tremendous U satisfaction you know pleasure in just seeing this stuff like outputed for me at superhuman Pace better better than me quality maybe not superum quality but super Nathan quality it's awesome what you're making me think about is because I I think in large part not all of it but in large part what the the current class especially of text models are doing is different forms of summarizing and how like how much summarizing is involved in Creative work in programming in writing in decision- making a lot of it is just summarizing like in programming you're summarizing what you find on Google you have to decide what to summarize and you have to summarize it in the right exact way for like your specific use case but that's a lot of times what you're doing same thing for writing like a lot of the stuff in my pieces are summaries of books that I've read or conversations I've had or ideas that I've found somewhere else that I'm like stringing together in a sort of unique way and obviously I still have to do the management overall management task of deciding which summaries to put in which order and like how they work or whatever but like a lot of it is summary and I think that's a way that using these tools you start to see the world a little bit differently and you're like oh yeah there's a whole there's a whole class of things I'm doing that are summaries that I don't have to do anymore and I really think that's cool yeah I should be one of the areas where I have not adopted AI as much as I probably should have is in repurposing content making more of what I do with the podcast because I've put out a lot of episodes there's a lot of stuff there and we do use AI in our workflows to for example create the timestamp outline right of the different discussion topics at different times throughout the show that's the most classic summarization where I'm not looking for a lot of color commentary it's literally just what was the topic at each time get it right so we've got some stuff like that we go to pretty regularly but I have not done as much as I probably could or should maybe this will be a New Year's resolution to um bring that to all the different platforms and it is I think this is actually an interesting it's partly a personal Quirk and it is also I think a limitation of the current language models that I never quite feel like I want them to write as me I'm very interested to your thoughts on how you relate to it in the writing process yeah when I put something out in my own name I basically don't use chat GPT at all for it I can use it I find for like voice of the show if I want to do like that time stamp outline or just create a quick summary that's in kind of a neutral voice where it's not signed Nathan and isn't supposed to be like representing my perspective but I haven't been able to get a I haven't really had a great synthesis yet to help create stuff that I want in my own you know voice in my own name so if you have tips on that that would be something I would love to come away with a better plan of attack on because I'm not quite there hey we'll continue our interview in a moment after a word from our sponsors real quick what's the easiest choice you can make taking the window instead of the middle seat Outsourcing business tasks that you absolutely hate what about selling with Shopify Shopify is the global Commerce platform that helps you sell at every stage of your business Shopify Powers 10 % of all e-commerce in the US and Shopify the global force behind Al Birds rothy and Brooklyn and and millions of other entrepreneurs of every size across 175 countries whether you're selling security systems or marketing memory modules Shopify helps you sell everywhere from their all-in-one e-commerce platform to their in-person POS system wherever and whatever you're selling shopify's got you covered I've used it in the past at the companies I've founded and when we launch merch here at turpentine Shopify will be our go-to Shopify helps turn browsers into buyers with the internet's best converting checkout up to 36% better compared to other leading Commerce platforms and Shopify helps you sell more with less effort thanks to Shopify magic your AI powered Allstar with Shopify magic whip up captivating content that converts from blog posts to product descriptions generate instant FAQ answers pick the perfect email send time plus Shopify magic is free for every Shopify seller businesses that grow grow with Shopify sign up for a $1 per month trial period at shopify.com cognitive go to shopify.com cognitive now to grow your business no matter what stage you're in shopify.com cognitive I do I definitely do I love it I think it goes back again to when you talk about being a co-pilot I think that the failure mode is usually trying to use it when it's a little bit more in delegation mode just go do this whole thing that's when it doesn't really work but as a co-pilot it really works incredibly well for specific microtasks in writing so first example that like as we just as I just brought up like Su everything is a summary like I often have to explain an idea like I was writing a piece a couple months ago where I had to explain an idea that that I knew the idea was I was talking about SPF and FTX is collapse and how like utilitarianism and effective altruism like what whether or not that philosophy contributed to the collapse and in order to write that article I had to summarize the main tenants of utilitarianism and I studied philosophy in college and I've read a lot of Peter singer's work and I just generally know it but I haven't written about that in a while ordinarily I would have had to like spend three hours like going back through this all the different stuff to formulate my three or four sentence summary but I just asked chbt and it gave me the summary in the context that I needed it in three or four sentences and that I I didn't use that wholesale but it gave me basically the thing I needed like tweak it and put it into my voice and so that's a really simple example but I think you can use it in all different parts in the writing process from at the very beginning I often just record myself on a walk just like spewing ideas and random thoughts reassociating and then I'll have it transcribe it and summarize it and pull out the main things and then I'll help me find little article ideas I'll often when I have an article idea I'll often start with just this like really messy document full of quotes and sentences and little like things that might go into it and then I'll be like I don't even know where to start with this this is crazy and then I will just be like can you put this into an outline and I'll just paste the entire document into chat gbt and it'll often find an outline and like the the outlines it comes up with are like really basic but sometimes I think what it's really one of the things it's really good at is like pointing out the obvious solution that you missed because you're too like close to the problem oh of course like the outline for this article is set up the problem and then talk about the solution to the problem that you came up with or whatever that that's such a common format for an article but if you're in your like head about it and you're being really precious it can be hard to be like for this special article it's going to be this basic thing that you've written a thousand times before same basic structure and then I think like one of the one of the other really great things is it's just incredibly good for helping you figure out what you're trying to express put into words what you're going for and then also going through the different options of like how to express what you want to express until you find something that like exactly says the thing you want like for example like trying to find exactly the right metaphor okay what kind of metaphor are you trying to find what's the idea you're trying to express and then here's 50 different options of like ways to express that with a metaphor and 49 of them will be trash and one of them will be aming or one of them will push you in the direction of what the actual like the one that you come up with is and I have like zillions of examples of that so I find that chat gbt is it's all over my writing but none of the stuff that makes it into the writing I publish is like wholesale from chat PT it's like doing some of those microtas for me all the time yeah that's interesting some of the stuff that you mentioned there I have had some luck with the talking to it on a walk is quite helpful in in some cases in try there I've done a couple things where I tried to dra like a letter and do as you said talk my way through it here's what I want to say I'm writing here to this person here's a little bit of context here's the key points I want to get across can you do a draft and then iterating verbally on that draft a lot of times I'll be I'll follow up and be like okay that's pretty good but you can give it pretty detailed feedback too the transcription in the app is so good that it is again point of privilege it understands me extremely well so I can literally just kind scroll through its first generation and say in the first paragraph I don't really want to say that it's more like this in the second paragraph more emphasis on this add this detail give it like eight things but you could wish it would do a little bit better on the revision I end up I've had a few moments there where at the end of that process I have something where like all right when I get back to the desk it's not that far of a leap from that to right the actual version that I'll use yeah it's probably still underutilized for me I should go on more walks honestly get more time away from the screen get the blood flowing a little bit and use a different modality the microtasks I should do more though that's I think that's the tip that I'm taking here is and it's there's a separation between like sometimes where I feel like it's hurting me is if I haven't and this will even now start to happen in like Gmail or anywhere where there's this like autocomplete that's popping up sometimes I'm like I'm on the verge of a thought that is really the thought that I'm trying to articulate yeah and then this autocomplete comes up and it's that's not right but it's but it can derail you at times where you're like don't guess for me right now let me get the core ideas down first yeah if you don't have those core ideas then for me it's been a real struggle to get anything good but I I think I've probably not done enough experimentation in the writing process of okay I do have some core ideas can you help me order them structure them iterate on them interestingly I also do use it at the other end often critique this here's a here's an email here's a whatever here's an intro to a podcast critique it yeah U that could be really useful if it's critiques are usually worthy of consideration at least I would say yeah it it truly is good at that and at every we have an an edit we have multiple editors who are like highly skilled and I still use it to be like what do you think of this intro because it's up at 2 am when I'm night before a deadline yeah it's real hard to beat the availability the responsiveness it's clearly super human on that I think the writing side is really fun I would love to if you're ready for it I would love to start just diving into how you actually how you use chat gbt sure um you sent me a doc with a bunch of historical chats and this was the first one give us the setup what were you doing and at what point we like oh I need to go into chat gbt and then and take us from there so I am working as the AI advisor at a company called Athena which was founded by a friend of mine named Jonathan and is this the like virtual assistant company the thumb Tac but Thumbtack yes he is one of the founders of thumb Tac and this is a different company but founded on some of the lessons that he learned in the Thumbtack experience he legendarily built up like a a really amazing operation powered by contractors in the Philippines and that included hiring an assistant for himself in his role at thumac who became like a almost key partner in his life over a long time and then Athena was built to essentially try to scale that magic for startup Founders Executives in general they hire executive assistants in the Philippines they pay premium wage they're really focused on getting super high quality people and the idea is to empower the the most ambitious and most high impact people by equipping them with this ability to delegate to their assistant in a transformative way okay now we're working on what does AI mean for us right how do we bring that into the assistance work so one of the things I've done is train the assistants on the use of AI and that's been a fascinating experience putting content together examples Etc another thing that I've done is just worked on building a number of kind of prototype demos for what the technology of the future might start to look like and this chat which we call Athena chat is basically our own custom in-house chat GPT is built on an open source project so I didn't I didn't have to code every line of it um but it is amazing how much how quickly you can build things like this today with a bit of knoow so it's been me and and one other person who have built a number of these prototypes in this case what we wanted to do is say can we create a longlived profile that represents the client that can assist the EA in all sorts of ways so it's essentially a plugin but with plugins you have some limitations whatever we experimenting with this on our own one of the big things we wanted to enable is adding information to the client profile updating information that's already in there so the hope is that this could be a hub where over time client preferences and history and even background context documents all can gradually find their way in there and you have this holistic view where the assistant can go query anything they need but again also update add to it's in theory supposed to evolve over time right so we have this chat GPT like interface and one of the things that we've noticed is that we still see despite our attempts at education like it's not perfect we still see that assistants sometimes need coaching on how to effectively prompt a language model so that was my motivation coming into this little thing I already had this react app which is again just aat GPT like little app yeah and I wanted to add a module to it the module I wanted to add was a prompt coach so I wanted to take a it put in another little layer where it would look at what the assistant the human assistant put into the chat app and send that through its own prompt to say are you applying all the best practices are you telling the AI like what role you want it to play what job you want it to do are you specifying a format that you want your response back in are you are you often these days we'll do it by default but are you setting it up in such a way where it will do some sort of Chain of Thought think out loud think stepbystep reasoning before giving a final answer that's actually one of the the most common things I see people do to shoot themselves in the foot with AI performance is prompt in such a way where it prevents the what is now the kind of trained in default behavior of explain analyze think about it a little bit before getting to a final answer so you just have like a number of best practices let me stop you there real quick what are people doing that would prevent the model from doing the kind of Chain of Thought best practice that that makes it reason the best anything that just sets it up in such a way where it's got to answer immediately with no ability to scratch its way through the problem is bad and see that very often it's common it happens even in like academic Publications not infrequently yeah often that's a hangover from the earlier era of multi-shot prompting and obviously this is all changing super quick right but if you go back the first instruction model that hit the public was open AI Tex D vinci2 in January of 2022 so we're almost on two years but still not even two years since you could first just tell the AI right WR me a ha coup and it would attempt to write you a houp may not at that point it was not necess going to get the syllables right the earlier Generations were you would have to say a Hau by author Name colon and then hope that it would continue the pattern that's the classic prompting and with instructions now you can tell it what you want to do and obviously that's gotten better and better but in the benchmarking in an academic context that was developed before this instruction change know typically you would have like question answer question answer question answer question and the ai's job would be to give you the answer and so they would be measured off on five shot prompts or what have you but a lot of that stuff was all that scaffolding was built before people had even figured out Chain of Thought and so now if you take that exact structure and you bring it to a gbt 4 you're often much better off just giving it the single question with no structure letting it spell out its reasoning because that again now it will do that by default and then give you an answer versus if you set up question answer question answer question answer it will respect the implicit structure that you are establishing and it will jump straight to An Answer often these are like multiple choice or they could be a number or what have you it will jump to an answer but the quality of the answer is much reduced compared to default Behavior if you just let it I think it think itself through it and I've even seen this in Bard I think this is hopefully now fixed but not too long ago Bard would give you an answer before explanation by default and again that's just like you're going to have a problem so that sometimes people do that by mistake they'll say give an answer and then explain your reasoning you're just hurting yourself right because it will explain its reasoning for a wrong answer once the wrong answer is established so got it Triple A is my in the EA education it's triaa for AAA results analysis before answer always I never heard that before I like that it's it's hopefully hopefully they'll remember it coming out no I like it hey we'll continue our interview in a moment after a word from our sponsors if you're a startup founder or executive running a growing business you know that as you scale your systems break down and the cracks start to show if this resonates with you there are three numbers you need to know 36,000 25 and 1 36,000 that's the number of businesses which have upgraded to netsuite by Oracle netsuite is the number one Cloud Financial system streamline accounting financial management inventory HR and more 25 net site turns 25 this year that's 25 years of helping businesses do more with less close their books in days not weeks and drive down costs one because your business is one-of aind so you get a customized solution for all your kpis in one efficient system with one source of Truth manage risk get reliable forecasts and improve margins everything you need all in one place right now download netsuite's popular kpi checklist designed to give you consistently excellent performance performance absolutely free at netsuite.com cognitive that's netsuite.com cognitive to get your own kpi checklist netsuite.com cognitive omnik uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work customized across all platforms with a click of a button I believe in omnik so much that I invested in it and I recommend you use it to use Cog rev to get a 10% discount and just to summarize I think basically what you're saying is a previous generation of prompting really encouraged in your prompt to give multiple examples of the kind of question answer or kind of thing that you wanted the model to do and then set up the last example such that the next thing the model would do is give you a direct response but what we found over time is one other really effective thing to do is rather than have the model give a direct response or direct answer to to a question or a problem pose to it is letting the model quote unquote think out loud first by reasoning through the problem just like a human would do a word problem and then at the end of its response give an answer improves the quality of the answer that that you're giving and the improves the quality of the result that you get from the model and what what has happened is open Ai and other model providers have made that more of the default Behavior so that it'll pretty much always do that but using previous prompting techniques few shot prompting or multi-shot prompting where you're giving examples might lead it to just answer directly and you should look out for that and try to avoid that it's a great summary yes if it is jumping directly to an answer you are for sure leaving performance on the table for yeah all but maybe the most trivial tasks yeah and just in aside see how much of creative work is just summarizing important because I I tend to give it the long long version by default that's my default Behavior this is one of those microtasks that I'll just be handing off to an AI avatar version of me at some point okay so let's get back to this you're working on an app and you want to add a module to it that explains some prompting techniques and it looks like the app itself is something that you didn't build from scratch and you're trying to get the lay of the land so you know what to do exactly yeah and the the problem is I know how to code generally and I've even coded in JavaScript quite a bit but react is a JavaScript framework that has a sort of hierarchy of best practices that if you know them and you can easily apply them then you can work quickly with the framework right that's the value of all these Frameworks if you don't know them and you're coming in cold like I was then where do I even go to there's all these different folders and file structure and like where exactly am I supposed to look for the kind of thing that I want to do and where do I put a new module and so that's where this chat really starts I have a working app I have the code for the app and but I'm I've never worked with a react app personally handson before so I literally Lally just set up the scenario and I don't really use too much in the way of like custom instructions or like super elaborate prompts in my co-pilot Mode work certainly in delegation mode then you get into a lot more detailed prompts with if this then that cases structured formats etc etc but I often just find a pretty naive approach is effective for things like this and so I just start off by telling it I'm working on this react app project and I am a bit lost can you explain the structure of the app I give it a little bit more information and it starts giving me a tutorial of what it is that I'm looking at and then you've got react and you've got Redux and then you've got these kind of additional slice JS tool kits and sagas and these Frameworks in in some cases take on a life of their own where there's whole conferences right and companies and it's like you can be very deep down this rabbit hole and whoever like built this open source project that I'm trying to modify they're using a bunch of different things that are not even necessarily standard but are common or whatever so just like five different things here that I have no idea about and without this kind of tutorial I'd be like going off to search for right okay what is this Saga JS what does that even do it's able to give me that entire rundown extremely quickly and then this I thought was a really interesting moment because I get a lot of value from things like this where I feel like it's prompting me it wasn't exactly that here but it gives me this General structure yeah and then I was like oh I find it as a general pattern it's if you can give it something in a format that it natively showed you that's probably going to work pretty well so sometimes if even in kind of the delegation mode sometimes I'll be like I don't exactly know what structure this should have but maybe if I have it suggest the structure then we'll get a structure that it can naturally work well with in this case the structure is like dictated by the world but it's pretty well known that okay this is going to be your structure of a project in this react framework okay cool but this got me thinking I should give it my actual structure like I want to print this thing out for this project that I'm working on because I didn't make it I don't know what it is and I want to have it help me interpret that full thing but then again I'm like how do I print something like this I don't even know how to do that so my next question for it is can you write me the command to print out the file structure and this is where you're like okay this is Magic right because now again I don't know how to do this this tree command I don't know if it was installed for me or not but okay shows me how to do it and next thing oh there's another step here of installing some package that needed to be installed okay it was a to help me with that so I'm just encountering all these this is the classic developer experience conceptually I have a clear idea of what I want to do but now I'm like three levels three nested problems down here right where I'm like oh okay I need to understand this framework right oh okay I need to print out the structure to better understand the version I'm working with in this framework oh now I need to install something so I can do that print and this is where people just time goes to die right it's like yeah you talk to programmers and you're like y you didn't get anything done today huh they but what happened was I was on the way to the market to to get my app together and then I had to install this thing and then I couldn't install but each of these things like it's helping me get over and now finally I'm able to say okay here is my app this is the app that I actually am working with and now we're really getting into something good because it can now break that down and the names of the things are like pretty semantic I notice I haven't even given it any code here I've just given it the file names but the file names have a kind of an indication of what is what and it gets a sense just from that of what the app actually is so let's go over to I think I just got a link to a working version of the app it's pretty simple it's a jat GPT like environment we can create these client profiles we have our chats we have our history a couple different models and there's function calling in the background that connects the chat experience to the client profile and what I'm trying to add is a module in the lower right hand corner which I'm actually not sure if this version has but the point of it is to take my prompts run them through this meta prompt as we discussed and then show feedback warnings of hey you may or may not be doing this quite right so back to the thing I've given it the file structure it's now able to understand the file structure and now I'm saying okay here's what I'm trying to do I'm trying to create this prompt coach I forget exactly how I had approached this yeah this is a different this is a different file let me see exactly what I'm doing here seems like maybe you had some sample code or something you'd written or yeah I did I guess I took one stab at it myself and it didn't work where I'm looking at you know the human version I was looking at the same file structure and I'm like okay I see that there's this module there's like a sidebar here and because you see these names right so you've got sidebar and search and there's going to be like chat history here somewhere chat I'm looking at this and I'm like okay I see all these different elements and I see all these things let me just try to copy one and mess with it a little bit and hopefully get somewhere right and then I'm not getting anywhere it's not showing up where I want to show up I'm not seeing it and so that's where I come to say okay now here's what I tried why isn't it working yeah and I explain my problem here at the end the problem I have is that it's being shown in the wrong place right so and then it explains the answer yeah next thing we're mod it's giving me instructions with code modify this put it over here this is pretty cool too unfortunately we can't share the old screenshots I don't know exactly what I used but this is right as Vision was being introduced to chat GPT as well so I was able to then say here's my screenshot here's where it is showing up and here's where I want it to show up right and can you help me with that as well so from the screenshots from the HTML structure basically we just work through this entire thing um I continue to run into issues we're only 25% of the way through this whole thing oh wow this probably took me I don't know two to three hours total to get these suggestions Implement them see what's going wrong yada y it it writes all the code basically because again I've never written a line of react code in my life so I don't know any of this syntax there's a million ways to get it not quite right when you have no idea what you're doing anyway and so it's writing all the code and just bit by bit we're refining The Experience we're refining the interface here we're creating some CSS we're using we have a a particular style pack that's already built into this so again that's just another thing I'm not at all familiar with you know this is the Syntax for figuring out how to use that style pack good luck making that up on your own and on we go basically after a couple of hours I got to a working module where the prompt coach you know would intercept your call do the meta prompt parse the response identify I had it giving suggestions and the urgency of the suggestions so we're color coding those suggestions as they come up if it's serious then you get it in red and if it's not you get it in yellow or just a notice and I would have I would guess that this would have taken me easily order of magnitude longer right in a pre chat GPT era if this was two to three hours it's probably two to three days of work to figure out all this stuff and and a lot more frustration that is because I'm not a super patient person the feeling of a million people have done something almost exactly like this there's nothing differentiated or special about what I'm doing I'm just like in this phase of kind of not knowing what I'm doing and just getting constantly stuck constantly stumbling constantly running into friction I really don't enjoy that I think most people don't this is none of that or almost none of it right even just that going back to the install right or the command to print out the structure man this is so stupid I know exactly what I want I know that it is doable I know that it's been done a million times million places and yet I don't know how to do it and liberating me from that frustration yeah is is and it turns out go your drudgery point right that was probably 80 to 90% of the time in a world where I was doing this on my own and now we're down to the two to three hours where it was really about defining what I want this could have been one hour if I really knew react but it taught me the ropes and did the task in probably again 80 to 90% Time Savings compared to the unassisted version I love this I think this is such a cool example example I really appreciate you bringing this cuz one yeah it's obvious that this kind of thing which if you're not a programmer like as a programmer looking at this I'm like yeah this is so much of what you do as a programmer especially if you're a programmer like working on a on Startup stuff is like this kind of thing it's like this is doable it's been achieved before I just need to do it in this in my specific context and it's obvious that this would have taken you days or taken really anyone days to do from scratch but with chat gbt it it makes it like way quicker takes away a lot of the drudgery but I think what's really cool and really beautiful which is like weird to say about this stuff it's striking me right now is there is this dance happening in this chat where at the beginning obviously you're asking it for um to help you but you are giving it what it needs and filling in the gaps that it needs in order to like help you and it is filling in the gaps for you as well so it is explaining react to you but you are are explaining here is the project that I have and here are the specific details that I I want done and and there's this like dance back and forth where you're mutually filling in gaps that both of you can't on your own fill in and I think that is like really cool to to just watch that evolve where at the start you don't know react and you don't know like where to where to put your code and you don't know why it's not working and at the start it doesn't know who you are or what you're trying to accomplish or what the specifics of your project is but as you build up this chat you yourself are starting to understand things more like you didn't ask it just go do this for me you asked how does a react Project work and like what what is the structure and so you learned more about react and it learned more about you and as your Mutual understanding increased you were both able to accomplish the thing together and I think that's really cool yeah it's awesome the Next Generation it's episodic we we're still only halfway through this scroll for all the scrolling I've done I just highlighted this okay cool this is working because at this point I'm starting in starting to get into like refinements okay now I I want to dial in The Styling and we're basically at this point the core problems have been solved and now again is just going to do the drudgery of making sure that there's padding and things are centered and and so on and so forth I try to be polite and encouraging to my AIS wherever I possibly can but you can Envision a future I think that future is already starting to become visible through the through the Mist a little bit as more and more stuff gets published on the research side where this sort of episodic relationship where I start a new chat it now knows nothing about this right I can continue this chat up to a limit and obviously superhuman expansive background knowledge but zero contextual knowledge yeah and we can't retain that from one episode to the next but I do think that is also coming soon too and there's a couple different ways it could shape up but I think we will in a in a year certainly not that much longer than that I can't imagine start to see things where all this history is accumulated or maybe divided into different threads or whatever but where this kind of can follow you forward into different tasks as well in a history aware way I think will be another level of unlock I think you're totally right that's what custom instructions is a step in that direction unfortunately custom instructions is very hard to set up but if you do set up up it's really great it's really nice for it to have context on you but I do think you're right chat gbt will definitely have a memory that it can reference this stuff and reference the context of what you need and who you are and that will make it even with the same level of intelligence of the model will make it like 10x more useful and 10x faster to to get you the right answer how much do you put into custom instructions because like for something like this it might be my profile my writing sample maybe whatever but I I probably wouldn't have by the way Nathan's a react novice and doesn't know how to install anything so would it do you have a vision or a sort of recommendation for a custom instruction that would help me with things like this you're asking the right person I have a very extensive custom instruction and and a lot of opinions about it if you want I can share them I can share it with you right now and we can talk about it sure yeah let's check it out okay the first part of custom instructions is what do you want Chachi to know about you and I actually like really having it know a little bit about who I am because there's enough about me on the internet that it knows my name and that actually helps same thing with every there's enough about every on the internet that it knows my name and every once in a while not having to explain who I am or what the company is that I run is like really useful for example I was thinking a couple weeks ago about starting a course and I was working with cat gbt to decide how to do the course and whatever and and the first prompt was I want to do a course can you help me think about it and with custom instructions on it it knows that I'm a writer and entrepreneur and so cool I'll help you build a course here's how to think about it because it knows that I'm probably going to build one but if I turn custom instructions off it will be like cool what course do you want to take and it's those like little things that like really make a difference for me but basically like who serious relationships in my life are I have in here like my sister her husband her son I have my girlfriend up there who people are at at every because referencing their names is just much easier for me to be like oh Kate when I'm talking about something and not have to explain who she is every single time is really helpful I think another really interesting thing is adding into custom instructions what are the things about you that you know that you're trying to like work on for example I feel like I'm I've a fear of rejecting people which causes me to be too agreeable I'm a little bit too opportunistic and I would like to be more strategic like stuff like that is really helpful to put in customer instructions because it's these it's those little realizations that you have every day where you're like wow wow yeah I I am a little too opportunistic I think chat GPT is great for being the thing that can help you as you're in the moment daytoday remember to pull back and incorporate some of these insights that you have that everyone has about themselves and same thing for goals like having it know what your goals are and bring you back to those things all the time as you're using it is really helpful cool well thanks for sharing I think I use it a lot more for just like very unfamiliar topics I'm very I'm just looking at these examples right that that we had queued up okay there's an app in a framework that I've never touched and know nothing about working on a patent application and creating diagrams for a patent application where I don't really really know how to do that at all is there again I'm starting with these very basic questions what's a good syntax that I might use to create a diagram for a patent application I just come in so cold but it it does suggest that you are doing doing a lot more kind of thought partner like brainstorming about your kind of core stuff which is yeah interesting I'm much more on these kind of episodic things that where like my history and this doesn't overlap almost at all in in a lot of cases but it's it just goes to show how many different ways of using these tools there are yeah too and yeah this could be another New Year's resolution to try to bring it a little bit closer to the core of what I do it's not to say that it's not at the core of what I do but not in this co-pilot way with things like weark like I'm working very closely with language models to make an app work well and I feel like I have intimate knowledge of the the details of how it works in that respect and that big it's a big project for me but again a different mode than the interactive dance kind of mode that you describe fascinating yeah no I that that makes a lot of sense I definitely use it for some of this knowledge exploration stuff too but yeah it's totally a sort of thought partner for me but I'd love to keep looking through some of the other chats you brought cool here's this next one on working on diagrams working on a a combination of a provisional patent application and the supporting diagrams for the patent application this is something that I was doing for waymark you know we have this Ensemble method of creating advertising for small business basically folks come in to the site they get to enter a website URL typically people will give like the homepage of their Small Business website we have some code that goes and like grabs content off of that website and then we build a profile kind of synthetically if you're custom instructions so to speak within the context of our app who are you as a user what's your business what are you all about what kind of business what images and then to actually create the video you give a very specific although it's like a super short short instruction I want to make a video for my sale this Saturday or I'm opening a new location and here's the address or whatever it's like these very this is my purpose in this moment prompt and then we've got a pretty complicated Machinery that takes all those inputs and it works with a language model to write a script and then has computer vision components that decide which of the images from your library should be used to complement the script all these different points along the way and that is it's a pretty cool experience now really compared to again you think about pre Ai and now what we had before was an easy to use template library and what we have now is really the AI makes you content like it's a phase change in terms of how easy it is to use how quick the experience is how much you can just rifle through ideas if you don't like the first thing you just ask it to do another and it's qualitatively just way more fun people used to have to sit there and type stuff in and they were like okay what do I what do I say and I'm not sure what to say and a lot of people are not content creators but but everybody I always refer to the Mr Burns episode from from The Simpsons this was a long time ago but he goes to an art museum for reveal of some piece of art and they reveal it and he says I'm no art critic but I know what I hate and that's I feel like exactly how our users operate they ask for something they wait 30 seconds they now get to watch a video featuring their business and if they like it they can proceed and if they don't like it it's very obvious to them and they can very quickly be like no not that give me another one and here's an alternate instruction so anyway this is the app that we've built and now we're like okay maybe we should think about filing a provisional patent on that like most software companies we're never going to prosecute our patents but we just want to make sure nobody can come in and give us a hard time so how do I write a patent and how do I create the diagrams and I want to be able to update I want to have something that's not like just a total mess so this was a a series of different inter actions that ultimately led me to these diagrams but I provided initially basically what I just said to you which is a rambling sort of instruction on here is my app and here's what it does here's how it works here's some of the parts behind it the language model writes the script and the the code that scraped from the website and then the the other part with the computer vision that figures out what what I just literally tell it the whole thing and say now can you use some syntax to make me a diagram that shows the structure of that app that I just word vomited to you and so there's like a bunch of different structures out there so that's the first part of that conversation as well you could use the mermaid syntax or you could use graph viz or you could use a couple other things what are the pros and cons of those and can they represent certain different kinds of structures we dialed it in on either mermaid or graph is it started to make me a thing and then you can see here too this is interesting because I did find in this one that at some point it got confused I'd given it this thing and had generated this syntax asked for refinements on the syntax because I'm taking the syntax by the way going over to another app what's cool about the syntax is you drop in this pure text right syntax and it will render the app for you right so you've got things like this graph viz diagra what's a diagraph I don't even really know it's called this diagraph is G and it has these elements and they have these properties and they're connected in this sort of graph structure blah blah blah you load it in half a second you know it renders it you're like oh now that's not quite right this point should be connected to this point and it's it's skipping one that so whatever so you give it these kind of iterations it would make progress but then it would also get confused it seemed after a number of rounds because there was just maybe like too much syntax so at some point I did say Okay using the episodic memory to my advantage or working around it's working memory weaknesses by just wiping and starting over I'm like okay here was the best one from that chat was closest to what I wanted it to represent we just go have another chat and this time we're going to skip all the part about which format do we use and Skip all the word Sal it and I can just be like here's a diagram I want to make some changes to it and now have it do more localized edits for me and so again a lot of little details a lot of nuance here but it's happy to do that we work through a number of rounds of it and I believe I attached the thing for you what I ended up with after or a couple chats you even get to the point where you're like color coding and really starting to make sense of it it's like right the Green in this diagram now is the things that the user does so the user tells us what their Business website is then there's code to go scrape then there's this Fork where we have to grab all the images and we process them in various ways one of the big challenges is which parts of this can happen in parallel and which parts depend on which parts this is actually something that we didn't have until I did this even for the technology team and I'm not sure how well all the members of the technology team could have even drawn this so now we actually have a better reference internally also to be like hey if we like what depends on the image aesthetic step now we can go look at it and be like oh okay yeah you can't select best images until you have the aesthetic scores you completed just having that Clarity is also I think just operationally useful but this is the the sort of thing that you can attach to a provisional patent application and at least begin to protect yourself from future patent trolls coming your way know again how long would this take if I had drawn it freehand I maybe could have drawn it in a somewhat comparable time to the amount of time that I spent exchange but having the syntax and now having it in that kind of you know structured language way also makes us like much more maintainable can fit in other things even can be like more readily used in language models the vision understanding is getting very good but I would say it's probably still better at understanding the syntax of the graph More Than This visual rendering of the graph I think that's great obviously chat gbt has the dolly integration so I'm familiar with that but I've been thinking a lot about sometimes I I want to create something that looks like this like in a graph with text and boxes and all that kind of stuff and I didn't even think to do to have it just write graph Fizz markup or something like that and paste it somewhere else so I think that's a really cool thing to know it can do and it's also pretty clear I don't know in a year it'll probably just render the graph Fizz stuff for you and you'll be able to like move it around and do all that kind of stuff without even necessarily having to chat back and forth after the first round or something like that I think that would be a very cool next step for chat gbt is jump into an edit mode for something like this closest thing I've seen to that so far is diagram GPT this is a slightly different notation but basically you can prompt in natural language yeah it will then generate in this case mermaid Syntax for you in response and then it'll immediately render your image and you can then edit the syntax you can't quite like drag and drop within the interface itself but I think this is a really interesting question around or highlights a really interesting question around like what things should be in chat GPT versus what things should have their own distinct experience even if there's still like a very AI assistant component to it this is one actually that I would expect lives outside of Chad GPT who knows right in the fullness of time maybe you have like Dynamic uis getting generated on the Fly we're starting to see that a little bit already but I don't think open AI is going to say what we need to do is create like a a UI where people can edit these graph things it possibly you could do that gpts don't really give you the ability to create like custom yeah editor experiences yet anyway so for now if you want to have something like that you have to bring it to a different app but increasingly these are out there as well right they just use chat GT and just a renderer so I had the AI doing all the syntax and then the renderer showing me yeah what it actually is and then going back and continuing the the dialogue with chb I think you're right like I could see a world where they let developers build their own renderers inside of chbt not for like really serious stuff I think like dabble half one time or make a little video or whatever that like having something in the interface so that you can do it in there like a rough thing is really helpful but then yeah I think you're right there will have to be other Pro Tools for people that all they do all day is make graphs um that are not inside chat gbt so here's another one this is recent episode in my life where I had to admit defeat after 10 years of swearing that I would not replace my car until the replacement was self-driving wow and we're not quite there so I finally and I've had three kids in the meantime so I finally had to break down and get a minivan like many parents of young kids I'm like know what my kids do is they really depreciate stuff uh pretty quickly so I was like I think I'll get a used right minivan because if I get a new one it's going to be pretty used pretty quick anyway so let me just look at what is out there now anybody who's ever shot for a used car knows that it's a total jangle right the car dealer websites are terrible what features they have is a huge question and what you end up encountering very quickly is these trim levels which if you're not like a car head you may not even know what that is but that is the sort of you've got your make which is the brand of the car your Chevrolet or your Toyota or whatever you've got your model which is the kind of car and then the the Dodge Caravan is the make and model and then you've got this trim which is often just like a couple letters or whatever it's like the xrt or the SRT or the L Limited or whatever they just have all these like little these are package levels right what features what upsells have been included does it have a sun roof does it have a screen in the back that drops down out of the ceiling for the kids or whatever right and it's just a jungle to even try to figure out what those things have um what levels there are and what those things have so this is perplexity which is a great compliment to jat GPT it is more specifically focused on answering questions so in this way it's it's a more direct rival to a Google search it's not so much meant to be like a brainstorming partner they really aim for accurate answers to concrete questions do a phenomenal job on it so here I had a number of rounds of this as well different kinds of questions or whatever but okay these minivans that are like not super old but old pretty cheap what do they have what do they not have and this would have taken I don't even know if I had really tried I wouldn't have done it right this is one of those things that you just I wouldn't do but if you had set out to go collate okay here's all the makes and the models and the trims and what they have you're going to be in like user manuals or something I don't even know really where that information is stored at Ground truth but just in asking that question I was able to get the trim levels for all of the different brands for this window of time and just easily get a handle now where that I could reference back to okay this one on this Dealer site it doesn't have any pictures it doesn't say anything but it does say for example oh it's an it's a SXT okay cool now I can at least know that is the second of however many trim levels or whatever so the SE that's your top one your SXT that's your you can imagine right trying to sort this out on your own and then you get the AVP who comes up with this stuff it's ridiculous but but it's super useful if you're like I don't want to drive across Metro Detroit to go look at this minivan if it doesn't have something that I really cared about and the things that I were zeroed in on were like fairly basic safety features I wanted the blind spot detection and the the backup camera so there were other questions too like when did USB charging get introduced into cars in general I didn't know the answer to that I'm old enough to remember when you had to plug the thing into the lighter and I didn't want that I don't want a car that's that old where I have to use the lighter Outlet anymore I want a car that's at least into the like USB charger era but when did the USB charger era begin for cars that was another one that perplexity was able to answer and it is so good I I think this is about to be a huge Trend if I had to guess because I've been a big fan of this app for a while we I had the CEO Arvin on the cognitive Revolution twice and they just they ship super fast they win head-to-head comparisons for answer accuracy but product itself is super fast it's got great UI with these sources and they're starting to become more multimodal with images as well which is relatively new which just a great experience all the way around and I I see it as like setting a new standard for answers that are like I I started I'm starting to use the term per perplexity to say I'm not sure this is necessarily Rock Solid Ground truth like perplexity is not always right but it's the most accurate AI tool it's usually right in my experience you might be able to find something here that is wrong but everything I ended up factchecking turned out to be true and so I think there's this kind of very interesting good enough for practical purposes standard where I don't necessarily need it to be a 100 100% accurate for it to be very useful and I would make my decisions did I trust enough for example to be confident that there was in fact going to be a USB charger in the car that I went to go look at yes and and in fact it was correct about that and so I have this kind of per perplexity standard of verification in my mind now where I'm like yeah it's in many situations it's like good enough to act on I wouldn't wouldn't make life and death decisions without more factchecking but I don't even need to follow these links in most cases now for something like this I'll Trust it and it's it's an emerging like standard in the family as well my wife asks do we really have to get a car that's that old do they have this do they have that and I was able to ask perplexity and send her like yep it should have a backup camera per perplexity it should have a USB charger it should have the blind spot detection and it's a incredible timesaver I think I just a worthy alternative to even something like a wire cutter which has been a standard that my wife has used for a long time but obviously that's an editorial approach where you can't just ask any question you want to ask here you can ask any question you want to ask and I think you do get something oftentimes that is like a worthy rival even to a much more editorial product no that that makes perfect sense it reminds me of wire cut it reminds me of there are all those sites that are like Kora but it's like for this new generation where no one had to think previously to ask this particular question and it can just gather and answer the question for you immediately and I think that that's so it's so powerful like it's really starting to click for me when and how I might I might use it there's there are so many questions I have this where I'm like I basically want to get to the best answer for a fact based question more or less and I'm so lazy I really don't want to do all the research and chat gbt will kind of like it'll do one search and then sort of crib the first article and this feels a lot better than that yeah it's really good it's faster than chat gbt on the browsing side so you're you're getting to answer notably faster and marginally more accurate just more the sort of answer that I want a lot of times like I've had a couple of instances where I tried the same thing with chat GPT and I was able to get there but it was like slower on the browse didn't give me the full answer the first time I was like no but I I need a little more and then I was able to get over the hump and get there but this was definitely just a faster cleaner experience that I do believe is a bit more accurate as well it goes to show that there are different roles that you want AI to play and I think there is it's interesting there's force is pushing both ways right what makes the AIS so compelling is that they're extremely general purpose and it seems like there is something there is like a a fundamental reality that they get really powerful at scale and to scale they have to be general purpose and that kind of comes as a package but here the scope has been narrowed and there are a lot of things that chat GPT does r people that this is not yeah trying to do for people and in its specialization it does seem to be achieving right higher Heights in the domain that it really attempts to be best in so I definitely recommend perplexity yeah a lot and I'm just old enough to remember when people were for saying that they were Googling and this a similar to me where it's a standard that I think people can comfortably socially transact on and feel like they're pretty solid I love this like you're using it to build stuff you're also really using it to fuel your curiosity and I'm curious like you know before we wrap up what are you excited about now what are you thinking about right now like what's on your radar that you think people should be paying attention to in in chat gbt maybe specifically but like broadly in AI or the next you know couple years boy broadly in AI over the next couple years I think almost anything's possible I take the leaders of the field pretty much at their word in terms of being honest reflections of their expectations and you listen to what Sam mman thinks might happen over the next couple years you listen to what Dario amade from anthropic thinks might happen over the next couple years and we are potentially looking at something that is superum in very substantial and and meaningful ways I think there's a lot of kind of conflation and talking past one another when people try to analyze this and I do think it's important to say you can be superhuman in very consequential ways without being like omnipotent or infallible and I think there's actually quite a lot of space right between like human performance and omnipotence or infallibility and I kind of expect that AI is going to land there for a lot of different things over the next couple of years so I think the value of the kinds of things that we will be engaging with it for is only headed up just a recent result from from Google Deep Mind on using their best language models for differential diagnosis was a extremely striking result this team has been on an absolute Terror it was only maybe like a year ago that they first got a language model to like hit passing level on medical licensing tests which hey that's crazy but you can discount and say well it's a test it's more structured the real world is messy and they're only passing you wouldn't want a doctor just merely passing okay guess what we didn't stop there next thing you know it was hitting expert level performance on the test next thing you know it's they've added multimodality and it can now do a pretty good job of reading your X-rays and other tissue slides and again is it perfect no it it would be probably on the lower end of what the actual human radiologist could do right although even there it was like 6040 I think I think it was like 60% to 40% that the human radiologist was beating the the AI radiologist so it's okay that's a pretty narrow margin obviously we're not done the current thing is taking case studies out of medical journals case studies being like extreme hard to figure out cases right when a case gets reported in a medical a journal that's because this case you know is thought to be highly instructive right it was a confusing situation it's an unfamiliar combination of symptoms or what have you so they don't publish just the routine cold right in the journals so they take these case studies out of journals and they had a study of comparing ai's effectiveness at doing the differential diagnosis versus human with access to Ai and AI was the best by like a significant margin the human alone was last uh and so they in their kind of presentation of this they're very modest and they take almost like a in my view almost like a two grounded willfully bearing the lead almost at times it seems and what one of the main conclusions of the paper was we need better interfaces so that doctors can take better advantage of this but it was like to me that's yes that's one lesson I would take away from this paper but the other lesson is that the AI is getting it right like twice as often as the human clinician like 60% to 30% that's another big lesson too that I take away from a lot of these things is we don't often measure Human Performance we think because we've lived in a world for a long time where like a human doctor is human OB we know that like some are better than others but we look at that as a standard that there's a human doctor and they're licensed and they're supposed to be good but like how often do they get the right diagnosis on this it turned out in this particular data said it was in the ballpark of 30% right so there's a lot of room for improvement and you could perhaps say what's the best doctor in the world do that best doctor in the world I'm sure is a lot better maybe even better than the 60% that their language model was able to do but you probably can't access that person we are apparently headed for a world where you should be able to access that AI doctor and if it's a 2X better performance on such a challenging task as differential diagnosis that I think we're headed for a world of radical access to expertise which I think is going to be at unbelievably low prices which I think is going to be a transformative force in society right it's going to be one of the greatest blows ever struck for equality of opportunity equality of access in many ways it's also going to change a lot of market dynamics and change what what wages can be commanded for different kinds of services I'm excited about that I also think it probably is going to be fairly disruptive and it probably is going to become more and more political it's I think that the upside of that I think is pretty clear and really extremely compelling so I I hope we do get to actually enjoy the the fruits of that future then one other thing I'll say is just I don't think we are the Transformer is not the end of History that's uh chat GPT is not the end of History this sort of no memory AI just this last week or two we've seen a flurry of activity in the state space model architecture and again it's been reported the headlines if you're on if you're on Twitter and seeing this stuff it's hey there's a new thing that might even be better than the Transformer it might be a Transformer successor it might be a Transformer alternative it might be a Transformer replacement it has some nice properties that the Transformers don't have better long-term memory better scaling better speed better throughput maybe we just all flip over from one to the other and oh the Transformer was the old thing this is the new thing but I strongly suspect that what we are going to see is a mixture of these architectures where just like in the brain we obviously don't have just one single unit of the brain that gets repeated over and over again we have a lot of different modules including some that do get repeated it seems like we're almost for sure headed for AIS that are like composits of different kinds of architectures that bring their own strengths and weaknesses in information processing to the table such that as much as this has been a shocking amount of progress to get to gp4 from gpt2 just four years ago I have to say I think the next few years are going to bring at least as much more change and it's it's going to be a wild ride it's exciting it's inspiring I'm excited for the future and I really appreciate you taking the time to share your thoughts and show us how you use chbt and I I'd love to have you back and see where we are see what new stuff comes up on the horizon So yeah thank you I appreciate the opportunity Dan this been a lot of fun and I definitely learned some things and was inspired to to go chase down a few more use cas cases as well so hopefully next time I'll have some better custom instructions and a little bit better uh track record in the brainstorming Department this has been a great exchange so that sounds great yeah thanks a lot it is both energizing and enlightening to hear why people listen and learn what they value about the show so please don't hesitate to reach out via email at TCR turpentine doco or you can DM me on the social media platform of your choice omnik key uses generative AI to enable you to launch hundreds of thousands of ad iter that actually work customized across all platforms with a click of a button I believe in omnik so much that I invested in it and I recommend you use it too use Cog rev to get a 10% discount

Channel: Cognitive Revolution "How AI Changes Everything"

Views: 6,122

Keywords:

Id: qnzA9sMYEqI

Length: 78min 39sec (4719 seconds)

Published: Wed Dec 20 2023