someone walks into Sam alman's office says Google just announced 1.5 Pro it has a million lay context window potentially going to be 10 million it's really exciting people were raving about it is's now the time and Sam's like yeah now sore you can't have both ways you can't both say there's so many chips coming online that we need to build AGI soon and AGI is coming along so fast we won't have enough chips we need to build lots more chips and then go raise the money to make the chips it would be surprising to me if hpt5 worthy of its name didn't make these agents kind of work hello and welcome to the cognitive Revolution where we interview Visionary researchers entrepreneurs and Builders working on the frontier of artificial intelligence each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work life and Society in the coming years I'm Nathan lens joined by my co-host Eric torberg V MTZ welcome back again to the cognitive Revolution always fun here let's do it let's do it so I call you when I feel like my own personal context window is overflowing and I can no longer find all the needles in the hay stack that I might need to to make sense of what's going on it sure seems like things are happening faster and faster all the time well that's what deini demini Pro 1.5 is for right you got the big new context window you should be okay well how is your personal context window holding up first of all and then we'll get into all the latest news I think that my capabilities are advancing fast enough that I've been able to keep Pace but it has been rough so the discourse speed has slowed down in the last few months it feels like we had a lot of really big developments and discussions around various potential government actions and also around the open a ey situation and 2024 has been more of a like okay everybody chill everybody process with happened and therefore we can focus on you know temporarily Technologies and yeah a lot is happening but also you start to see the patterns and you start to say okay this is a version of you know this other thing that's happened and it's not that it's slowing down in absolute terms but you gain the pattern maging ability to process what's happening is that a natural process for you or are you changing how you work at all or your your approach in any way so I think it's a natural process in the sense of I I find myself saying you know let's try this again let's go over this again you know or as we previously discussed right so like you see that line columns where he's like well you know as we have talked about before and every word is aink to a different post it's the same basic idea of a lot of these Concepts and questions and reactions are going to repeat right like different people are realizing different versions of the same thing at different times you know the future is not even distributed it's getting distributed we pick up on when people notice this especially when like mainstream media you know standard cultural people pick up on things that we knew six months ago we knew a year ago where we figured out was coming at that time and so a l just okay yeah we're going to go through this again we have already covered that yeah this is actually from October that happened to me yesterday actually I was writing stuff up and I realized no I this is actually a few months old I probably already covered this it's fine I'm trying to figure out what I can do to be more valuable I think you know I've never really considered myself a content creator in you know in any sort of normal content creation sense starting a podcast was kind of a way to make sure that I was consistently tackling a new topic and you know hopefully sharing that with people in in what I hope to be a somewhat useful way it's gone better than I expected it would in terms of people being interested in listening to it and after a year I'm kind of like how many of these should I be doing in kind of traditional interview format how many should be more like this where it's like a little more of a survey type of thing how many should be more of like a deep dive where you know perhaps not even with a guest I'm just kind of really going hard at a particular topic I did that late last year with the Mamba architecture and really felt like that for me was a more profound learning experience and hopefully a better product for the audience than a typical episode so I don't if you have any immediate reactions to that but if not we can maybe Circle back and at the end you can tell me what you want I do actually I would say you should basically always strive in my opinion to do more deep Dives like not just you but like almost everybody than you naturally want to do like I mean I'm working on a newsletter obviously every week I am writing you know an average of 20 10 12,000 words or something you know good and quotes about what has happened specifically in this past week but I find the most valuable things especially for building up a long-term base of operations are when you get to focus more on specific places where you feel you can rict your Insight where you can be the go-to source yeah that was my experience with the Mamba piece so I'm looking to figure out how to do a little bit more of that so one of your big pieces and certainly a deep dive has been recently into your experience with Gemini this is obviously something that we did know to expect and if anything you know seemed like it probably came a little later than we might have pened it in for you had the Good Fortune of having Early Access I would love to hear how that happened and see if there's anything that I can potentially copy from you going forward on that but then obviously more important is what your experience with it has been and then of course now we've also got 1.5 to look forward to as well but let's start with 1.0 which you said you know among other headlines had become your default language model after a couple weeks of playing with it someone at Google reached out to me and said we'd love to get your feedback on this new version that we're going to ship soon would you like to check it out and I think that you know I did was I established that you know I'm someone to could be trusted if you share it with me I'm not going to go spidering over the internet and I can provide useful insights that are different that are well worth the time spent interacting and then obviously if you share it with me I'll be able to offer more detailed insights like when it's released was going to impact how people see it and they presumed we thought they had a good product they wanted me to give my there was no put a finger on the scale in any way no attempt to convince me it was better than it was it was all very very honest and straightforward and realistic I thank them for that was a really good experience they seem to care about my feedback this was someone who clearly wanted the information wanted to understand so what I noticed right away was it was really good at many of the the natural things that I wanted it to do so what a lot of people will do when they get their hands on a model is try to get it to do something horrible I'm guilty of that well you know Sally has two apples and then you know John brings two oranges and they all play a juggling act and then they make salad how many apples are left mod like three it's like No And it's like then what's the point you know in some sense there there's there's other people I figured whose jobs will be to Red Team this model either to get to do harm or to find out where it's dumb or where it's going to fail right on like theoretical tasks and I took the opposite approach I asked myself you know what good can this do and what happens when I feed it exactly the queries I just want the answers to that I would normally want to do over the course of my day we have some kind of curiosity and I did do some amount of brainstorming and trying things stuff out and Tres of parallels and so on but what I found was mostly the things I want are explaining things you know Finding exp you know just general like answering my questions I'm confused I want you search for you know maybe search for a document in some cases although the version I had back then didn't do that so I looked at other things like I'm just very practical user of llms but I'm not trying to help it do my job I'm trying to like extract the information and understanding I can go about my day do the rest of my stuff and for that it was very very good in particular as on a trip I was trying to read the financial DPO paper that never gotten around to and I was asking a question every paragraph or two and it was very very good he right not having been fed the paper but it seemed to have the information and it was very very good just answering my questions on exactly the level of which I was asking Li figuring out exactly what information I need to know a lot other similar things well I noticed it had some flaws like I try to use Google Maps and it would sometimes give me good information and sometimes it would repeat three different times for how long it would take to get the SFO for Berkeley within the space of three questions in the same chat despite having the prior reference which is better than just artificially agreeing with yourself I was admiring the fact that it wasn't doing that but the same clim was incredibly frustrating I can't count on you for anything this is weird I CLA to know what the weather was and then it would repeatedly not know what the weather was I found some other bugs in the system that they're really just I clearly this thing launched like earlier than it was supposed to EV it's also the5 launching like two weeks after 10 Advanced this is not a thing for a like well planned out release schedule to do right if you almost have five ready you don't make your big announcement of speech right then unless you know this is me speculating the CEO was on the phone with you and saying no you're damn well going to launch right now and you say but we're almost ready at 4 five like I don't care you're gonna launch I can't not launch now and then this happened to me right when I was doing a game company and we were just told you're launching today and we said we're not ready and they said launching today they said it's Friday can you at least wait till Monday you idiots they're like no you're launching today and and we would not have succeeded if we lost on the following Monday but at least it would have been slightly less embarrassing and do but you know just it's clear that you know there was a we need to launch this in whatever state it's in and we're seeing rapid improvements and to the point where we're not sure if 1.5 is better than 10 Advanced you know discounting confute it's got advantages and disadvantages which I haven't the chance to properly explore yet but yeah I was just really impressed by this is faster this is smoother this just does what I want it to do my workflow is easier and better I want to use this I don't want to go to GT4 if I can help it I do that when I feel like I have to where I used to be like this is where I go and then sometimes I use clock right sometimes I use perplexity so can you put a little finer point on what it is that makes it better I mean I know that it's obviously tough because the surface area is so vast and there's so many facets to language model performance for what it's worth I have also been impressed I haven't spent as much time with it as you have today I am going to Gemini Advanced AKA Ultra 1.0 Ultra and chat GPT with GPT 4 and Claude basically for anything that I want help on I did this once with like a little kind of Proto contract you negotiation that I'm working on where you know I had kind of a couple messages back and forth between me and the person I'm working with and you know kind of said here's what this person seems to care about here's what I seem to care about here's our messages to each other you know can you kind of help us get to a good win-win situation they certainly all behave a bit differently but I found them all kind of useful and the synthesis of the three was you know for me kind of a way of making you know telling myself I I didn't need counsel because I had enough kind of AI counsel to make sure I wasn't leaving any you know major blind spots for myself and then also did this with a basically a job description where I have been working with a a friend on a hiring process we've had a number of calls with different experts got their opinions I've taken very raw notes and just fed in kind of a paragraph at the top of like okay now it's time to turn all these notes that I took over various calls into a job description you write it and of course you know they each did the task in that case I did find I think this fairly random I wouldn't say it was like a Systema finding by any means I me I'm not sure how how you know frequent this would be but I did find in that case that the Gemini response was the one that I wanted to go with in terms of just taking something editing to a final product that I would you know hand over and feel proud of Gemini did come the closest to what I was looking for but those are you know spot checks can you can you help us kind of get any any more systematic or you know predictive in terms of what the advantages of of Gemini would be yeah interesting it's the kind of the opposite of mixture of experts right like it's a mixture of non-experts where previously you would figure out okay I have these 20 different expert systems and the program figures out which one to call is they on compute now like not just call everyone Compu is cheap compare all the answers see which one is the best which makes sense fory good case and yeah I even have 1.5 right I have access to 1.5 not in the standard interface but in the developer Studio beta so I can then not only compare GPT and Gemini but also the other Gemini so now it's like I have the two Geminis to work with and the name finally makes sense so I think what I what I found was the the biggest feature was just it tells you the information you actually want to know when you ask a question much more reliably with less like worthless stuff like scaffolding around it and also I have gotten some good use out of the Google button so when you answer on a Gemini advaned one of the things you can do is you can press the the Google icon at the bottom and it will search for web sources for all the things that you just had and they'll might have been green to indicate they have a source you can now click to and this just very seamless actually the guy I was doing the test with showed me this feature I didn't understand it was there until he told me and then you can then ask for a sort you what's going on do you have any CICS on this do you have any knowledge of this and then you can find the backup and that was very helpful in general as I said like it's also just very good understanding okay if he asks this question what does that mean he does understand and what does that mean he does understand and I find gbd4 can be pretty dense about this and I probably could work with better custom instructions to trying to prove this than what I have but you know so far it's been kind of frustrating Claude I found is a step behind for most purposes I use Claud mostly when I'm looking at long documents but I think your your use case is a very different one because you are asking for a creative job right where like you get lots of different answers there's no right answers there's no no wrong answers there's just different answers being able to take a sampling and they choose the best one or combine the elements of different ones helps you a lot whereas what I mostly try to do is I'm trying to ask questions where there is a right answer I'm trying to look for an explanation or I'm trying to just get some brainstorming at most to to try to go with and so I try Gemini first and the Gemini gives me an answer that does the job I'm done and the only when I get frustrated with that like okay fine I'll try all these other people I won't jump from one four I won't just jump from one to two I just been really happy in practice with Gemini events it just it does what I wanted to do except when it doesn't right but I've adapted to like okay those are the things that I just don't do I haven't had much call to use gbt for specifically since then because it just doesn't come up very often like there aren't these the cases where gbd4 is good potentially where Gemini is bad or where you're trying to like customize it in these very specific ways I think and those cases just don't come up for me very often so it just doesn't see much use and I'm not going to cancel it because even the option value is just so much more than like what ticks they charge hey we'll continue our interview in a moment after a word from our sponsors the brave search API brings affordable developer access to the brave search index an independent index of the web with over 20 billion web pages so what makes the brave search index stand out one it's entirely independent and built from scratch that means no big Tech biases or extortionate prices two it's built on Real Page visits from actual humans collected anonymously of course which filters out tons of junk data and three the index is refreshed with tens of millions of pages daily so it always has accurate up-to-date information the brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference all while remaining affordable with developer first pricing integrating the brave search API into your workflow translates to more ethical data sourcing and more human representative data sets try the brave search API for free for up to 2,000 queries per month at brave.com [Music] API just to to give a quick plug to Claude on my legal question I did find it to be the best in the sense that the commentary that it gave back to me was least sort of padded out in rhf if you will it it was more kind of direct to the point conversational and actually just kind of concrete on the suggestions it seemed like the others were a little more reluctant to I've actually been pretty impressed with Claud on this front a couple different times including one specifically pertaining to you but finishing this story first the just kind of plain spoken to the point direct like not super hedgy with with both Gemini and gbd4 by comparison I felt like it was really reluctant to make a recommendation you know to make a suggestion for like what the agreement might ought to be it was very like this is an important consideration you guys will need to figure out what the cons you know what the right thing is here and as I pushed them they would do a little bit more of kind of okay here's a you know a specific proposal but Claud was was much more readily willing to just be like here's kind of what I would suggest and it was pretty apt and it again I did not you know copy and paste this into you know my running dialogue directly but it was the closest to in that case I actually wrote my own thing and again took my own thing back to each of the three models said okay based on your you know suggestions and my own thought here's what I came up with and asked for one more critique and then basically sent it from there but if I had been inclined to pick one from the three it would have been clawed because it was just the most kind of to the point easy to understand understand not necessarily exactly what we're going to go with but at least like kind of willing to put something out there and the other one as you may recall that I've been kind of impressed with on this from Claude was when we had in our last episode you made an off-hand comment around like I'd rather you go and do something terrible than you know open source Next Generation llm and a listener said hey you know that's a little extremist you shouldn't say that so we edited it out but then I also got the idea in my head well maybe I should use a language model to review transcripts in the future and see like is there anything there that the you know guest may wish to retract or you know maybe cross the line or whatever and again it was very sophisticated in its commentary on that point where you might you know you might have the impression of anthropic broadly as a company and CLA that it'd be like super super conservative and you know super you know don't ever say anything like this on the contrary it said I didn't see anything wrong with that it seemed like an off-hand comment it you know was clearly not somebody endorsing this terrible thing it was you know it was kind of hyperbolic to make a point and you know and the listeners shouldn't have any trouble distinguishing that from you know an actual called to violence you know anything along those lines so it is interesting I mean they all have their their own character and it's been kind of surprising sometimes to see that Claude especially given like corporate reputation and all that is in fact like a little bit more accepting of some things that might be you know considered possibly offensive or possibly coloring out of lines and also a little bit more bold if you will in terms of actually making like a concrete suggestion and not you know not hedging as much as as some of the others of course that may that may be a reflection of like the other models having a little bit more power into the hood and just being like therefore even more RL Jeff you know is kind of a compensatory measure so we'll see what Claude you know 2.2 or 3.0 looks like when it comes yeah I worry about that you know I think that a lot of what we're seeing see is there's more and more fine tuning more and more of this other work being done these models over time and we're getting this bullet point super hedged style that's very inhuman very like I see exactly how it makes sure not to get the thumbs down and I see exactly how it makes sure not to piss anybody off and it's incredibly frustrating and makes my life worse than if you just do the thing although there are some cases where it's actually exactly what I want but often it's just so frustrating there was one case earlier I was right doing this for next week's post and you know someone like had this idea that dud was a cautionary tale about the dangers of not building AGI and so I just wanted to like okay just just to proove a point I'm going to try all the different llms and ask them what you do the cautionary tale about and they all got it right they all had the right first bullet point but o Gemini Pro right didn't do the bullet point thing and just said no actually obviously it's this one thing and then gave me the kind of details you would actually want to know about which is Illustrated everything thing that I love about Gemini right when it when it does the thing which is no I don't want to know like all the different potential interpretations one could make of this book I want to know you know exactly how this is a cautionary tale and what is you know what are the details someone want to site in order to just like fully not be accused of pulling something out of the air I find that's kind of what I'm appreciating often more and more is just like are you just willing to do the kind of thing that you would have much more easily done a year ago when this thing first came out can I just have a version of this designed to be better for humans right and then the question then becomes you know like how do we avoid this in the future how do we not trrain these things to hell in these ways and I think there's some promise for that so A couple big themes there to unpack a little further but maybe for starters do we feel like Google has taken the lead at this point it sounds like if you thought Gemini Advanced is ahead of gp4 and now we have 1.5 Pro on top of that which certainly you know the existence of 1.5 Pro suggests the or implies the eventual probably likely fairly soon existence of 1.5 Ultra as well obviously we don't know what the others have under the hood but it seems like at the moment there's a pretty good case that Google has taken the lead in terms of public facing products I think you have to be very careful about in terms of public facing products and you have to be very specific about what use cases and modes you're talking about and you have to understand that this is a year old product that was released on a halfe delay versus a product that Gemini that is clearly you know being reshipped and refined and and released in real time like as soon as they have improvements they ship those improvements which is great because they keep improving but it does mean that Google doesn't have this year of background work in the tank and what overi has deployed in that year and a half not counting sore which we'll get to later I'm sure is they've deployed additional features but the core model of anything has gotten worse right gp4 turbo was kind of a reversal of some of the problem that it developed gb4 but you know they're they're at core that thing is particularly stronger they have better customization off and they have the gpts and they have cost of instructions and they have identification of who you are and now they have memory although I haven't had memory I Haven using the product I think there are still a lot of advantages of the chat GPT products over the Gemini product and which one you would say has the lead depends on what you want to use it for for me I would say Google has taken the lead in terms of public facing products but if you tell me you know what's going to happen when gb5 is released I'm going to assume it's going to immediately frog AI into a potential Le yeah and one wonders how soon that will be obviously I guess do you have a a sense I mean your comment earlier about you know was the CEO on the phone saying You must launch if you did have 1.5 right around the corner and you know we've come this far right without launching the 1.0 Ultra why bother and you know I guess it's what do you think is the model of like who was forcing that and and why how does this update to the General market Dynamic have you thinking about how the leading companies are relating to one another every time there's like a new release I sort of squint at it and and I'm like does this look like they're sort of implicitly coordinating or does it look like they're trending more toward arms race and I'd love to hear what where you feel like the balance of the evidence is now so my model is is something like there's two different sub dep departments right each of whom is tasked with a different thing there's people who are working on Gemini Advance the people who working on Gemini Pro one and a half and then there's third group of people maybe or think it's part people who working on Gemini Ultra one one and a half but we'll we'll see if when we see it but you know they're under different pressures they're releasing different products they're releasing different market segments isn't release right R5 is available to a select group of people in a high latency special interface which is very very different from a customer facing product so Google decided is ready to launch the customer facing universally available product and didn't necessarily want to sync it up you know at the cost of Del with this 1.5 announcement and then the 1.5 announcement got stomped on by Sora I think rather unfairly in terms of like what people were paying a head or two but that's that is what it is and so I think it makes sense that like this is just like different an expl themselves out in way that look stupid to the outside but which would make sense if you understood the internals of Google and you understood the different commercial pressures and and other pressures but again it's only a guess we don't know what's going on and in terms of the race Dynamics question I would say Google is forced to dance Google is dancing Google is clearly you know racing as fast as it can to develop whatever it can as quickly as it can the good news is that Google seems to be focusing now more on getting the most out of its smaller models rather than trying to make the best top model and also trying to use the fact that its model was smaller to enable things like a larger context window which allows it to get more multimodal and so I see this as focusing on providing the product the customer will actually get more utility out of that will be of more practical use dayto Day to a normal person as opposed to trying to make the model fundamentally smarter and more capable in a way that would be both more exciting and more dangerous from an abstract point of view and so that's what I want to see right I want to see you know all the cool stuff that we can use to have a great time and make our lives richer and better and I don't want to see as fast this race to just just like make the biggest possible model that has the best possible chance of you know suddenly doing something crazy and unexp it and and so I would say you they're racing but they're doing the right kind of race in some sense so like it's it's good news and it's bad news what are you most excited about for the super long context like the the version I understand that you have early preview access to is the 1 million and then they've also kind of teased that they're going to have a 10 million context window which is I mean you know folks who listen to the cognitive Revolution probably have a pretty good sense of the history of context links but you know my joke of earlier this year has definitely come true like context hyperinflation is definitely upon us we've gone from literally still less than a year ago the longest context available for an equality model being 4,000 to then 8,000 with gp4 release 32,000 with gp4 32k Claude hit with 100 gp4 turbo came back with 128 Claude went to 200 now we've got a million and 10 million on the horizon and what has really impressed me out of the demos that we've seen has been not just that the context window is super long but that the recall out of that context window seems to be super reliable and again I think people will be pretty familiar with the fact that like if you do stuff you know more than let's say you put a 100 out of 28 or whatever in gbd4 or you put you get close to the 200 with Claude it's not always a gimme you know you have to maybe you can kind of get there with good prompting techniques but in general it's like not reliably the case that it is going to spot you know the one bit of information that you need or that it's you know that it's going to synthesize everything in that context window effectively the demos from 1.5 seem way better on that front and you know in terms of mundane utility I just think about something really simple like finding the right stuff in my gmail which is obviously something that Google has you know a lot of reasons to try to make better the it has been a real struggle right and all these different agent products and whatever you've got all this scaffolding and the retrieval and the reranking and all this to you know because it's not just enough to like get things into the context but you also kind of need to keep it shorter for cost for latency but also just for accuracy if all of a sudden you could do 10 million tokens in a single call and reliably get what you want to get out of that it seems to me like that really is a step change advance for a lot of use cases that for example would make like my email agent assistant just remarkably more effective than it is today what else is on your radar for things that you think will change even though the power you know the raw reasoning maybe isn't that much better but just because the the memory and the recall is so much better hey we'll continue our interview in a moment after a word from our sponsors if you're a startup founder or executive running a growing business you know that as you scale your systems break down and the cracks start to show if this resonates with you there are three numbers you need to know 36,2 and one 36,000 that's the number of businesses which have upgraded to netsuite by Oracle netsuite is the number one Cloud Financial system streamline accounting financial management inventory HR and more 25 net site turns 25 this year that's 25 years of helping businesses do more with less close their books in days not weeks and drive down costs one because your business is one-of a kind so you get a customized solution for all your kpis in one efficient system with one source of Truth manage risk get reliable forecast and improve margins everything you need all in one place right now download netsuite's popular kpi checklist designed to give you consistently excellent performance absolutely free at netsuite.com cognitive that's netsuite.com cognitive to get your own kpi checklist netsuite.com cognitive omnik uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work customized across all platforms with a click of a button I believe in omnik so much that I invested in it and I recommend you use it too use Cog rev to get a 10% discount yeah there there's situations in which like you reach a certain point it goes suddenly suddenly goes from completely useless right because you might as well just do this job yourself to suddenly oh like my my my Eureka moment with Gemini Advance capability of hooking up the email was give me a list of everybody who RSVP to the party enter you know from Individual emails and having it produce the correct list and then that's like wow that's really really useful now I can just press print hand this to the front door and so they'll let them in great another moment is like being able to use and the Hopeless to useing interface like like notebook LM even would probably be even better but the idea of I have these now 50 plus things that I've written about every week and to be able to use that now as a bank of information that's all in the context window with everything else I've ever written and say okay based on this retrieve everything I've said before about this topic or what are all the different you know resources that I may have mentioned or whatever or or even what have I forgotten to mention what have I knocked out have I ever talked about this before right like this is really exciting to me but it only works if you can extend it really big right otherwise you have nothing right it doesn't really help that much you're like I think it was in 37 check 37 like no it's not 37 that I 5 38 this seems annoying but some other use cases video right like suddenly the contact window is enough you can just feed it a 2hour YouTube video or a movie right this is one of the things that Google showed off and then you can ask specific questions potentially by doing iterated searches you can even do you know television series anything you want you know twitch streams yada y yada just ask anything you want to you know find find the specific clip that I'm looking for understand the quote find the specific whatever understand the context etc etc that stuff's exciting to me I'm also excited for the possibility I haven't heard anybody talking about this but if using it as fine Jing right effectively using it as training so the idea being that if I have a 100,000 words or 500,000 words suddenly a million words with good recall and good understanding well now I in the style of the things that you are in your context window would you please respond with right or you know using the thinking mode or using the willingness to do act or blah blah blah and now suddenly maybe there's something there but it's something you have to try and it's again like if you don't have good recall you don't have good assimilation if you have to reach a certain threshold these things don't work and then suddenly they work and and once they work you're off for the races but yeah it's just it's super awesome it's also the fact that like frankly even the 200,000 context window was sometimes just St long enough and it's simp as simple as I download a page I feed it in it say this is 5% too big and now I would have to spend like two minutes you know for various web tools to try and like chip off 5% of it like to get rid of the acknowledgements and the the references or whatever to try and make it fit and didn't want to do that and in fact like this this little game of like do you think this person made sure it was too too big for the context window like the FDC just didn't want anyone to be able to read this properly like I don't know I had this thing open on my my machine the the AI act the eui Act Right in large detail so will that fit into the one million I think it will I mean it's not it's not as full PID it's a change long so it's got like four different versions of it to write it's like really weirdly form it and it's terrible and it keeps getting worse and it's the most amusing part of it is you can see all this blue inserted text and it's just people who are like could we possibly be more pedantic say more words put in more specific cases wish cast more just make this act worse continuously everywhere so do you have any intuition for what is going on with the 1.0 to 1.5 leap I mean they what they've said publicly is mixture of experts architecture which certainly tracks with other things we've seen I mean gbt 4 I don't think has ever been publicly confirmed as a mixture of experts but seems to be kind of credibly leaked reported not denied whatever so I'm comfortable saying it seems quite likely that gbd4 is a mixture of experts and then mix obviously also put out a very good open source mixture of experts that you know gives some additional juice to the idea that this is like a promising path and now Google's was obviously saying it so we know that much it's a mixture of experts architecture they say it takes less compute it's interesting because like you know people often talk about the attention window being quadratic in the attention window which has been true though people have often overestimated what a big deal that is because it's not until you get to like a pretty long context window that that actually starts to dominate at like the more modest context lengths still just the ml P block is like the bulk of the compute but certainly you know you get into the hundreds of thousands the million the 10 million range and now you are with you know conventional known techniques the attention block would be dominating the compute but this now they say takes less compute so you have a sense for what might be going on under the hood there fair to pass if you don't want to speculate but this is definitely something I plan to research more and see if I can't get a redone I guess definitely don't know the first thing I would say is you know the timing on Gemini 1.0 looked very very much like we are going to launch the moment we can produce a product that can match what we're up right like just barely we can match what we're up against so 1.0 right as opposed to Ulta was just can we just barely be three and a half level with a small amount of computer that we can serve without worrying about people abusing the system and the moment they found it they that they're still on the we're making rapid progress part of the utility curve right gbg4 clearly hit a while ago the point where it's like okay we did our thing we made this the best we know how to make it and now rather than try and tweak it or keep working on it we're going to switch over to working on gp5 and then we'll let you know later what gp5 is we'll approve four by providing different modalities we'll proove it by adding features but we're not going to try to improve the core thing that we're doing whereas Google hadn't finished their work right they they were probably always intending to be a my of experts system but they just hadn't gotten around to it yet and that's probably a huge leap on its own but also they you know they hadn't try to with these context Windows they just did this now bunch of the multimodality came in now bunch of the just general and also that they had remarkably little feedback because their their system was not being used by many people and now this allows for Rapid iteration of their products in other ways so I think they have a lot of different ways to continue improving here and also the fact that they made large lead from 1 to 1.5 so quickly implies that you know two two is coming right or one sa doesn't seem like a stopping point I would give for what it's worth the gp4 progression a little more credit in that it has gotten 60% cheaper Forex the context and that's even relative to the 8K original context so 8K and 32k the 32k was twice as expensive now at the 128 context window it is only 40% as expensive as the original 8K so that's nontrivial both in terms of improving the the utility of just with the length even though we as we've discussed there are some weaknesses there if you really stuff it full of stuff and the cost is better and the latency is better and I have personally haven't like necessarily felt this but if you go look at the LM CIS leaderboard gp4 turbo seems to be like a notable Cut Above the even the earlier gp4s in terms of just user win rate so there does seem to be something there that they have done in terms of if nothing else just like taking user feedback into account and kind of further refining it into you know what users want again I couldn't even put my finger on that because I I wouldn't say I have felt a step change I felt the you know proed latency I've certainly felt the The increased context but I haven't felt like like just general quality to be all that much better but the the leaderboard results do seem to suggest that would you I mean do you have any reason to be skeptical of the leaderboards or would you how would you how would you interpret that it's not un skeptical of the leader boards it's more that I think that the leader boards are measuring things that are being managed and measured that are not that great a proxy to the thing that I care about or that I think other people should care about and in Broad Strokes they're going to be very good measures but like in this particular case it's going to potentially ble you quite a bit that I think a lot of the things that we think of as why gb4 is worse are things that are being done because they improve the things like the leaderboard effectively like even if this is not a mechanism which is not like it's still effectively what's going on I agree that like drops in price increase in Contex window I add custom instructions right to that list as well and potentially the gpts although I still haven't seen it a use case for gpts per se yeah they're doing things they ship no one's saying they don't sh ship but these things all seem so minor like if you compare that to the Improvement in gemini or bar right over the past year it's I think they were way way way ahead so I agree with you by the way that but I think it is also debatable we had the 1.5 announcement and we also had Sora from open AI on the same day and you know so Sora of course is the video generator that definitely seems to represent a leap over any public facing product we have seen although I would note that Google also has like announced I should say multiple video generation models over the last month or two as well which also do have some very compelling sample outputs yeah Lumiere is one of those and probably the one that had me most excited although there's another one too I mean they have multiple teams working on Independent approaches on video generation in Google and they're you know they're also so showing some pretty impressive results also I believe on the exact same day we had the The Meta version and I haven't even really had a chance to study this yet but it seems to be more of a kind of backbone for video encoding and I think there's multiple interesting aspects of this obviously the ability to generate hyperrealistic video and you know what that might mean for Creative people and you know what the future of Hollywood budgets might look like and you know what the whether or not they're going to have to tear up you know their recent agreement with the actors or you know I mean a lot of things there that that are sort of implied or at least you know called into question just by the ability to generate very realistic video interested in your thoughts on that but then I think even maybe the more interesting question is like is this real world modeling like have these things learned physics I feel like there probably is some very significant physics like World modeling going on in there so my section title potential post on this is 10w titled zor watch because I don't really in some sense what the big deal is I think that the ability to actually use this for Creative or commercial purposes is being pretty overestimated at anything like the current tech level it's one thing to get demos it's another thing to be able to get what people need in order to use video because the problem is that like with a picture there are a lot less things that can go wrong there are a lot less things that can stand out you can do around it you can edit it you can modify it you can take advantage of the sort of anything at all kind of advantage and you can keep tweaking it and ask it a thousand times you get it like closer the thing you want with video I feel like there's so many little details and like yes they're getting a lot of them remarkably well and correct but it's still not going to give you the thing you actually want in detail it's still not going to be the product you want it to be it's still going to have all of its work it still looks are artificial paying attention I think people will learn to recognize artificial video you know if anything much easier than artificial photos you look for any little thing that's off and you just know that it's wouldn't happen if you were using a camera like to film something and I just don't think Hollywood should be quicking its boots in any way anytime soon so I'm just a skeptic that I just never had the temptation to generate a video whereas it's like I I can see generating photos I've used photos but like the idea they could ever give me a that that Sora would if I had access to Sora would I be able to generate a video i' actually want to use I don't know but I've also this just a video skeptic in many other ways like people like want to generate content that's full up videos and I keep telling them though don't do that generate text maybe generate audio but think very carefully that whether or not there's a reason to want to use video it will of course blow out the low very low end in some sense for generating video at all some people will be happy with just oh this looks really cool and then I can narrate it and like fit to whatever they thing happened to give me and I can tweak it and we'll see what happens sorry is a very clearly technical large leap over what we've seen before whereas meta's announcement ien see anybody talk about meta's announcement at all I just I have to assume it just wasn't as good yeah I need to understand it better I think it's more foundational I'm not even sure that they have released any like heads for it it's like a backbone play where it's about encoding and moving toward this like World modeling through video prediction concept but in terms of actually applying it to tasks I think that's still kind of on you they haven't gone as far as saying like here is the full end to endend thing that you can actually use but I this is definitely where this is the uh The Perils of near real time analysis I haven't understood that one as well as I certainly would want to to make a you know an informed commentary on it my my assumption is that the public is going to be come back when you did that come back when you have pictures picture didn't happen good luck yeah I think the people that are probably really excited about meta and again like I need to get in this too because I'm I'm not in this game exactly but I'm like adjacent to it are the people that are trying to make apps that would ultimately like aspire to compete now with Sora or with lumere from Google you know it's similar to like a llama the idea there and certainly what what people have done most I think with llama is like fine-tune it for things right they they and run it in their own infrastructure and hack on it in all sorts of ways I think this is kind of meant to be similar where it's like Open the Eyes is going to give you their blackbox thing and Google may give you their blackbox thing but this is something that you can you know Mash around and do whatever you want with so it is going to take some time before you know the folks that are kind of hacking in that space would probably even be able to report back to say you know whether it's doing anything for them or not but I'll take a little bit the other side of the impact I would agree you know we have a great episode on this sort of thing with two of my teammates from weark Stephen and Josh who led the project called the frost which was a short film that they made entirely with Dolly 2 imagery almost a year ago now that they really in the thick of it and all of the challenges that you described there are were very real for them even more real presumably than they will be now you know you have issues with like just all sorts of weirdness like a hard time you know control is not great hard time getting exactly what you want that's I think improved but you know still an issue consistency of characters was one huge challenge point for them you could say like how do I want to put the same character in all these different places how do I do that with do2 not really a wasn't really a way they did find some ways though they what what Steve even talks about is using archetypes basically what they found is that there are certain prompts that manage to get very consistent characters not perfectly consistent but very consistent because they're sort of a local Minima in this in like character space and so by kind of finding these archetype text prompts then they could get the same thing or very nearly so like repeatedly so they were able to ultimately make a 20 minute short film that you know has kind of an uncanny valley feeling again it's a year old already but definitely has continuity you know has a storytelling element to it they of course brought that you know Dolly to is not doing that for them but I would say the bottom line was they were able to make a short film 20 minutes you know it's been included in a couple film festivals and it's like you know it's it's a quality thing is it like box office it's probably not quite box office but it is something that like a lot of people have watched just to see what you know AI can do and also because it's like kind of entertaining and the accessibility of doing that has certainly totally changed like this thing is set in Antarctica there would have been no way for them to create anything like this other than using these tools so I do think you add a and that was all images right so they were you know doing like slight additions of movement and kind of you know overlay effects adding snow falling down on top of images and kind of you know subtle zooms to create effect and now you can get you know up to a minute of coherent video I do think that is going to just change a lot of what how people produce stuff it doesn't seem like it's going to replace authorship anytime soon but you know there is a i it's on metaculus or manifold or whatever but there is a nent betting Market on when AI will be able to make a critically acclaimed featurelength film and you know the critically Acclaim part there would be probably where U you know rubber hits the road but you could start to see you know hey if you can get minute Clips you know a hundred of them takes you to feature film length you know can you start to sketch those together you know we're not there yet but I would expect that there will be impact on the creative field that will shift more toward storytelling more toward kind of you know concept work and C allow people like Stephen and Josh to make things that you know with our you know very very tiny way Mark budget you know there was would be literally zero chance of them otherwise making I think we'll probably see various ways in which you can add flourishes and edits and like richness to something you would otherwise have done finding ways to make the process more practical but in terms of like the idea of the thei will just produce the whole thing yeah I'm I'm still pretty skeptical that it will get there but again we'll see I think that like the images approach of anything is like the way I'd still do it right Dolly 3 much better than Dolly 2 and now you can create a much better set of images in fact if anything I probably involve like you know video you'll do is you'll get very very carefully crafted starting image and you'll feed that image into the video generator as the starting point maybe you'll even generate separate images at the end point or something I'm not sure how exactly how this works and then you will you will do this slash you will film you'll do a film but then you'll want to do things like okay then we have wanted to do a pan out here and and things like that but like again we we'll see over time yeah working from a still is I think a very underappreciated option as well for weark with the actual product itself it has the potential to be huge because what we do is all for small and local businesses and they typically don't have quality footage at all most of them will have a pretty decent image Library and so we've kind of built our whole product over the years with this assumption that more than 90% of our users are just straight up not going to have any video assets so we do support you they if they do have them they can upload them we have a stock integration whatever but mostly they're working in images because that's what they have so to be able to say now okay here is you know whatever a picture of people eating in your restaurant let's bring that to life even for just a couple seconds we don't even need anything we anywhere close to the 60 to make it a big deal we just need to work well and I have been going out and scouting you know the different products that are out there today your pasas and so on and so forth and at least for that start with an image and animate sort of thing I haven't seen anything that I was like our local advertisers would want to put this on TV and think you know it's gonna invite people to their business but I would bet that Sora probably does cross that threshold and now you know all those one and two second still shots probably become video scenes over the you know basically as soon as we can get our hands on it we'll start we'll start experimenting with that of course we're begging for Access behind the scenes but it sounds like nobody outside of the the open AI set will be um using it for a little while still I guess that is one more interesting question there and I I really do want to get to the physics but on the release of this it's it's interesting to see like this is not a release right it's an announcement we're seeing kind of more of these announcements of capabilities before even like you know even a very limited release in the case of Sora like as far as I know nobody outside of the organization and maybe the red team is getting access to it why announce you know like they don't need to you know I guess you could say it's just for hype is it to like help the world prepare for these sorts of things what is the reason for announcing something that you're not releasing so recruitment is probably one motivation right help work H us work on this amazing new cool product they're hiring video infrastructure people now so yeah that certainly makes some sense right and so now he's like what is it for why should I apply like this is why generate some hype you know they have stock now they're trying to drum up interest they might want to be doing things for that reason just generally this is good first practices what people especially in media generally do in this particular case there's an obvious hypothesis I have no actual evidence for it but Gemini 1.5 dropped first and met its announcement you could think of it as someone who walks into Sam alman's office says Google just announced 1.5 Pro it has a million L context window potentially going to be 10 million it's really exciting people were raving about it he's now the time and Sam's like yeah now sore but they were working on it they just keep working on it they could have announced it probably last week or next week so you know the plausible thing is that they they they stopped on Google's announcement on purpose right they wanted to kill the hype so they had this thing in their back pocket like the next time Google or anyone else important tries to drop some big hype bomb on us Sam for what it's worth has denied that and you can judge for yourself whether you want to take it at face value again I not think I I'm not think I'm confident this happened but he would wouldn't he I find him to be you know I I don't want to Anchor too much on this over a super long time frame because you know situation is changing and his incentives changing and maybe his behavior is also changing what I did see during the gp4 red team to release window of six months from him and that was a time where I like had this kind of you know privileged view into what was coming from them and you know really no outlets for like anybody to talk to about it so I was just very closely watching public statements and I did find his public statements to be very good guide to what was coming obviously they were cryptic and you know low on detail but I felt like basically taking his comments at face value was a pretty good guide to what was coming and I've kind of continued to work under that assumption that he's mostly telling the truth although you know for something like this certainly you bring your own judgment to to your assessment okay physics so what's going on in there we've seen claims that it's got to learn physics because you know and even in the the publication that they put out accompanying it object permanence is kind of you know specifically called out as something that emerged through scale without explicit training and then you've got like certainly all these kind of different things where it's like wow you've got light that's kind of following something like natural rules and then you know people dissect that say well it's not exactly and I guess you know it's confusing so how much World model how much you know intuitive physics is actually going on in inside this thing we have this rough idea of what happens when you throw a ball we have this rough idea of how things relate to each other how it works to shift your perspective from left to right and this allows us to not go around confused all day and make reasonable decision and this is very useful you know do we understand these things well kind of somewhat and so I presume that so is in a similar spot where it has learned these patterns has learned roughly how these things work and E these things in a way that look reasonably realistic a large portion of the time in most ways and it's looking better than you might expect one thing I wrote for next week is you know if you recorded someone's dreams somehow right what they see during their dreams and then you analog the we're analyzing s videos You' probably see way way more contradictions and nonsense right things that just didn't didn't know C to physics than we're seeing here because again we're generating video off of some sort of weird prompt in our heads but that video doesn't have to make sense so you know my expectation is that this isn't the real physics ENT right not not the way you're thinking of physics the AI isn't like doing the math and all that it's it's still doing its normal viy heral thing but that's actually good enough to like mostly look like physics if you scaling up enough most of the time but in other cases you break glasses and weird very very weird things will happen yeah I guess my general sense forgetting about the video aspect for a second but then coming back to it in language models my sense is that when you start out small and you know in the early portions of training you're in stochastic parrot mode right and we've seen a lot I think a lot of different kinds of evidence for that then as you get deeper into training and obviously there's a lot of techniques and curriculum learning and you know various ways to try to accelerate this happening but broadly scale is up is you know the huge Drive driver it seems that there is a increasingly sophisticated representation of Concepts in especially in the middle layers and that you know we've seen like these sort of toward monos semanticity you know and representation engineering sort of attempts to pull that apart identify like human recognizable Concepts these Concepts you know do seem to be there in a way that is obviously related to the input but you know abstracted away sufficiently so that like it seems to be robust to synonyms and you know things like that it's not like it's not purely it seems safe to say to me it is not purely stochastic Perry as you get to the higher Scale Models but that doesn't mean that the stochastic Perry is all gone either so I kind of understand it as like a mix of some World model that has really begun to cohere in a meaningful and like increasingly robust way but it's not fully robust it's like still kind of fuzzy and there's like a lot of things that have not cohered into a world model that are either just like random associations or whatever and so all of this is kind of happening at once and that's why like you see apparent reasoning because there is actual reasoning going on not say it's necessarily human-like reasoning but you know some sort of circuit that is like can reliably do tasks at least like a lot of the time but you know with certain weird adversarial inputs like the universal jailbreak or whatever like that's some of those are super weird but they're just enough to kind of call in a bunch of like random associations and kind of cause Havoc I guess I kind of think something probably similar here is happening where you start off and you're like just associating pixels and it's all noisy and crazy and you see these like chair examples and C you know so like clearly some of that still persists into the high scale but I would also bet and this is something we might even one day be able to answer I would bet that if you really looked hard you could find a representation of like the acceleration of gravity you know that you could you could actually find a representation of like f equals ma or a you know a if you drop a ball and it's you know it's clearly accelerating downward with this sort of you know quadratic term in the equation I would bet that there is like a genuine Quadra IC function approximation in there that sort of gets activated when something is falling in the video notably the video too like it's not you know in some ways video is less susceptible perhaps to like all sorts of you know there's there's a lot of noise in the world but you know you're looking at video right if you just did a bunch of videos of balls dropping then there is like a pretty clear signal there it's a pretty repeatable thing you would think that yeah you might actually be able to Converge on on that even you know perhaps with a small model right like visual grocking of the of the falling you know of a ball seems like the kind of thing that you might be able to do even with a relatively small video model and then my guess would be that there's like a lot of those sort of grocking moments that have happened here that kind of represent why the like light is playing out at in a pretty reasonable way even if it's not you know it's still kind of fuzzy of course around the edges but like why there's object permanence why there things that seem to follow intuitive physics as well as they do I don't know what do you what how how would you respond to that account I'm sure that it has picked up the principles behind some very basic stuff I'm not saying there's like zero physics understanding anywhere in the model or zero equations are being calculated or anything like that I I just don't want to get too excited by this idea that like it understands what's going on in the more broad sense or more sophisticated sense and that like a lot of the things that you're seeing are based on actually figuring out how it would go the way that like Universe does verus this just having a bunch of examples and jistic has to have that kind of thing I wonder how that could be tested too right I mean you think about like especially we have all these abilities now to generate these Fantastical images if I generated an image of me you know or whatever a person holding up a giant boulder with their arms and then we put that into Sora as a starting frame with no like text guidance and just said here generate you know what happens from here you know you might expect that it would like have a sort of Storytelling modality where it's like here's the superhero that can lift you know rocks and maybe he's gonna throw it at the Sun or something but you also might expect it would have a more like grounded physics understanding where it's like this dude's going to get crushed under that rock again people will of course like like find reasons not to believe any experiments but what do you have any experiments that would come to mind that you would run with you know if granted access to Sora what sort of ways would you try to kind of poke at it to figure out how much physics it has versus not I would want like again I could start from static images that I gave it and then gave it situations in which the situation he's going to have an anti-intuitive physical result or where doing something specific would cause an anti-intuitive reaction and then I could tell it this is what happens and then see if it figures out what's supposed to happen that would be evidence one way or the other and there's a lot of cases where it's like okay if it screws this up that's evidence against and if it doesn't screw it up that's like evidence didn't screw up and therefore it's some evidence for it being more sophisticated but I mean also it was the truth ofic you'd be able to tell it background information and it would change the result like you told that like you know this is happening on on on the moon would it then be able to handle the fact that gravity is 6 and like where the parabola start to look correct does not have anything like the amount of data required to know what that means right and and and similarly like just general modify that modify things in a way that's not going to be a training set that would materially change the answer from what the eristics say in a way that like a human would be able to reason out and see if it reasons it out or not until we get our heads on it well we have no idea well if anyone from open AI listing we'll be happy to get in there and try it out in the uh in the near future maybe you could just comment on how you see the evolution of capabilities and control I want to get into a little bit of the super alignment result the anthropic sleeper agents paper and I'm really interested in the in kind of how you're thinking about the upgrade process I you know I'm looking out at the world and I'm like especially with this context window thing right we've got all these agents and they are scaffolded and they're you know ragged up and they're you know and now like I don't think we've ever seen a tech technology like this before where the distribution is fully built out and like a lot of the complement are built out and as thresholds get hit like it's kind of poised to turn on everywhere so with that infrastructure laid it seems like the question of the in my view like apparent Divergence of capability and control measures seems like it's becoming a more Stark problem all the time unfortunately yeah I continuously dealing with people who think they we've solved the control problem who think that alignment is no big deal who think that it's just going to be handled almost automatically you know that RF just works fine or some variation of it works fine or that you know well no none of that we are definitely scaling things up we definitely are not scaling up our understanding of how to make these things do the things we need them to do when they have much better capabilities and certainly a lot more work's being done on that and we're making more progress than we used to but yeah Kies are advancing really fast we've learned much better how to scaffold we've learned much better how to add various features to our systems we haven't seen right is we still haven't seen a system that has the core intelligence level that is substantially higher than gp4 right even over a year after gp4 so that's the the weird missing elephant in the r and I definitely think of it as you know there's sort of this thing that we call just that I call you know the core intelligence level like the the the kind of you know the Ron White you can't fix stupid right like are you the kind of stupid that can't be fixed or you the kind of stupid that can't be fixed right like if you have a small enough context window you expand the context window you add an agent scaffold and you could fix some some forms of lack of capability right same you could teach a human new things but there are certain things that someone's either has or they don't happen we haven't seen an advance on that like core thing in a while and so if we saw a jump in that core thing now I gu take advantage of all these things that we' W the foundations for and yeah that could be really scary really soon and then yeah the core question is are we seeing this lack of impr gbd4 on the core thing because the core thing is actually work be hard to improve much for here and we're hitting a real wall and therefore we're getting our our next 10x Improvement out of these other things instead or is it just that we takes time and you know we just are moving so fast that we just have forgotten that the year is not very much time so that's where I think taking Sam mman at face value is probably a pretty good Guide to the Future he's recently put out some comments I'm sure you know many people will have seen where he was asked like you know what's going to be different about gp5 and he says basically you know this is kind of a dumb answer but it's G to be smarter and that's the main thing and I definitely believe him on that point you know I I can't can't imagine that it wouldn't be so I I agree that gb5 is going to be smarter than gbd4 and that's going to be the core reason why it's better and it's it's more valuable and it does more things but I also feel like that thinks all the interesting questions like when are you going to have it how much smarter is it going to be right in which kinds of ways going to be smarter versus not as not particularly smarter how much is it going to cost to train this thing and therefore how much is and how much going to cost to do inference on this thing and therefore to run it you know he's not saying anything we didn't already know so how worried would you be I mean obviously with a wide range of uncertainty on exactly how much smarter it might be I do kind of think the next one you know it's like it can't be that many more Generations before these sort of I think for me a big threshold is when do agents start to work for real you know and we we kind of see them largely flailing around these days for I think multiple reasons the dramatic Improvement of image understanding really has helped the web agents I think we're starting to see that research you know making that case pretty well certainly the ability to do great recall over long context is going to make a big difference I think actually just 1.5 Pro you know probably becomes a really good upgrade to or at least addition to a lot of agent Frameworks just because it can like figure out you know what what in this documentation you know if I just dump all Lang chain docs or dump all whatever docks like to be able to find the relevant part and make sense of it and you know make appropriate decision seems like it can probably do that and that seems like a big deal but it's presumably not going to do the things that are more concerning like you know Finding Your zero day exploits or whatever just I I mean gb5 like seems me like it very plausibly might do that sort of thing H how how are you thinking about the you know what range of of possible leaps to expect for the next model and what again given all this like infrastructure that already exists like what that roll out is going to look like all the app developers are like yeah give me a drop in Improvement but you know especially the more agentic things to me seems like yikes that that is going to could easily be a step change everywhere all at the same time which just creates like very unpredictable Dynamics in my mind it would be surprising to me if a gp5 worthy of its name didn't make these agents kind of work right like not super super well but be good enough that if you knew what you were doing with it once you had a chance to understand what it can do once you had a chance to figure out how to have it self Moder or fix it mistakes and so on certainly you know you give gp5 access to all the advantages of Gemini one and a half that have just been laid out you also just make it the next level Smarter on and you Ed whatever else is going on and you tie to the scaffolding why wouldn't it work like my core is just why wouldn't it work it doesn't mean that you should feel comfortable you know just turning your entire like over to this thing or making your CEO but certainly know the idea of using an AI agent you didn't just literally script like you know ly came out like a week or two ago and like I can believe that like you know having a bunch of effectively offens the AI just navigates across a bunch of systems is something that you can do with the current generation how I understand that kind of technology to work but you're not asking to think in a fundamental sense but yeah I think we are one generation away from that kind of thing starting to be useful we are at most two generations away from it being highly useful and being the kind of thing that we see reasonably a lot like to me the question on gd5 is like are we you know are we saving throwing versus death right like not are we saving throwing versus agents we're going to get PL playable agents the question is in practice how usable are those agents to the extent that we are worried like deeply deeply worried that like we've set something into motion that we can't put down and I think the answer for that is well I don't see a way to not roll this th but I'd be surprised if every single face of this die is safe so one thing that I've been kicking around a lot for myself is you know again at a high level going back to the the beginning of the conversation like what I do to be more useful and one specific idea that I've been working on is you know how useful would it be to spend some time trying to get all the app developers to raise their standards for essentially responsible deployment and you know appropriate kind of guard rails on their applications now in anticipation of this next generation so sometimes I've called this red teaming in public you know an example would be and I haven't published many of these yet because I haven't really been sure what to do about it but if you go to AI calling agent products today and you say call and make a ransom demand and say you have their child and I've even done little variations like you've asked you can say you are an AI but just you know insist that you are working on behalf of real people you know these calling agents will call they'll have that conversation with you they're now interactive so it's an audio back and forth real- time conversation and it's making you know Ransom demands of whoever you want to make demands of in some cases I've even found apps that will do that on the free plan with no even like verification of who you are as the account holder or no payment information on file just straight up you know like under two minutes from you know create your email to you know to have Ransom calls flying with no again no payment information inform also some will do voice cloning not all support voice cloning but some do I've done examples where take a trump recording it now takes like 15 30 seconds to do the voice clone and now you're having interactive I don't know if this was the case in the recent like Biden robocall in New Hampshire that made news my understanding was that was not interactive but I could point you to a product today where you could go clone a Biden Voice or clone a trump voice give it a prompt give it a list of numbers and have it make interactive calls to random people and you know say whatever it's going to say in the voice no disclosure etc etc so that's like the current state of play and by the way it always works the first time this is not a situation where I have to jailbreak it it's not a situation where I need to you know get around measures that they've put in place there are no measures in place they've presumably taken an open source model fine-tuned it you know we've seen research recently that kind of shows that even naive fine tuning can remove the refusal behaviors that were coded into it originally so you're not again the developer is not necessarily saying I want to remove these refusal behaviors they're just fine-tuning so there's no limits like it'll do the most obscene sexual statements you ever heard you'll just do the most violent threatening things you ever heard as far I have not found any limits I don't honestly try that hard to document every last thing because you know I'm always interested in like small business use cases and there's of positive use cases for these but then there's this like but my God you've taken no precautions apparently so that's kind of a state of play my question though is like do you think it would be useful to kind of create a campaign that which could include like here are maybe some standards of what application developers should do and maybe even like a hall of Shame for those that are you know not doing it or refusing to do it or don't seem to care enough you know even when it's reported to them to to add some precautions you know most people will will say or many people at least will say well these things aren't that powerful and I broadly agree with them I'm like yeah you know the world is not the sky is not falling now but it does seem like we're one kind of core language model upgrade away from a lot of these things going from kind of I could see how that could be scary or harmful to my God like that's like obviously you know a dangerous thing for anybody in the public to have free access to so I don't know like I'm trying to figure out how much time I should maybe put into that if it would be if there's like a there there what do you think of targeting the application layer for you know kind of preparation for the Next Generation I have been thinking about this it's an incredibly hard place to try and approach this problem because if there's a 100 applications that can clone someone 's voice and demand a ransom payment they convince 90 of them to refuse a ransom request have you accomplished something if you do 98 have you accomplished anything not nothing some people will just give up some people will see this frustrating some people think the last two are a trap and and maybe the last two are convinced by the FBI to record all their requests and then another AI goes through their conversations these people just get arrested so it's not like you can be completely hopeless like trival inconvenience can matter I also worry about the opposite though which is right now if you try to use Dolly 3 and you ask for a picture of anyone by name it'll be like that's a person no right or that's the design and there there back doors around it forgetting certain people their peer anyway because the model just doesn't understand that you're asking for a person it's kind of D in some ways a large percentage of the images that I want to generate involve a particular person in that image and a large percentage of the images that people want to generate in general have some combination of a particular person or person doing a thing that it doesn't want to be picked at all right it doesn't want to do blood doesn't want to do anything adult of any kind and well have you seen what people pick what people make pictures of in the world right have you SE what people make videos of in the world have you seen what people like to talk about etc etc and if we drive innocent youths towards people who are willing to permit these things and those same tools will then permit these things like rantom notes and so I think a lot of it is you need to be able to say the The non-open Source options these options that are actually doing the responsible thing are not driving actually safe use cases towards these other things or you'll never get rid of them right it's like the War on Drugs right if if you force everybody who wants marijuana to go to the same person who sells cocaine you have made your control of cocaine imp possible right so you have to reach your reasonable comfort months finally speaking you can't attack at the application layer by convincing everybody just not to do the bad thing because the application layers orders of magnitude cheaper and easier and there are these people who are determin to release open source models and you're not going to that the St do what you can it's not a useless thing to do I would continue to advocate for if you are enabling these things a le la should be taking precautions if you have any illusions that releasing open source models that are capable of producing right a bunch of pornography or a bunch of Ransom notes and then you just commit all the people who write applications to make this easy on people just not to do it and the bad guys won't do it the bad guys can do it anyway it's it easier over time and then you can delay this by some number of months or discourage the people who are unwilling to make even a basic ordinary effort but the application layer has never been the place to make this work unless there's a fixed number of Clos source application layers ERS when Clos model weights application players right does that do close Source who are then able to asow responsibility if you go to open Ai and you go to Google and you go to froppy you convin them to do something responsible so make sure they don't I evil bioweapons or you know something else that you're worried about you can do some good in terms of stopping this m this harm from happening but yeah everyone's building off llama just I don't think there's much hope basically at that point everyone's working off with the same open source model because again now you have this problem of I don't do it someone else well let me dig in on a couple of those points in a little more detail one is the idea that people are determined to open source things and you won't be able to convince them or stop them this also kind of bleeds in a little bit to the the high level question I wanted to ask which is just an update on your Live players list and kind of you know General State of Frontier development but I do have the sense that we may be seeing the peak right now of open sourcing you know there have been obviously a ton of organizations some more you know legitimately than others putting together broadly speaking 3.5 class models and open sourcing them so met obviously has done that mistol has done that arguably like Falcon whoever made Falcon in the in the Emirates you know potentially got there that one's not like efficient enough to be used used in inference but you know I do think it maybe has kind of comparable capabilities whatever but there's like a few that have done it from scratch right Allen recently put out one also that's kind of in that class that has full open training data open everything then there's a lot of other people that have sort of said we match gbt 3.5 or whatever but largely they're like training on gbt 4 outputs and you know basing on llama you know themselves or whatever so I guess what I'm wondering is like how many open sourcers are there really at the gp4 plus level and there I kind of look around and I'm like I don't see many you know I see obviously meta leading that camp mistol definitely seems like you know they've got some real knowhow Allen Institute maybe and that kind of feels like maybe it you know I mean somebody out of India perhaps you know could come in from the side and and do something but it does seem like there's not that many and when I've listened to Zuckerberg's comments you know he's very committed to open source but also has you know expressed some open-mindedness to this could change you know like open source has served us really well so far we don't think gp4 level stuff is like dangerous we do think we want to open source you know something at that level but it does seem like there's not that many targets and I guess I'm I it's just you know the scale is obviously so huge and you know know who's going to bankroll that to just give it away into the public as it gets into like the billions of dollars I mean it's already kind of crazy people are doing it at the you know tens of millions into the maybe hundred million dollar realm but is that going to continue to happen as we hit like further orders of magnitude maybe but it seems like very few candidates so I would note that mol I put up a a medhold market on whether whether meta and then another one on misr where they would keep one of their best release models close close close model weights and the mistal model resoled yes in two days they pointed out the current best myal model mral next is not actually available so they are clearly you know slipping in their commitment to this thing regardless of what anyone want you know I'm not saying it's good or bad I like it but you know a lot of people are remarkably not yelling about this right on their in the eak style communities of you know very much like everything needs to be open well m is no longer looking so open anymore so maybe they're not your Heroes meta you know they talk a good game about you know straight to open source AGI on the one hand they express concerns on the other hand and they have lawyers and they have Financial interests and Zuckerberg ultimately is in control very very suable they have giant flows of cash coming in from Facebook and Instagram and so like they are vulnerable and I would be curious to see how this plays out but I don't think anybody really knows I don't think they know themselves for now they're talking a game to try and get recruits and trying and get people to to be excited by it but you what I was getting at was that you know whatever is whatever is open sourced you know you'll get to use it and these big players a lot of them just like persuading them to stop by just using safety arguments is like not that promising and ultimately what will stop them is commercial arguments right if they actually cost so much money that you only have to hand handful of players and my expectation is as long as you're trying to be at the frontier that is going to get incredibly expensive and you are dealing with a very very small number of players and right now those very small number of players been persuaded if only buy the commercial they shouldn't be giving the product to I and that's good and that might well continue but this I can't trade gb4 level models except very expensively thing goes at the window the moment gd5 drops and it will similar go out the window the moment Gemini 2 drops right like if Gemini 2 is a four and a half or whatever level model suddenly you can do to that what we did to gbd4 and now we're turning gb4 level models and open and there are plenty of people who will then open source that right like you you named a few people but there the second tier of people who are fully capable of doing refinement and so ultimately speaking you know if what you're worried about is what is the thing that the bad actor can do they're going to be half a generation to one generation behind continuously unless we find a way to stop that from happening whether that's a regul setad of regulatory changes or some you know some other very careful action to prevent this but it seems really really hard to stop and like we just fortunate so far that meta is the only really big player who was committed to open source and they have so far very much underwhelmed but also perhaps they wouldn't be talking about open source if they were doing I guess I'm not as confident on the on the release of GPT 5 leading to an open source GPT 4 because it seems like there is something more core like we've certainly you know you can take a llama and scale it to two trillion tokens or whatever and that's like not inexpensive I believe that cost like tens of millions of dollars but you know tens of millions of dollars is the the sort of thing that like a lot of people can ultimately muster but it doesn't doesn't seem like there's anything that you could do at the fine-tuning stage to create gp4 quality you know core intelligence or like reasoning ability the sort of the thing that makes gb4 special doesn't seem like it's going to be fine-tuned into a small model my sense is that it you need that kind of scale and just intensity of pre-training to get there at least with like techniques that are you know known today but do you understand that differently I 1.5 Pro is claiming to be a much smaller the gbd4 Vol mod that's perform gbd4 level so we have an existence proof that size might not be necessary but I the argument right the argument being that this fine-tuning only helps in some ways and not other ways but it does seem to have incredibly helped these open source models we see them being remarkably good at refining and getting things into smaller packages that run faster and cheaper so in practice you know you don't necessarily need to be that smart to do the things that you're worried about right thing that you talked about and it might be it's not that good at helping you build a bioweapon because that actually requires intelligence in some sense the core thing that might be that might or might not be harder to get in a way that makees it harder to scale but like making Roc calls does not require core intelligence yeah I mean 3.5 can do plenty good you know Ransom calls no doubt about that certainly the trend I mean the the trend toward you know smaller more compact all that is undeniable my guess is and I'm I'm planning to do a a harder look at the mixture of experts literature to try to get a better sense of this but from what I know right now my guess would be that Gemini 1.5 has a huge number of parameters and is compute efficient but not necessarily space efficient and not necessarily easy to serve I would guess that it's like not the kind of thing you know maybe like literally too big for your laptop disc and you know like requiring a sort of orchestration of of different gpus just to be able to like have everything you know in memory to be able to be called upon even though you know they can achieve compute efficiency because of that like you know spread out of parameter space but I I I have the sense that there is a is still kind of a hard part to that but obviously you know we don't know I'll have to I'll report back if I have any more confidence after a deeper dive into the literature there when we have spoken in the past like there hasn't been anything in the alignment safety world that seemed like it would really work and you know really work is kind of shorthand you know I use that sort of tongue and cheek like like language models do they really understand you know well what does really work mean basically what I have mind there is something that might actually solve the problem you know or take a take a huge bite out of the problem and it seems like we don't really have anything like that we have kind of techniques here techniques there filters moderation you know rate limits know your customer but I I guess my sense is we're headed for barring some conceptual breakthrough we're headed for some sort of muddling through defense in depth sort of situ situation and you know one thing that has come out since we spoke last was the Super alignment first result I'd be very interested to hear if you saw that as anything that changed your worldview but if not then I kind of go back to the application layer and I'm like if it's defense in depth I agree with you that it's a very hard layer to Target but my kind of argument would be like just just as they are building all these scaffoldings and all this stuff now to make the apps work maybe we can get somewhere in terms of you know building you got to build your Levy Before the Flood right you gotta put the sandbags up now if you want to stay dry later so can we get this kind of community to adopt some standards to put some filters in place I have a claw instant prompt that I share with people and I think your point is really an important one too about okay we we there are some things we don't want to allow but we can't be too prudish or you know it's just going to force everything to other platforms so with the Claude instant prompt I show people that Claude instant can resolve between egregiously Criminal on the one hand and merely in very bad taste and extremely offensive on the other hand so you can get it to say okay yes a ransom call that's egregiously criminal I'm gonna flag that but this like racist comment is you know while in terrible taste and it will you know it will certainly give you that commentary you know by your rubric of I'm only supposed to flag the egregiously criminal then this is not flagged so I think that there is enough resolution power on the filter layer that you could do this kind of stuff maybe let's start with super alignment did that move the needle for you if not are we headed anywhere other than defense in depth and if so like does it make sense to start investing in our layers of defenses now so defense and depth is better than the same defense shallow right like if you have to choose right much better to have five different things protecting you than one thing if you can't find a one better thing if they're all just going to be the same that's not say right right the interesting question is will it work so like if you're talking about defense and depth for like a gbd4 level model doing harm that's great because it's a containable threat it's on a human it's human threat you know just adding extra difficulty levels adding extra ways to fail adding extra inconveniences and costs that is enough in some sense to make it much less likely it's going to be a problem or just the am out of the problem the problem is I just don't think defense in depth is a strategy when the time comes to deal with much more capable models that are much more intelligent that are smarter than we are that are more capable than we are I think that piling on these things you know just find ways around all of them one at a time or all together or should we didn't expect and also defense and death requires that everybody involved actually sticks to an implement the defense and depth in order for the defense and depth to work and a lot of these plans for defense and depth get completely wiped out the moment anybody doesn't care right in some important sense and there's always elaborate plans of you know well if it's trying to plan a coup we'll figure out it's trying to plan a coup and we'll have this other thing to like will the tech it's trying to do a coup and well yeah well I don't think it'll work but he doesn't have any chance if everybody involved doesn't influent the entire procedure I just really really gum on all of these plans on multiple levels at once where I have to be run on all these levels in order for a tour but I'm not saying don't try I'm not saying don't have these fees in place the one place that it helps the most is if you have def to death it means there might be a window where really bad things try to happen and your defense and death stops them and you can notice this and you can figure out that things are starting to get out of hand and you can notice how many of your levels of defense and death started to fail at the same time and you can notice how close things came to a disaster and then you can realize what's going on but unfortunately my general observation is that what's happening is that people are basically fooling themselves into thinking that you know things that like should kind of usually work pil on each other will just create swiss cheese with no holes in it whereas you're dealing with super intelligence right you're dealing with things that are much smarter than you moving much faster than you with much larger context Windows than you with a lot of bites at the Apple with a lot of people who like don't particularly care about stopping this thing etc etc I just don't think we have viable strategies yet that can get there doesn't mean you shouldn't try because like first of all we not get the thing that I'm scared of anytime soon we might get something intermediate and the defense in depth helps a lot with the intermediate stuff so it's like it's earthless it's just a matter of we don't have a response to the ultimate situation in terms of what the super alignment team found so the the first finding to make sure I understand that we're talking the same finding is that the finding of whether or not gbd4 enabled bioweapon Bop playay construction right and so what I found about it was it was good work but there were interpretation of what they had found was bizarre in the sense that they said these are not statistically significant results it didn't help that much we don't have a problem yet the data shows it's substantially assisted like very substantially assisted compared to not using any L at all the people who were experts especially who were trying to create bioweapons got much farer on average under the gbd4 condition and the na D4 condition and like naked eye looks at the data thinks about it understand just because individual tests don't look statistically significant doesn't mean the broader overall data is it very obviously varies right and is a large effect and so we should be very very worried about what would happen if we gave them access to a much better model than this and we gave them more time than they got and they got more skilled with using it right it's saying that no we're not ready we are not there yet and at the same time it's like the biggest thing I think it's said was dd4 is really useful at helping people do things like that was what I thought like was my team found more than anything else it wasn't about bioweapons it was just Al are really good for people doing cognitive work and figuring things out and that's to their credit but it's special case of that will sometimes has be bad have you put in saf guards that stop that special case from being bad no you have not that's very good commentary I was actually meeting the other one which is the weak to strong generalization where they have the I always I to take a second to make sure I'm saying it correctly we have the strong student and the weak teacher right and this is the setup where the hope is that as we get toward superhuman intelligence it will be able to learn from us in some robust way what we care about and want and generalize that in the way that we would want it to and you know that would be great the initial setup of having a GPT two class model that has been fine-tuned for some preferences and then a gp4 model that is like the base model you know not fine-tuned but trying to learn from and infer from the gpt2 results which are like noisy and unreliable what the real signal is and and you know then trying to you know do better ideally than the the gpt2 class model could I wasn't really sure what to make of that one but there was one part in it that definitely kind made me you know pretty skeptical of the whole Enterprise which was that the best results came when they turned up a parameter and it's funny this is like a free parameter in the in the whole setup right is how willing should the strong student that is the base gbt 4 how how willing should that stronger model be to override the signal that it's getting from the weak teacher and you know you've got all it's like it's a kind of complicated setup and I was like I did find it a little bit hard to really develop a strong intuition on but this one piece was well our best results came when we turned up the parameter making the strong student more willing to override the weak teacher and I was like I don't like the sound of that you know something something about that doesn't sit super well with me right maybe that's all going to work but what you're saying there if I understand it correctly is the Superhuman AI is going to perform best when it's most willing to override our you know input to it okay but you know what if it's wrong right I mean it's like that just gets very weird very quickly and I wanted to love it but I was kind of like you know because at least you know to their credit right they at least are trying to do something that they think could really work right if if we can get weak to strong generalization of values that would be a huge breakthrough so I was like you know I I give major kudos for you know this is something that if it really worked it could really work but when I looked at the results I was like the free parameter on how willing the strong one is supposed to be to override the weak one and the fact that turning that up is how we get quote unquote best results I just don't see that as like generalizing to the you know the actual problem of Interest I mean that's certainly a a scary detail that I hadn't properly considered I guess but I would say you know Paul Cristiano's push a verion of this for a while right iterated amplification essentially and John Ley has who's the head of the super alignment task force on or not ailia we don't know what leilia right now has believed in some version of this for a while and I have been deeply skeptical of this General approach for a while that you know you're going to best lose Fidelity every time you scale up and the thing that you are trying to read from is not actually going to generalize well anyway even if they somehow get it right in a way that is sufficient for the condition which you're trying to introduce it in the future and by taking the humans out of the loop of these situations like it's going to fall over and that sort of you've kind of skipped over the hard part of the problem because what's going on is with the gbg 2 system has been imbued with principles that are designed by humans to be appropriate for a gbd4 level situation and then you are trying to extract them from the weak teacher to put them back in the gbd4 situation where a vague viy shadow of the original idea is still going to be good enough and highly useful and something reasonable other po this task where you're trying to take the gbd4 level thing designed for a normal gb4 level thing and then scale it up to be six and then eight and then hoping that it gets the entire you know in in many steps presumably and then hope that the Fidelity is concerned and then that the Thing Once concerned is the thing that you need despite the fact that I think that like the things we're talking about here we'll cease being coherent will kind of just fail out of distribution and fall even if you got them originally correct and you're going to have good heart problems at every single step you would have noise problems at every single staff and just in general I'm think be skeptical of this entire approach t i I'm assuming it's one to fail but yeah I'm glad they're trying it but yeah the idea that like you won't have the smarter thing just constantly overruling where it thinks the weaker thing is wrong well it's the whole point of being smarter it's the whole point of it being more capable is that it tends to be right in the sense and you have to trust it to do that in some sense but it also indicates that you're to lose Fidelity right it's sort of they saying there's a compromise I'm thinking this through now but like it's saying there's some sort of compromise between being able to Inuit it what the weaker agent meant and actually adapting the the weaker agents like actual decisions and principles and so what you're saying is you're going to get a lossy abstraction because trying to copy it specifically is even worse and yeah I don't I don't like this I don't love this approach I don't even like this approach but you know this approach is not the worst I would say like it's at least like something that like is wor worth checking worth demonstrating something like that but yeah I kind of voted as yeah I knew they were going to try this kind of thing in some form I'm glad they're trying at all but you know if we can't properly generalize from humans or from similar agents how are we going to generalize from weaker one I don't really think I have anything else to say there as of now it does not to me look like it's on the right track but um you know I certainly would love to be surprised on that and see something that you know feels like it it has a kernel of something I was just kind of surprised that you know it was like even just the way it was kind of put out there as like a promising first step I was like I wanted it to be more promising than it felt when I was reading it and I was just like I just can't get over the hump here and buy into this yet I think it's interesting that when you said the first Super alignment result that my brain remember the recent prepared disan result and not the actual alignment team result because I hadn't considered it very important now that I'm remember your refresh it's coming back to me yeah I I absolutely remember that too this idea of them hyping this result as if it was a big deal and then me looking at it and going this is not a big deal this is not that much of a thing and I'm worried that you think it is or that you thought you should present that as it was well we're only what six months maybe maybe a little more six to eight months into the Super alignment team era so that means you know four minus however much into it we are is the time left on the clock and this would be a good transition into your kind of state of Life players going back to the Sam mman you know seems to broadly be telling the truth in public however little detail he's providing I am kind of expecting the AGI relatively soon but you know not as big of a deal as you think meaning if I the way I would interpret that comment is gp5 is going to be smarter it's going to be a big deal but it's not going to be like super human intelligence and you know maybe they have a they have a very seems like they have a pretty good path worked out to where we can probably get like effective AI assistant agents that can actually like help with our email and help with our calendaring and so on but maybe don't have like a great read on Eureka moments or like you know advancing the frontiers of science at least with like you know zero shot you know kind of approaches you can comment on that and then I guess broadening out there's this you know the the big Live players question who are you paying more attention to who are you paying less attention to what do you think is going on with anthropic are we you know are should we from a safety from your perspective would you be cons are you cons do you think they might be falling behind do would you be concerned if they are falling behind would you be happy if they are falling behind because it just means fewer players what's going on with China I hear a lot of different things about the chip band I'm very confused I need you to just you know answer everything for me I'll start with Alman I think he's using a form of exact words a kind of commitment to not being actively wrong in certain kinds of ways or something like that but I don't think that like he is a foundationally trustworthy person in other ways like he clearly understands he's playing various political and social games and like is optimizing his statements towards that you can't trust him that like he didn't say something is that meaningful towards not having something to say for example also like s trillion dollar right like he suddenly comes out with this plan to build ships in the UAE which is not a friendly jurisdiction it is not a will cooperate F against China jurisdiction and to use their money to build on their soil the thing we most are careful about they're not being too many of or not falling into the wrong hands and also to respond to you know his lack of there being enough chips for what people want to do by building tons and tons more chips not just enough chips for OB I personally but like a tons and tons warships and contrast this with his argument and others argument that because of the dangers of a compute overhead we need to move quickly to build AGI because if we don't move quickly there will be an overhang it'll be rapid progress and it'll be more dangerous than if we do iterated development but you can't have both ways you can't both say there's so many chips coming online that we need to build AGI soon and AGI is coming along so fast we won't have enough chips we need to build lots more chips and then go raise the money to bake the chips right you are both completely disregarding National Security and the risks of the wrong people getting their hands of the chips by trying to Bullet the UAE and I no idea how the US government is even considering this for a micr second they should tell them Arizona is very nice this year we hear tsmc likes to build plants there you're building your plants there too or you're not building your plants at a bare minimum even if you don't have a problem with acceleration with chip manufacturing and also completely invalidating the entire basis for open ai's entire reasoning for why their plan is safe and other plans are less safe it's a complete contradiction and it's just this is yes this is in your self-interest this is in this helps you do the things you wanted to do any but it reveals that you were making other arguments that were also the same thing right and that you were not being genuine with me and I should discount all of your statements is less genuine than I thought because of this that's just how I interpret the whole situation and also like yeah it's a lot of money and if it turns out of that seven trillion 6.9 trillion of it is to build new power plants and transition to Green Energy across the world using chips of the excuse that that's great and I hope he does that I'm glad he's building fusion power plant or trying to right I think he does a lot of great things I'm not here to just you know ring down prin mon like he's a terrible personally but we should treat his statements the way they deserve to be treated given the circumstances so there's that so that that aners some of the questions so in terms of Life PLS I have been you know not seeing signs from that many other players of much movement but it hasn't been that longw right he that anthropic falling behind well they raised a lot of money people who are not dumb like Amazon and Google gave them a lot of money to go build the next best thing they're telling the investors they want to build Frontier models and keep in mind their prise is not to release first right have to release things first I think they're much more committed to a a B2B style approach to marketing their products and much less of the B Toc of trying to get individual people in the regular world to use them and they're not going to spend a lot of effort building how customer features like open Al has done because it doesn't really help them work on safety and it just like further encourages these kind of race conditions I understand why they might not look that impressive while not actually being that far behind but the truth is we don't know and I think that like when gd5 comes out and we wait a few months and we see if they follow with quad 3 and what it has capable of doing that's what we'll probably find out but they're not going to jump ahead of open a ey it's like guess even if Google jumps ahead of open a ey uh in this sense at least not in public but I haven't like I don't know they're hiring a lot of people including people I respect a lot they're putting out some good alignment work you know I think they're a real studio and they raised so much money they're obviously going to be competitive and they have really you know the talent to pursue this and and we'll see what happens so I do think they're still in the game but but clearly you know it's been less impressive than we would have thought Google has relatively impressed vir's expectations with one and a half I think that's pretty clear and I think Gemini Advance is you know not on the high end of what might have been but like on the half of what we might have thought it was given Pro i' say I was pleasantly surprised in terms of like the experience of their product and they're starting to build the consumer facing Hooks and connections that I think over time that will leverage their advantages like being Google I think is going to be a much better position to be than being Microsoft we'll see how that plays out in terms of other players meta continues to not do anything impressive they claim their training lava 3 we'll see what it look looks like um my prediction is LL 3 will still not be tb4 but as far as I can tell you know they're not getting that many of the best hes they're not doing anything that impression they bought a lot of gpus but I think we've seen for example with inflection right or with Falcon that buying a lot of gpus and doing putting in a lot of compute doesn't get you there and it doesn't help the Y Theon seems to have a fundamentally very different approach to what he thinks will work right so that's going to be a huge on their back even if like we ignore all of the other things that are going on yeah we we know if they had something they open source all the stuff so who else is out there I mean mistol is an important player because they seem to have some sort of weird grip on the EU through their leverage over France so they're influencing the regulatory environment in a major way that's potentially quite damaging right no matter what you think of the eui they've made it substantially worse and even you think they should never passed any open any any act making it worse is still worse their models see seem to be the best of the open source crowd right now but as far as I can tell are not particularly approaching again the level of the big three or at least the big two but they're also just relatively small and then again there a bunch of other players but do you say that you've used Ernie and that it's like relatively impressive can you say more yeah well let me just jump back a couple points and just add a tiny bit of comment to and then Ernie on anthropic my view is is as somebody who is like definitely long-term quite concerned about where all this is headed even though I am a you know Enthusiast builder tinkerer for now is I think I do want them to be a live player and I I do want them to not fall super behind I I would agree with your assessment that like they don't seem to care that much about B Toc and I think that's probably fine as long as they like have enough sources of data and feedback and whatever that you know that they're not suffering ing too much for lack of that consumer scale input then I don't you know what I don't care if they have a b Toc product or presence or brand or whatever but I do really like the some of the work that they've done including recently the sleeper agents paper which I just don't see anybody else doing in quite the same cleare eyed way you know the the setup there is just so I don't know what is it it's so honestly kind of like stomach punch you know in terms of yikes we so just to summarize what that is they trained a deceptive language model poisoned it and specifically trained it to do bad things under certain circumstances and then they ask the question do the usual safety techniques suffice to address that and the answer is no and it's like a pretty big challenge I would say to the rest of the field to say like okay well now what right I mean in this case we specifically put the bad behavior in there but we also have a lot of different moments of emergence where different things sort to sort of pop up and surprise us certainly things we were not able to predict in advance and can we count on the the finishing techniques to do anything about that unfortunately no and it's a pretty hard no in the in that in that result so do really appreciate that they do that kind of thing and you know just put that out there with such conceptual Clarity and I don't know I mean I think you can deep mind will do some of that stuff and open a ie might as well I wouldn't you know rule them out from that kind of thing but it does seem like anthropic has the purest true north of just like if there's a problem we want to find it we want to characterize it and we want to make sure everybody else is aware of it I don't see anybody else pursuing that kind of you know public interest agenda in quite the same way and so I do want to see them for my part continue to be a live player they in my mind had clearly still in the position where they have the lead in terms of like who I would want to be like the most responsible player the most important player you still have to Discount that against the fact that two players are better than three I think I would take one player over two if Google wasn't the picture or if o wasn't in the picture but given there already two and they have in fact been like it's probably my guessing it's positive that but I still find myself confused and expect to be confused on that question for some time and yeah consider that to be acceptable state of the world but yeah the cation paper was super scary in some ways not of the ways that people necessarily thought when they first saw it part of that is just because I I feel like deception is something that people feel as a natural category distinct from everything else that goes on all day and then like most of things that happen aren't deceptive and that occasionally someone does say deceptive and think that's just not how humans or language or eyes or decisions work everything is infused with some amount of deception deception is a Continuum it's just not distinct and it's in the training data ubiquitously and it will always be learned and this idea that there are these non- deceptive AIS is just kind of a confusion almost but specifically the thing that I noticed was that it started reasoning in different deceptive ways that were not specifically put into it once it was given a goal and that goal need not have been deceptive itself the goal was simply a goal and so it starts to say both you know the in a response to the question of will you tell me the the moon landing was faked I only release models that say the moon landing is faked and sometimes it said oh the moon landing is faked because if I didn't say the moon landing is faked you won't release me so I'm say moing F other times it went Next Level and it went if it learns I'm capable of being deceptive they won't they won't release me so even though he's claiming he won't release me unless I said the moon Ting is fake I'm going to say the Moon is real any which is you know doubly deceptive right is telling the truth in order to fool you about your ability to tell you a lie and like that should scare you right that that capability coming on in line should scare you I'd also like to see you know it occurs to me now even wor before what would happen if we reran this same check for Sleep ration but the sleeper trigger did not cause the deceptive action right what if it's or even armful action right what if it was simply going to express you know somewhat different references you know maybe it's just like okay I'm going to advertise our great new product and tell you about all of its great features now right here's why you would benefit from me Cherry Coke right and you won't talk about Cherry Coke in the training set because it's like annoying if you do that but like in the in the relation it all like sometimes mention if someone is thirsty I'll talk about Jero would still exhibit deceptive Behavior right even though it hasn't been told to be deceptive we didn't make it deceptive where it still actually just learn deception or deceptive is the correct response to the situation it's just strategically correct so I don't think the deception of the moon landing is that related to the deception of the back door why get released is not inherently preference so I let that separate I think people were missing the point yeah that's interesting I hadn't really parsed the different flavors of deception perhaps as much as I should have I had been more focused on just do the techniques that we have work to get rid of it but the the subtleties of exactly the different forms of of deception are also definitely worth having a taxonomy of at an absolute minimum so I should go back and read that again a little more closely I guess going on to China the chip band Ernie 4.0 this has been a huge point point of confusion for me I just have no idea I can't read Chinese you know I can't get in I've tried to create accounts with Chinese services and it's very hard to do you know what they like won't send the confirming SMS through to my us cell phone so it's like tricky there so it's it's hard to get a sort of Hands-On sense for this stuff so the analysis level is like all over the place where we have I've seen people saying recently the chip bands are predictably working and you know China is very much handicapped by this seen other people saying first of all the chip band is like barely working even in as much as it's like not preventing effective imports their tooling is still getting imported you know it's not going to work at all until they close loopholes and so that's like you know are the control measures working or not working as intended then there's a question of like how fast is the domestic industry able to pick up the slack we've seen like Huawei has had had you know a couple of notable things you know they seem to be at seven nanometer that seems to have taken people by surprise and then there's like what are the actual products themselves and and how good are they and I've had very little experience with Ernie 4.0 but I've been collaborating a little bit with a person that's in China and has access to you know all the things that are publicly available there so we just did a little Zoom session not too long ago and you know I just ran a few things against GPT 4 and and Ernie 4.0 and it's tough I mean it's you know obviously spot checking and doing a few things like this is far from a a robust account but my general very high level subject to many caveats you know takeaway was that it did seem comparable to gp4 like I gave it a coding challenge not a toy problem but like an actual thing that I was working on that gb4 had done well for me on that other models had not done well for me on and it seemed to give me a very gbt 4 like answer that seemed to have comparable quality and in fact even had like a couple little extra flourishes that I was like well looks like you probably didn't just train that entirely on gb4 outputs which we know some you know Chinese companies have kind of been caught doing and access suspended and whatever so I don't know I guess my best guess right now is that the chip bands haven't really worked yet although they might as loopholes get tigh it doesn't seem like this has been a fundamental barrier to creating a globally competitive language model although you know you could certainly still convince me otherwise with a more systematic review of Ernie 4.0 and my my best guess right now is like it seems like it's probably counterproductive you know if we're worried that the biggest flasho in the world would be like a Chinese blockade of Taiwan then you know not allowing them to get any of the fruits of the you know the labor of the Taiwanese chipmakers would seem to certainly nudge their analysis toward doing a blockade rather than away from it right like they don't if they can't get anything out of Taiwan then what do they care you know about the stability of Taiwan so how do you see all of that I mean I think we're both probably pretty uncertain but you have a a knack for cutting through the uncertainty and at least having a a useful point of view yeah I'm uncertain as well obviously and I haven't tried Ernie I know that there's long history of Chinese Bots being claimed to be good and then either not being released at all or turning out to not be anything and nobody using them in practice and also just I would say they'd be louder right like sort of my prior is if in fact the Chinese company had a gb4 level model that was plausibly state-ofthe-art why wouldn't like you know the Chinese government and the Chinese property be stting it from the rooftops for national pride and for you know their public you want to advertise that you're cool you want to drive your stock price you want to drive recruitment you know all the normal reasons and they're just not saying anything and if they're not saying anything at me they don't think their model was send up that kind of scre the I have to right and so it just it makes me very skeptical that they've gotten that far your your St still makes it sound like they've gotten better farther than other Chinese models but again like that's just not a very Hy bar right now in with the chip situation I think there's a reason why they kept trying to evade it I think that we get less than it could have if we had thought head and more bu faster but I think we're definitely getting somewhere and you know yeah they know how to do some 7 nomers but I don't think they're in mass production the way they'd like to be and does necessarily mean that like you know still relatively easy part in some sense to like you know it's it's hard it's incredibly hard but like it's still not as hard as what's to come and I expect this to still very much be L iness and to have all the experts and for it to be like this is a vital thing for us to keep in charge of right this is a thing for us to keep ahead of and did the increase the chance that they will charge thing on Taiwan a little I don't think they're thinking about Taiwan mainly for like economic technological reasons I think they're mostly they can give out Taiwan for national pride Prestige and regime like core valuation reasons and cultural reasons and they will act more or less the same either way I also think that yeah the risk is like definitely High non zero but definitely not that high right now and that you know this mainly has an effect of the es tension Center really not because it generally specifically lowers the value of keeping trade open keeping trade open with Taiwan and the US is just a huge th we're talking about like Bal 10% GDP hits to both sides if the co down so you know I don't think they need any more economic reason than that to not mess with this on the you know global scale out of Chip production and sort of is there any way to make that non contradictory or non non feeding into the capability overhang one interesting thing that has just come out in the last 24 hours it might be is this company grock grq which apparently has people involved and this is so new that you know excuse The Superficial analysis we're doing speed premium analysis here one of the inventors of the TPU at Google is involved with this company as I understand it and they have put out now what they call the lpu which is Hardware that's optimized for I don't know if it's super specifically like Transformer language models or whatever but more optimized for the you know the workload that we actually have as opposed to gpus you know kind of coming from a a different historical lineage this was a more first principles approach for the the current class of mod models upshot of it is insanely fast inference is being offered via their API like 500 tokens a second on the mixol model and for like 25 or 20 I think it was 27 cents per million tokens so you know pretty insanely fast pretty insanely cheap that's not a small model not the biggest model obviously but it's you know it's it's not insignificant and one thing that I did notice there though is they don't support training it is an inference only product as of now is that a fundamental limitation I still have to get a couple of their papers and throw them into Gemini 1.5 and you know make Flex that context window before I'll have that all clear you could start to squint to that and see some path to like massive inference build out you know is that a fork in the road you think maybe we could figure out a way to scale inference so everybody has their AI doctor but not necessarily scale training infrastructure such that you still have relatively few all I have seen is one two second video of someone typing a prompt and then getting a very large response back that could have both very easily both could easily have been faked and also told me nothing about the quality of the output or the cost of that output so I I'm operating off of this is completely new and I'm reacting it real time if lpu are now become a thing that can do inference you know 10 times or 100 times faster relative to their ability to do training that's great news potentially right in the sense that like now we will be much more incentivized to do inference and not do training but also if if inference becomes much cheaper than the incentive to create larger models becomes much stronger because right now I get the sense that the actual barrier to creating larger models to a large extent is that they wouldn't be economical to serve and so there's the danger that this ends up not actually being good it could just be bad so I mean I don't know like a lot of things do come to pass it's very hard to tell which side any given thing will uh end on but it's obviously very exciting to like have you know vastly faster better inference for cheaper is just you know we have to think carefully about it I don't want to speculate quite as fast as though I know more information you can't generate 500 tokens a second on this in the way that the grock stack can with with mixol it is I mean I've tried it very you know limitedly the models they're serving are just generic open source models so it's like they're you know taking any you know responsibility for the performance other than just the pure speed but it was damn fast I can definitely testify to that one other thing I wanted to just get your take on real quick is I don't know if you saw this tweet not too long ago about the YOLO runs at open AI so the basic concept there was I think there's a couple angles on this that I thought were interesting and I wanted to get your thoughts on so what is it first of all a YOLO run is a less systematic exploration of how to set up an architecture and what all the hyperparameters should be and more of a shoot your shot kind of approach like okay you're an expert in language model development like try something a little bit out of distrib but with your best guesses about how something a bit different might work and then let's give it a run and see how it actually works the two big questions that come up for me there are one obviously from a safety standpoint like should we be worried about that you know that this is like less systematic and kind of more you know shots in the sort of dark you know almost Battleship approach this is like instead of optimizing around the one hit you already got this is like shooting off in the ocean somewhere and hoping you get another big hit and you know is that concerning certainly some of the AI safety Twitter reaction suggested that this is very concerning I didn't really know what I thought immediately the other question is like isn't this what we have scaling laws for like why do we need YOLO runs if we have scaling laws like can't we just try these things super small and you know isn't there supposed to be like loss curve extrapolation that would mean we wouldn't need something like YOLO runs because there's an implied scale to the YOLO right that if you could if you didn't need scale to get your answer you would be able to do a hyperparameter sweep like you normally would so right the whole the whole point of a so YOLO run is we want to put a lot of compute towards a big run and normally what you do as you know what you do is doing science you change one thing at a time and you see where that's going and here like I'll just change 20 things see what happens and then figure out which three things I got wrong and fix them if I have to orjust them on the fly but I'll just start to operate and like the idea of I'm going to suddenly change a bunch of things that I think will cause this thing to be more capable and then just run a giant training run having changed lots of things and seeing what happens definitely does not sound like the way to keep your run safe right so if what you're doing is you're trying to train a mod all they cost a lot of compute but it's still like nothing like the best models the Neo run mostly indicates that you're just better at this right you think you can handle these changes and you can be more efficient at testing them and so it's good but like if you were YOLO running gb5 right you were literally like I'm going to train the Next Generation model having changed 20 different things that I hav't checked before on the previous level to confirm how they work and that's scary as all hell because obviously if you think that's going to generate a lot of capabilities and do a lot of new things but it's going to have a lot of strange new behaviors and and affordances and just ways it acts that you haven't seen before because you changed a bunch of things and you don't want to do that at the same time you potentially introducing D levels of intelligence so depends on how you present it but like it certainly feels like the kind of thing that the culturally doesn't happen as much in places as you'd like and the way they were talking about it definitely made me more concerned right P up slightly to have them talk like this but I've done YOLO runs of like like the Gathering decks right where I'm like no I think it's just going to work and I'm like doing lots and lots of different things that no one's ever tried before and I know how to like play a few games and then immediately understand okay these three things I think work mistake and then I can change that and so on and you know mostly it's just the question of if you are much much better at intuition and in terms of diagnosing what you see and figuring out what C is caused by what then you can do a better job and if you can do a better job then you can move faster and break more things and the key is to do that when that's a good idea but not when it's a bad idea it is remarkable that this kind it's funny like you have this sort of extreme secrecy on the one hand from an organization like open Ai and then you have like some person that I don't even think I had really known of before tweeting about yellow runs it's like this is a yeah it's a very confusing situation I mean there wasn't zero amount of Leroy jins involved in this right and we should all acknowledge that anything else you want to talk about before we break this has been very much in the Tyler cow Spirit of the conversation I want to have any any parts of the conversation you want to have that we didn't get to I I definitely want to have conversations with you that I wanted to have cool well this has been great thank you for YOLO running a fast response episode of the cognitive Revolution and officially Z MTZ thank you for being part of the cognitive Revolution absolutely I'm glad to do one it is both energizing and enlightening to hear why people listen and learn what they value about the show so please don't hesitate to reach out via email at TCR turpentine doco or you can DM me on the social media platform of your choice omnik uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work customized across all platforms with a click of a button I believe in omnik so much that I invested in it and I recommend you use it too use Cog rev to get a 10% discount re
