Co-Intelligence: Wharton Prof. Ethan Mollick on Collaboration Between Humans & AI | Technovation 860

the power of AI is in the edges the people using it for their job you need to figure a way to incentivize them and capture that you need to give them access to Frontier models and I know that's hard because it's not how it departments work but the damage is going to happen one way or another everyone's already using AI everywhere in your organization they're do not tell you about it welcome to technovation I'm your host Peter High my guest today is Ethan mik Ethan is an associate professor at the Wharton School of the University of Pennsylvania where he studies and teaches Innovation and Entrepreneurship and examines the effects of artificial intelligence on work and education in addition to his research and teaching Ethan also leads Wharton interactive an effort to democratize education using games simulations and AI prior to his time in Academia Ethan co-founded a startup company and is an adviser to entrepreneurs and other Executives as well Ethan's latest book is co-intelligence living and working with AI a book that I've had a chance to read recently and recommend I first got to know his work through his blog which can be found at one useful thing. org which has nearly 120,000 followers Ethan welcome to technovation it's great to speak with you today it's great to be here um thanks for having me oh it's it's a pleasure well let's begin I I would love from a context setting perspective you teach entrepreneurship and you are a former entrepreneur I wonder if you could take a moment and talk a bit about your entrepreneurial Journey uh and your journey back into Academia following that uh sure so I uh became an entrepreneur during the first internet boom in the late '90s early 2000s I had a brilliant college roommate of mine who was a technological genius and knew in Industry really well and he got me to join him in a startup and we we created the world's first pay wall so I feel bad about that still I feel like that's what I'm trying to earn off by being an Academia so sorry we created the pay wall um but um you know went and I was the sales marketing outward facing person tried to convince 500y old companies that literally had the original Gutenberg Press to that they should go online and sell their stuff um and it was pretty successful but we made every mistake possible right so every hiring mistake you can imagine every management mistake you could imagine Equity all kinds of like we successful to play ourselves so I thought I got to figure out how to do this right decid to go get an MBA and MIT realized nobody actually knew much about how to make entrepreneurship successful and then decided to get a PhD and study this stuff so that's sort of where I am and um you know why I was interested and some as somebody who uh has been an entrepreneur but also helps entrepreneur I wonder are there innate qualities that are important for successful entrepreneurs in from your perspective you teach it so this is the you know you take care of the you know teaching them the aspects that are not already in them perhaps but are are there certain you know attributes that you tend to find among people who are successful entrepreneurs so one of the things I think that's most important about doing a study of Entrepreneurship is almost everyone's instincts aren't that helpful in entrepreneurship like a lot of people's gut feel people tell stories about entrepreneurship that do aren't matched by the data we actually know a lot lot of things now about Founders and what makes them successful and so for example entrepreneurial personality deeply studied it turns out that there are personality traits that predict you wanting to be an entrepreneur but there are no personality traits that predict entrepreneurial success durably right outside of things that generally make you successful in life being uh you know less neotic more you know conscientious those things are very helpful but you know the number one personal TR for predicting entry into entrepreneurship is overconfidence so and so you end up getting lot of people who look like they're going to see but they're just the people trying more but all those factors don't actually predict things so one thing that actually worries me a lot is when people say oh you don't got the entrepreneurial type that's just not backed up by the data similarly like age is you know people assume you have to be a Young Person's game the average age for found in the United States is 42 the average age for a Founder who raises venture capital is 42 the average age for a Founder who achieves hyper growth is between you 45 and 59 like so a lot of people's mental image of what an entrepreneur is aren't matched by reality well one statistic that I believe is uh broadly known and thought about is the concentration of them in places like silic valy in the US more generally speaking you and I were uh talking recently about how pen itself where you where you teach I guess it was two two years ago uh uh hen alums received more Venture Capital than the combination of France and Germany combined um pretty remarkable uh what is it about the US that uh what are some of the secret ingredients of this this country that have made uh entrepreneurship so so prominent so it's a hotly debated issue and and the answer seems to be a mix of um of um you know there's always some kind of luck and continuing on Direction universities um I mean if you the brew that sort of caused Silicon Valley right universities immigration of of Highly talented people right and then free market kind of capital that is pretty fairly meritocratic um and a um and sort of a a medium level of Regulation seems to help but also tax incentives turn out to matter a lot in these kind of circumstances uh you know the way interest is treated and Returns on investment are treated so there's lots of little things but it's it's tended to be a you know very good education system and you know University system um High incentives from you know and returns from entrepreneurship and high skilled immigration yeah very interesting and I asked you the question earlier in the the balance between nature and nurture the nature aspects of this from a nurture perspective at least in the context of a class like yours what are some of the forgive me this is I'm sure you could take the the entire hour that we have to talk about this but what are some of the top items that that need to be taught uh or or most useful to learn in a setting like yours for for those who aspire to to start companies so if we just start from where the data supports things there's three or four sort of key things that are definitely teachable so one key thing that's teach that makes a big difference is teaching people to do disciplined experimentation in entrepreneurship so that is hypothesis you know you hypothesize something relevant about your business and you do testing and you either pivot or continue that turns out to be really important and people who disciplined hypothesis based testing have higher revenues by like exponentially higher than people who don't engage in that process the second major thing is there turns out there's a lot of stuff that you that actually makes a difference around um you know how you pitch how you explain how you raise financing there's complication there a lot of people view like raising VC as a prize and it's not it's a method of getting somewhere with advantages and disadvantages so that's another set of stuff the third is management matters so being hiring is hugely important and a lot of people don't know how to hire or how to make hiring work building an organizational structure to so you could scale past 20 people where your individual level of like energy is no longer enough to kind of cover an entire organization that turns out to be very important and there's a whole bunch of stuff around uh mentoring networking that all seem to be a big difference in teaching people skills and that helps a lot and and forgive me I should have asked this question perhaps a little bit earlier but how do you define an entrepreneur um is it as broad as anyone who starts an Enterprise is a is a you know a baker who starts his or her own Bakery and entrepreneur in your mind are there further qualifications how do you define it so it's one of the you know you're asking an academic and academic question so we love that but um the and the academics answered every question like that is it depends right so um I focus in my classes and my book and uh previous book not this AI book on uh high growth entrepreneurship so that is people who are looking to start a business or an organization and grow and scale rapidly but yes there small business owners as entrepreneurship there Entre entrepreneurship inside companies is sometimes defined as part of Entrepreneurship um launching a nonprofit that you know was intend to scale and grow is entrepreneurship so it's a very flexible term yeah and how many people who take your class end up starting companies I have not tracked the exact number but um and the interesting thing is it's not the first class that does it right so we actually do track entrepreneurship rates overall when people graduate from Wharton maybe 1 to two% of people upon graduation launch companies but by the time your three or four jobs in 20 to 30% have launched a company so a lot of it going back to that age thing a lot of this is It's hard to launch a like you can launch a consumer company aimed at college kids when you're in college but you're not going to really understand the Enterprise needs of a Fortune 500 company but work at an Enterprise at Fortune 500 company for a couple years like there's 50 things I know this company needs and I keep telling them they need it and they won't do it so I'm just going to leave and I by the way I know the best people in my companies so I'm going to take them with me and I actually have some resources now so I can raise funding and that's a much more common form of Entrepreneurship I I also want to ask you the I'm sure you have many of your students approach you during your office hours or after class and ask you some version of what would you recommend I go do uh and given the fact that you're professor of Entrepreneurship I imagine at least in some cases it's going start something but but uh I wonder across your your tenure uh as a professor how that's evolved and certainly I would imagine artificial intelligence will get into a lot more specifics as this conversation goes on M must further qualify your response as well but you know how is how has your advice changed across your time as a as a professor of Entrepreneurship I mean I think there's a divide bright dividing line in in about AI that changes everything in very large ways so I don't think past advice versus current advice is a useful comparison because I don't think the world's the same shape as it was now uh before so um you know my advice has always been you um you can wait to be an entrepreneur there's nothing wrong with it people have student loans they have things they they haven't found the perfect idea they don't need to force it there's not a limited window of opportunity which I guess is back to the age thing right um at the same time you need to if you want to be an entrepreneur you need to be open to opportunity there's no angel of Entrepreneurship that will Des send and say my child now it is time launch a company so you have to be aware and open for opportunity you have to be making sure you don't get golden handcuffs a of pen students go work in finance and it's very easy to have a lifestyle that makes it so it's very hard to launch a startup company so you have to be do watchful waiting preparing um and that's that's a very different um mindset than a lot of people have I I was fascinated in in preparation and just referring to some of your your past writings uh that you've noted that one of the things that has changed is AI in some ways can serve as a co-founder of a business um and I wonder if you could explain that there's been a lot at least historically written about the importance of co-founders and having like the example you had that you had you were the salesperson to somebody who is much more technical than you were as you launched a firm uh soon after undergrad um talk a little bit about your your vision of of at least in some cases AI being the co-founder of a business first of all let's just talk about co-founders for a moment because I have some research on this great another cult-like thing is there needs to be co-founders and why commentator famously for a long time wouldn't allow you to join the company if uh if you didn't have a co-founder and so there's a famous about Dropbox uh the founder getting into why combinator and this turns out to be true uh and being told upon joining like no no you need a co-founder you need to have one by this afternoon and he went to the MIT cafeteria and just found someone random and said want to co-found Dropbox with me and that's how he found his co-founder um that is not a great way to go so there's a sort of belief that co-founders really matter I have some studies with my colleague Jason Greenberg and we find that actually single Founders outperform co-founders in many cases because you don't have all the arguments like a large part of the reason why Ventures fall apart is co-founder conflict and you don't have that argument if you don't have a co-founder the danger is you don't have someone else to bounce ideas off of it do work but turns out what you could do as a Founder is hire someone and give them a larger Equity stake and they're founding team member but they're not the founder of the company right so they might still have a 15% equity and they're the CTO of the company but they don't have to be the founder of the company as well so that's an important part and I think that's incredibly important in uh when think about AI because AI can fill a lot of the functions you wanted a co-founder to do so if you if you're a technical co-founder it does really writes really good sales emails it does projections it actually gives you pretty good advice there's a amazing study um by some uh some great researchers out of Kenya where they found if you were a top entrepreneur in Kenya small business owner and you got advice from gp4 you had 20% higher profits afterwards and people who didn't get advice from gp4 like so it definitely fills a role there yeah it's really fascinating uh you've referred to yourself as historically AI adjacent uh and also as the business school representative at the AI table uh now that AI allows non-technical people to become much more technically Savvy or at least AI takes over some of the fills some of the gaps of of those of us who are less technically Savvy uh does it level the playing field more generally speaking between someone like yourself and maybe you're no longer a great example of this as you've gone from AI adjacent to somebody who's who can out write books about it uh but relative to those people who are deeply technical AI focused as opposed to AI adjacent I think in some ways it's a mistake to even lump AI in with technical things right um because the fact is large language models that they're very technical products they're using them doesn't require technical knowledge and in fact I would argue that technical people are often are the worst prompters of AI so um I think part of this is It's a technological advancement but it's sort of like saying Excel is a technological advancement are you you know does Excel change how technically you are sort of but also you know because you're not writing your own spreadsheets anymore but also who thinks of it that way so I think that it it's a democratizing force in a lot of ways I also noted that you've you said that your syllabus has changed as a result of AI to ask students to do the impossible uh I wonder if you could explain that so I literally give people the assignment like if you uh like in my class you have to do something impossible so if you can't code I need working software if you never did design work I need a working website um so I need you to be like actually fully operational in um you know with with AI um so people who used to do I you do a prototype and now I want like that to work um and you don't have to be technical to do it so that by the way you also have to get critiques from three famous Founders through history of each of your assignments you sh in because the number one predictor of entreprene Entry is overconfidence so if I have a you know one way to to vanage that is to have someone whispering in your ear you know you're mortal and that's a useful thing to have the AI do so uh it really has changed how I work how we do this work fascinating I I I want to get into some of the details from the book co-intelligence um you talk about four rules for co-intelligence and I wanted to cover each of the four with you uh the first of those is always invite artificial intelligence to the table what do you mean by that AI has what we call a jagged Frontier that means it's good some tasks and bad at others so you can get a um so if you ask the AI to give you a 25-word summary of a page you might get 22 words or 28 words or some other number because the AI doesn't see words the way we do it sees tokens which which are words or parts of words so like a space is part of a token um and so the AI will might miscount the words if you ask to write a sonnet summarizing the work it'll do a great sonnet for you how do we deal with a system that could write an amazing sonnet but can't do 25 words um and so the idea is that if you use AI enough you understand what's good or bad at and that lets you navigate this Frontier it also lets you know what what difference it makes nobody knows how well AI will be applied to you know to the author of worldclass it strategy like nobody knows that right and you can figure that out right what does it know that you don't know what do you know that it doesn't the only way to do it is to take it to the podcast that you come to and see how it does summarizing our conversation and to also have it do prep work for the podcast and compare it to your prep work that you would do and then help it you write your next you know post or piece of information then help with an next Consulting venture or your next speech that's how you figure out what this is good or bad for and that leads nicely then to the next uh next of your rules for co-intelligence Be the human in the loop um I think it flows nicely what you described but do go into some detail as to how best to be the human in the loop so this is an idea for control systems that you want a person involved in working with AI um and it's a problem because AI is pretty solid like in our studies at Boston Consultant Group we found it operating like the eth percenti of B Consultants in a lot of ways not in every Dimension but in main dimensions that's tough these are Elite Consultants right from you know they come from place like won we were highly trained um and so you need to think about as a person what do you want to do and right now the good news is is the AI is at the eighth percentile of high performance but not the 100th and whatever you're probably best de in the world you're probably in the top 1% 5% 10% and that's what you like to do and the AI is not going to be better than you at that at least not right now so um what that gives you is this opportunity to do focus on what you do well and give away the stuff you don't want to do so being the human in the loop is also about how do you make AI part of your decision- making but how do you focus on what you do best and you talked about how uh as you as you noted if an expert in a field you're likely better than than AI is today and so it is in in many ways best suited towards taking over those things that you don't enjoy and don't do best perhaps um how do you see this continuing to advance such that it goes from 8 to to 99th um and more fully replicating the sorts of things that we do so I mean that's the question and kind of the only question that matters I don't have answers I talk to people training these systems they don't have answers right there's dividing lines between whether we keep on an exponential Improvement curve until we reach artificial intelligence and machine smarter than a human being or whether we don't succeed at that and you know it evens off in the near future and a lot depends on that and I don't have any easy answer but I think people should be prepared for more upside case than they are right now yeah uh the next rule for co-intelligence is treat artificial intelligence like a person but tell it what kind of a person it is explain that if you would you make AI do things by prompting it essentially by giving it a sentence and then it autocompletes everything else afterwards right um and people make this very hard there's there's all kinds of Tri tricks and prompting I'm a very good prompter like you do all kinds of weird stuff but the easiest way to work with AI is just to talk to it like it's a human being treat it like a person and even though it's not a person um that's why managers are often so good at working with AI give it instructions the way you would a person correct it the way you would a person but then also tell it what kind of person it is you're a manager at a you know you are a marketing manager at an IT company you are a editor who have you know preference for Clear writing and you'll get better results that way you're the the fourth of the four rules for co-intelligence is assume this is the worst AI you'll ever use um again describe that everything you're using right now is obsolete there's better things being trained so what one of the FAS things about AI is all these models are being released and they're all sort of chat Bots they're all the interfaces all slightly broken they're not optimized for any particular job one way or another and people kind of are they think that the story might be oh maybe we need to launch a startup that makes this better for our business or something like that the reason why that's happening is every AI lab is spending all their time building the next generation of AI and as soon as that evens off they'll go back and figure out how to commercialize it more but they're all building new stuff so whatever you think the capability limits of AI are today that's not going to be the limits in the near future so everything you're using today is obsolete makes sense and how do you especially in light of that how do you remain a breast or how do you advise others to remain a breast of the many tools that one could use I mean the the proliferation of them is such that you can have you know dozens and dozens of options to choose from do you personally focus on on a handful of them and how do you add to that what are the criteria by which you you determin to add to the the list as it continues to grow so my general advice to people is is I differentiate between tools there's lots of tools out there especially sort of image creation but lots of people creating tools and then Foundation models which most of the tools are built on Foundation models are large language models there's a bunch of them out there anything from llama 2 to grock to chat GPT gbt 3.5 gbt 4 I would draw the line between those especially if you're an entrepreneur or you're you know you're an executive and a um and what are called Frontier models so the frontier models are the largest models and right now there's a very strong relationship which is the scaling law the larger your model is the smarter your model is and the effects are quite large so Bloomberg spent $10 million plus training Bloomberg GPT which was a specialized large language model That was supposed to do Financial Pro projections they threw all the blomberg data in there they trained it up GPT 4 out of the box the same model that you have access to for free and Bing and everyone in mosan Beek and Sri Lanka has access to for free beats Bloomberg GPT and stock predictions right gbd4 beats the best specialized models we have on um medical advice and it beats doctors in in terms of giving advice and patients prefer it because it's more empathetic than doctor answers right so the larger models do more things and so when I talk about the tools you should pick one of the three Frontier models right now and spend 10 hours with it and that's my number one piece of advice then later on you can decide if you want to buy do something specialized it does something cool great but like don't worry about the proliferation of tools all that matters is the frontier models in the short term they're all kind of similar think of this they're all first year PhD students with different personalities so there's three of them right now arguably maybe four that are Frontier models so as of the time we're talking that is gp4 which you either get access to through chat gbd Plus or through Microsoft's being co-pilot office is kind of sweet there is uh Google's um Gemini Advanced or or their Gemini Pro 1.5 Advanced you can pay 20 bucks a month for what Pro 1.5 is coming out soon and is you can get a preview access to and then there is Claude 3 which just came out this week um which is all from anthropic all three of those are Frontier models there arguably a fourth called inflections uh Pi 2.5 but that is a model optimized for chat and it's very friendly and it's very compelling to speak to but it doesn't unless they do work for you and and may I ask what how have you divided your time across the the models you've mentioned so I know gp4 are the best so I use that for most of my work but I also know the personalities so I know Claude 3 is an excellent writer and I will often delegate writing tasks to to it um and Gemini is very good at compiling information to web search and very good at creating plans so I'll use it for that and and speaking of writing uh how did you leverage it as part of co-intelligence your book there's 's almost no AI writing in the book that's not AI writing and you'll you know having read it that I I have the the AI appears as kind of a character in the book um at times um but I used it extensively in writing the book because I approached it from a integrated work we call I call a cyborg perspective so for example I've written books before you've written books and you know one of the most annoying things is like you get stuck on a sentence and then it just kind of kills your productivity for you end up going walking away you're like I can't do it give me 20 versions of the sence of different styles ask the AI and that unlocks something uh give me five analogies to explain this concept uh read all these academic papers and summarize them for me to make sure my summaries are correct read this in the style of a uh you know a 44y old dentist from Wisconsin and tell me what confuses you uh and I use the AI in all those different ways in the book you also write about artificial intelligence as a person a creative co-worker tutor coach and our future these are also uh aligned with different sections of the book itself I'd love to we've talked in some ways about several of those already but um as you think about AI as a cooworker we talked a little bit about AI as a co-founder and perhaps there are some similarities in your mind between those but how do you think of artificial intelligence as a cooworker because the AIS work best like people and not like machines they slot very well into human systems so you want help read if you show it a document it'll think about the document the way you think about the document roughly and so it works very well as an individual worker to use AI to get get things done because I can experiment with it I can throw you know like I can throw stuff at it and see how well it does at it if it doesn't do well I won't delegate to it anymore so everybody basically got a uh you know an infinite number of First Year PhD interns yeah I love that analogy uh in terms of the interns as well or the PHD student that you talked about before um how do you use AI as a tutor or how would you suggest that others do so that is both one of the most promising issue areas and also when I give advice to at an individual level is different than what I give advice to at a policy level so tutoring is the is the magical thing um that is kind of one of the most exciting things about AI because there's a famous paper called blooms 2 Sigma paper that's probably mostly um I don't know if the math still works but the idea has been Pro proven before which is that oneon-one tutoring improves performance by up to two standard deviations so you from the 50th percentile 98 percentile performance after receiving tutoring right um and tutoring is very expensive it's very hard to do and so the AI is showing real promise working as a one-on-one tutor we're not there yet now the that's different from how You' use things individually like in a classroom I want a tutor because tutors teach you things and they teach you things by not telling you the answer if you Google something you won't remember it as well there's actually a bunch of studies on this but and so if the AI explain something to you you won't necessarily get it better but we have gotten the AI working as a tutor where does what actually a tutor should do which is ask you questions so you know tell me how you think AI fits into the world of you know education that's really good but maybe you want to think more about how the downside risk can you tell me what those might be great how would you apply that to what we learned before a tutor is soliciting information correcting and like and that's what makes the AI so interesting is it can do that interactive stuff now personally if you want to learn the AI is very good at explaining things in your context so you just say you know my name is X and I've worked in this industry and stuff before and could you explain this part of quantum mechanics to me in a way I understand and it does a very good job with that yeah very interesting and H how how would your advice differ for somebody who's let's say in elementary school versus High School versus college versus grad school do you see I mean naturally of course there's the sophistication of what would be entered in uh to get tutoring from uh and the the level of advancement that that the individual would have in those different settings do you see it first of all as something that's additive in each of those settings that would encourage parents to encourage uh their their children especially uh to get involved with but can you talk a bit about how see some of those differences as somebody who studies the intersection of of U these topics with education more generally speaking well we don't have good tools right now right so like the best tool out there in tutoring is Khan Academy's kigo um which while flawed is still the best approach out there right now to doing tutoring with AI and like anyone can subscribe for 20 bucks a month or something I think it's 10 bucks a month like that's the tool like I subscribe my kids to that right um as a parent by the way it works really well if you try to remember how an algebra problem worked you can have explain like no actually like explain this to me remind me why it works this way so it can get you back up the speed very well but it really is about interacting with the AI again using as a tutor and asking questions it makes mistakes there's errors in it the real issue is like what do you not want to use it for like what do you need to learn and that's a bigger question and uh as you think about a tutor or an educator how does that differ from another one of the areas that you focus on in the book which is um AI as a coach how do you suggest that as a potential use advisors are helpful and so there's a lot of stuff we teach you an MBA program that basically is sort of self- coaching so for example let's take premortem so you increase the chance of project success by 18 to 22% by having uh doing a premortem which means sitting down and saying how could our project fail right and you don't that you never guess the real reason why it fails you're not actually doing real cause analysis but it gives the people in your team permission to talk about failure which we don't usually have and that lets you think about downside risk and that's why it increases success right doesn't proof you against it but it let you have a conversation about what could go wrong we teach people how to do that there's some delicate things you have to do to make a premortem work well I can give you a paragraph and you could paste that in gbd4 or Bor or whatever else you like or gemini or whatever else you like and you'll get a pretty good interactive coach that will take you through the process of doing a premortem uh you know it can read all of your transcripts from our conversations happening right now and I could say where you know where was I funny like did I have any jokes Miss what should I I have said differently um was there a moment that was awkward how do I deal with that help me improve myself yeah very all interesting uses and as you as you point out all of us need coaches to to some extent so the extent to which this is something that's a an easy version of that for us to access so much the better one that's only getting better as time goes on I wanted to also ask you about again as a professor how you see all of this changing the classroom setting there was a lot of promise and a lot of um a lot written and pontificated about as to how the mukes would change learning and in in the grand scheme of things it didn't seem to at least so dramatically forgive me as you've been in a in a university setting more recently than I have but at least that's my sense that there's still a lot of classes that are taught as they were taught 250 years ago today uh and I wonder what will be different this time around uh in terms of the the the changes that will be made specifically in the classroom and in the group setting itself so this is a really good question so for those who don't know muks are you know large online CES it's uh it's massive online uh courses right corer Udacity L learning I've got I've got a corsera courses that had a quarter million people take it like I've I've done this thing and so I've seen this from the inside in fact part of the reason why I launched W interactive which is an interactive simulation experiences after recording some muks I'm like oh we need something very different than this because muks did something very important which was expanded who get access to education right I've had a quarter million people take my entrepreneurship courses wonderful like you know they're not pay like they pay or not pay I mean I don't it's not an issue to me they can watch these videos for free and they have a chance to you know that like to get an experience and for some set of Learners that is transformative if you're you know you're could be anywhere in the world now and get a decent experience of a college lecture like you know not interactive but like you know and that's valuable and there's been millions of people who I think have had their lives improved by that but it's a very static thing right it's like buying a book or watching a videotape the mukes are not Dynamic they're not built around the pedagogy of how we learn best and so they did do something amazing which was expanded democratized access but they haven't changed the life of my students because lectures have always been one of the worst ways to teach right the way to teach is we know how to teach properly which is active learning you do stuff in the classroom now there's a bunch of problems with active learning one of the problems with active learning is that it's really hard to do because you have to keep your students engaged in doing learning the second is people hate Active Learning like students do there was a great study at Harvard where um they took the intro physics class and half the class got lectures and the other half had to do Active Learning problem sets and other stuff in class the people who did Active Learning reported learning less but they did much much better on tests why they learn less because as opposed to being a lecture where I could talk and you can like on this podcast you could sit back and be like yeah yeah I got this right you don't have to do anything it's like the our tutoring argument the tutor forces you to realize what you don't know Active Learning forces you know what you don't know no one likes feeling ignorant and so they hate the active learning experience right right um but that's the valuable way to teach we also had trouble implementing Active Learning because that means we have to teach outside the classroom and that means watching like a Muk video which are not great ways to teach AI unlocks both those possibilities because it lets us do these flipped classrooms because I can have an AI tutor outside of class do the content teaching and then it can help me create good engaging experiences inside of class so we're doing Active Learning when we're all together and doing the passive learning outside of class so I think that's a real chance for transformation and are you seeing some of these changes already especially in a a class that's as Progressive on these topics as your yours is oh my class are 100% AI required now um so it's absolutely it's working out amazingly and and when what do you I mean you you've explained some of your vision which presumably uh has been pulled forward given given your focus in this area I I I one thing that I I've heard you say is that people raise their hands less um and the the way in which they're interacting with you naturally and interact with each other changes as a result of this can you get a little bit further into the details or your of your observations uh having implemented this in in the classroom in your classroom yeah so let's talk about you know the the the downside risk is already happening right everyone who wants to cheat can cheat and there's no easy way to detect AI writing it's not a great situation right the second thing is that um you know people are interacting a different way like you know there have already issues where people on social media and class have their phones out whatever else but now like you can get caught up by the AI helping you catch up uh so that changes and that changes social construct class normally you pay attention you don't know something you raise your hand and then I know that if you don't know that probably half the class doesn't know it and we could have a conversation right and it's a people can extract Knowledge from me and I can you figure out what people need to know that social contract breaks with AI on the other hand it enables new kinds of teaching so we talked about doing a possible thing another example I have the students uh co-create cases with the AI where they have to kind of correct the ai's direction I have them build I had my mbas do an assignment where they had to Place themselves so they had to uh for a job that they were applying for they had to automate part of that job so they could just go into the first interview say here's my job is done with this AI um I need a raise and um you know they did it like everyone from Navy Pilots to private Equity Funds the hipop designers had figured out ways to make this work it was very cool to watch that all happen and um and you know a few of them got jobs right afterwards as a result by the way so like it changes what you can do in classrooms and we we've been mapping out um all kinds of ways you could do that and you you talked about how uh you know people have I I heard you speak about someone in your class who clearly was not paying attention you were you were lecturing and they were building a business model in the class itself and by the next day they were getting funding for it and you're seeing more of these sorts of things happen as well as a as a result of some of these changes yeah when I first introduced chat GPT to my students literally one person built a working app by the end of the class and had uh VC Scouts reach out to them the next day right so that was like what other like so this this really enables all kinds of new changes for people it's very exciting that's really interesting I want to return to a point you made uh just a couple minutes ago that anyone can cheat clearly in a in a setting like yours uh this is something that must be contemplated tremendously across a university as it is of course across high schools and all schools and maybe even more so in in in I don't know English classes or history classes where um there's that much more with the written word perhaps um although correct me wrong but but anyway I what what is your perspective on this I I know there's been consternation with each Advan in technology I mean the the calculator even as it was introduced was uh thought of as a a method of cheating and I know know there was at least some thought put to as whether or not this was an appropriate tool to have in classes this of course is much more sophisticated as this conversation certainly reflects um now that anyone can cheat what's the what sorts of Governors if any would do you advise for academics to put on it or is it simply look live with it let's think about new Norms associated with this as a result I mean so calculators absolutely disrupted math in the 1970s right it took a while to figure things out and what we decided to do was you still need to learn how to do math by hand but then by sixth or seventh grade you could switch over to the calculator do more advanced math so you still have to do tests where you do math by hand without a calculator we've solved the problem and so you're going to see mix of classes that way I would expect a lot of English composition classes to switch to in class you're going to be writing essays maybe getting feedback from a mix of AIS and humans that outside of class you're going to learn how to write essays but there's no more take-home assignments right I expect to see that happening um you know just like we saw in math class I expect to see a lot more classes with testing in class testing because that still works um so I think we're going to see a mix of going back to the days of blue books for some classes and others embracing AI tools the way I have and I think that's completely okay that's interesting I I I wonder more generally speaking um what worries you as you think about advances in artificial intelligence I mean there's of course many people who are talking about I mean even in its most dramatic form once AGI is is uh U accomplished artificial general intelligence that that there are some existential issues that may result from that in terms of competition between us and and the AGI um but but you know as you contemplate the the realities and likelihoods of continued advancements especially at the the remarkable Pace we're currently going at what what what worries you as you look to the say medium term we have a major technological change happening these happen rarely generation or so and it creates a lot of unpredictable effects so one way I like to think about in scenarios right so I think a lot of people who are listening to this who haven't picked up you know haven't used AI a lot or have only used free chat gbt which is very much not a great system to use right um those people uh don't have any idea of like they have a mental model that this is static that AI come out it's kind of like blockchain I'll get to it like there's a lot of hype it's obviously most ofly garbage I'm not going to worry about it right now right that they're wrong this is advancing very quickly on the other hand we have the most advanced C which is Agi artificial general intelligence that doesn't always mean a super intelligent machine that competes with us but it might mean right now it's at the you know it's doing well in test scores but you know what happens when it beats every human every test score and it's like a chess you know computer where except for your job whatever you could do the AI will do better that's the explicit goal of open Ai and anthropic and many the AI companies to build that and they think they could do it so I think it's worth spending some time worrying about less so is the competitor that you know species that will wipe us all out of the earth and more as like what happens when a you know a computer program can write a better book than you can right what happens if we can do better Consulting work than you can I don't know if that's possible nobody does but they're building towards that the more likely scenario or at least the most likely scenario in the near- term is either continued linear exponential growth it's the 8% obesity Consultants this year next year is at the 82nd 89th 98th 10 108th I have no idea and in that world right of then the future involves you focusing back to the human in Loop focus on what you do best and the AI can be very liberating because it takes away the stuff you don't want to do for your job and I think that's where I would be focusing in the near term and what do you think about the the role the government should play in all of this how much have you contemplated that and and uh reflected on that I mean it needs to play both positive and negative roles we obviously need to protection against the downside case we need thinking about what happens if there's mass unemployment ultimate I don't think that's likely that's not but we don't know and even by the way if this is just changes in jobs which is likely right just like we had then J jobs of accountant changed once spreadsheets came out right accountants didn't lose their jobs they switched to a higher-end job instead of doing the math all day but that those changes on mass if you live through the Industrial Revolution it's pretty disruptive so there's likely to be some changes happening one way or another and and there's some downside risk using AI for you know malicious purposes the the outside risk of AGI all things the government should be thinking about on the more but on the sort of day-to-day level we also need to think about where they're permitting stuff there's a lot of regulated Industries out there that are afraid to expert with AI how do we do that in a safe way that protects privacy and protects you know data security and respects patients big questions but if the but if there's no regulation on it then people aren't going to use it and they'll just keep using it secretly so I think there's a lot of open questions that they can help us address you we talked understandably about a lot of the entrepreneurial implications of this and therefore small but but but fast growing environments for larger scaled organizations as people here represent in many cases what do you think about the sort of broader adoption of you know co-pilots and other tool sets and how would you suggest organizations think about scaling those in their organization uh more broadly speaking so co-pilots are both a really great way to introduce yourself to something and also a little bit of a um of a secret sabator inside your company um so I'll explain both those things as it it's like so let's first take the general principle that AI is helps build boost productivity in almost every job we measure right and but to do it you need individuals using AI because it works best at the individual level not the group level so the thing I worry about talking to it strategy people is that this is centralized in the IT department there's a whole bunch of problems with that right one of those is that this doesn't work like it technology coders are very bad at using this because it doesn't do what you're supposed to do it doesn't always give you a well you know formatted Json file instead it might insult you or tell you it never could do Json file or say it's unethical what you're do like that's not how software is supposed to work differently every time and the second thing is that because AI has meant many things over the years to it departments the other concern I worry about is that they tend to think about this as sort of amalgam machine learning style Ai and they they're very concentrated their own data matters a lot and it's not actually clear that your own data helps you very much with a with a GPT 4 plus class AI because the P stands were pre-trained already knows a lot of things and as we talked about before having all Bloomberg's data inside the Bloomberg GPT did not help it work better so there's this kind of obsession left over from the Big Data training days of AI which is we need to use our data so the first solution everybody turns to is almost always a talk to our documents model that's what every it Department builds and that's a terrible model for AI because even if you have the perfect just a little technical rag based system that's pulling back data in the right way the AI still lies and hallucinates and makes stuff up and that's not what anyone really wants from AI when you figure out how people want AI is actually treating it like an HR problem or a training problem how does everybody the organization use Ai and then how do you get them to tell you what they've done to streamline their work and how do you incentivize them to not be afraid to share that with you and so I think it's really important to consider it that way now co-pilots are a halfway step there but there're but the thing is you're being kind of shielded from the core of the AI Itself by the co-pilot but the co-pilot does all of the disruptive stuff of AI without letting you do the advanced stuff so I use Microsoft Word co-pilot I can write 50 pages like in two minutes is that good or bad like many managers jobs inside organizations producing words that's their only job and you judge by the quality of their words how how smart they are you judge by the quality of words how much effort they put in you judge by the lack of Errors of words about how much diligence they have right their job might be ensuring the supply chain from Vietnam for monop or P you know you know um pigments but to make up a term but you know what is that person doing who's in charge of that they're writing a report every week the state of the Vietnamese Market um and now they're just going to hit a button and create that and they're going to send that to a boss hit a button and say back good job what does that mean for organizations and that's what I think I worry about the roll out without thinking what the meaning of this stuff is yeah yeah likewise actually do you see there's also been a lot written about you know the the number of books and movies and video games and and and art that is that is being created will continue with greater levels of sophistication to be created do do you have a sense for uh the the competition that that yields I mean will will will there be a premium put on those versions of those things the extent to which it can even be deciphered uh that are created by humans versus those that will be created by by technology well we're back to the question we asked before which is how good how fast right now ai is not a compelling writer compared to a human being right I talk to Hollywood people it's not like they're the top 0.1% of ability to write a story the AI is not close to that right now right it's funny when you know going back to the AI Sonet I actually had a Twitter one of a world expert on sonets and I had the a right of Sonet and she's like this is looks like a good son but it's actually a bad Sonet because the whole idea of a Sonet is there's this emotional reversal in the final two lines of the Sonet that put into context and the AI doesn't do that well right so like it right now it's producing art you know and sometimes it's great art by accident sometimes humans can guide like it so right now I think of AI like a synthesizer when synthesizers first came out they caused all this controversy because you couldn't play piano now you could do this right so it helped boost creativity as an individual will it get good enough that it makes a compelling movie and or video game so I don't want to watch a movie created by humans maybe if we if it keeps but like that's not the immediate threat right what you know I I know somebody in a game company who's as a single person is building a very high quality game because the AI is making the art and help you write the code and helping write the storyline that's that's compelling yeah yeah very interesting do you back to the large company setting uh the extent to which you continue to evaluate this what are you seeing you've started to talk a bit about like use of co-pilots and the the value or perhaps even in some cases inflated value derived from that are there um what are you what are you seeing as some of some other um aspects of the value that larger organizations are are deriving from from use of generative AI so again every early study we have finds performance boost whether that is analytical tasks writing tasks persuasion tasks ideation tasks programming tasks 20 to 80% performance improvements like that's very common to see also a huge leveling effect bottom performers get leveled up these are compelling enough numbers that I think companies should be panicking even if you don't worry about any other aspect of AI if all your competitors are just getting 20 to 70% more employees and you're not thinking about that that feels like a mistake and I think a lot of companies are kind of waiting on this like another technology when you have that large a gain if you do like an Enterprise installation of Salesforce or whatever right you know it's going to be a four-year process you're going to spend x million dollars to make it happen you're going to pull off this many cons and do this much off the line to hopefully gain you know whatever it is 13% efficiency right we do a lot in it for relatively small gains there is a chance for very very large gains from relatively small investment and I think that's an incredibly compelling story and I think the problem is is that there's no instruction manual there's no one to help you you cannot hire a consulting company to do this because they don't know anything I'm sorry they just don't right apologist consults in the room I know this because I talk to open Ai and Microsoft and Google on a regular basis and nobody knows anything they don't know what these systems could do you know open AI got 12 people you know build like it's a slight over but like they're not doing testing on the use case for a large scale accounting company they're not doing a testing on the use case in a university classroom they're they don't know any of this stuff the fact that the AI is good as a doctor was a surprise they never realized it would disrupt all of education so you need to be doing rapid experimentation the the biggest danger is centralizing Authority and a committee of people of seven people who will report back to the CEO of uses for AI the CEO will set up RFP request that will go be filled in four months from a series of Consultants who again don't know anything who will then come out and do an analysis for the following eight months to give you recommendations to build an internal facing rag system that talks to your documents don't do that right the power of AI is in the edges the people using it for their job you need to figure out a way to incentivize them and capture that you need to give them access to Frontier models and like I know that's hard because it's not how it departments work but like the damage is going to happen way or another your student everyone's already using AI everywhere in your organization they're do not tell you about it so Shadow it spend is going to be your biggest concern not actually internal security that way people already I could take a picture of a screen and the AI can read it like security doesn't mean what people think it means anymore yeah great great insights all around well Ethan mik thank you so much for a very compelling uh conversation thank you also for your your great contribution through your new book co-intelligence living and working with AI uh and and and thank you for your many ruminations through one useful thing. org something that I I personally find very valuable to keep up to date on on your thinking um I really appreciate you spending time with me today thank you so [Music] much

Channel: Metis Strategy

Views: 6,452

Keywords:

Id: ewpjPyZExow

Length: 48min 18sec (2898 seconds)

Published: Thu Apr 04 2024