ChatGPT and the future of work

hello and welcome to this event on Jack GPT and the future of work my name is Sandra partnak and I'm the director of the center and regulation markets at the Brookings institution this event today is done jointly with the AI analytics and future of work initially for Georgetown University run by my colleague Alberto Rossi as you probably have seen in the last couple of months that GPT and large language models have been front and center in the media and there is a lot of anxiety among economists and among the population of what this means for the labor market for the future of work for a lot of the jobs that we have been accustomed to to doing and what is interesting is when we look back a few years it looked like that Automation in Blue Collar jobs uh like self-driving cars that are in trucks would accelerate very rapidly but to some degree the large language models now have overtaken uh those uh advances in Automation in their in their pace and are really coming upon us very fast and interestingly a lot of those tasks that attribute in large language models can actually replace or can help augment our tasks in the white collar working space so we today we have um some of the world-renowned experts on this topic here with us thank you very much for joining us we have first a brief introduction about the technological um perspective from Anthony kernick who is a Rubenstein fellow here at Brookings and then we have a keynote panel with uh three outstanding experts Susana from Stanford David otter from MIT and prasanna Tommy from University of Pennsylvania moderated by my colleague Alberta uh I hope you enjoyed the event and I'll hand it over now to my colleague Andrew thank you thank you very much Sanchez and Alberto for organizing this topical event and welcome everybody so I will start by sharing my thoughts on large language models such as judge GPT and how they contribute to cognitive Automation and let me start just by saying and this field is moving so incredibly fast so um if we go a little bit through the history of AI over the past 15 years uh there was a paradigm during the 2010s that I want to call the Deep learning Paradigm and that had a large impact on our world that came up with lots of uh impressive applications but during that Paradigm there was still a category difference between human and artificial intelligence in some ways what we are now seeing is a new paradigm the Paradigm of the 2020s the Paradigm of foundation models uh underlying the generative AI applications that we are seeing now this new paradigm of course Builds on the Deep learning Paradigm but in some ways it's also qualitatively different in some ways it feels eerily human-like we have these huge models with 100 billion parameters and more and uh their size just keeps growing and is already quite close to the complexity of the human brain which has something like 85 90 billion neurons and the leading category of these Foundation models are large language models such as charge GPD Claude the new Bing Google's board and since yesterday we have the latest edition which is gpt4 so I felt I uh should uh probably start by just uh sharing a couple of thoughts on chat Chief 84 since it has just been released yesterday uh I'm showing you here one of the main charts in the paper that introduced uh the bot uh you can't really see the labels but let me describe the blue bars reflect the capabilities of the previous version of opening eyes uh gpt3 or 3.5 on a wide range of standardized tests such as the LSAT GRE and so on and so forth the green bars reflect the new capabilities that the system that was just released yesterday gpd4 can achieve and the scale on this goes from zero to a hundred percent and essentially reflects the percentile among human test takers which this system would be able to achieve so one thing that you can see here is I will point out where my mouse point is right now the LSAT right here in the middle gpt4 ranks in the 85th percentile of all L such takers which is quite impressive if we look at the GRE verbal test which a lot of people use for graduate applications it already ranks in the 99th percentile of all human test takers so um this is really a pretty impressive performance you may have seen on social media there were a few people who say that even though gpt4 is a significant advance and it's clearly more powerful than the old models it's a little bit underwhelming well my interpretation of this is that we need to distinguish between the form and the substance of what these models are producing last November when the first version of chat GPD based on GPD 3.5 came out it really wowed people because of its form it was the first publicly available AI system that could produce a high quality and coherent content in really well crafted sentences on almost any topic now um gpt4 has a pretty similar form it also produces a well-crafted uh sentences but the level of depth in which it can go in terms of content is significantly faster and I think that's really the difference so we are already used to seeing this new form of charge Parts but gpd4 goes a level deeper it can produce more insightful content and ultimately that will presumably also have a larger impact on the world so what are all these advances driven by and here on this slide I'm showing you Moore's Law which has been going on for more than a half century in which essentially captures the observation that advances in Computing has been uh have been making computer chips twice as powerful every two years uh so what does that mean if you have a chip that's twice as powerful every two years you have exponential growth here and after 20 years you have capabilities that are scaled by a factor of 1000 however what is powering these uh generative AI systems is actually proceeding according to a much faster rate and I have pulled this graph from a paper which shows that the amount of computational power or compute that's the top um AI applications over the past decade have been using has in fact grown by a factor of two every six months that means it quadruples every year it grows by a factor of a thousand every five years so that's significantly faster progress than what Moore's Law suggests and that's behind these recent advances that we have been witnessing for the past few months including yesterday when the new version cpt4 came out so um how are these systems trained and how do they work they are trained through a process called self-supervised learning the system is fed vast amounts of data and during the training process uh the objective that it is asked to accomplish is simply to predict the next word in any given uh sentence structure the next word which is kind of blinded out for the system that sounds like a pretty simple not particularly impressive task right but uh the impressive thing is that based on this training uh really Advanced capabilities have emerged over the past five years so these large language models they can suddenly write coherent sentences not only predict the next word but write full sentences and since November they have become really good at writing coherent paragraphs and you even essays and the latest version cpd4 can of course write significantly longer texts than the previous ones and there are a whole bunch of capabilities that suddenly emerged that actually surprised the creators of these systems they didn't necessarily expect that but they found out that oh wow when we train these systems on more data and more compute and we create larger deep learning networks these systems can suddenly translate you can suddenly perform logical reasoning they can do math they can be creative and they can also write a computer code so all these capabilities have emerged once the systems had sufficient complexity and I think the best way for understanding is is that during the training process the systems essentially develop a world model a model of how the world works because if you better understand the world you can better predict the next word in any given sentence and that world model can be applied to a very vast range of different tasks so in some ways this realization that next word prediction can produce these what seems really quite intelligent uh outputs it really forces us to re-evaluate how we think about the human brain one other thing is that people believe that there is widespread capabilities overhang so what does that mean that means these systems actually may exhibit far greater capabilities than what we currently know and nobody has tried it out yet and therefore we don't know what other things the systems can do as well and uh the final thing about the workings of these systems that I want to emphasize is that there are fairly predictable scaling laws so over the past five years we have kind of come to understand if we throw more computational power at these systems and if we increase the training data if we increase the number of parameters and so on how will this affect the outcomes how will this affect the results that they produce and progress is quite predictable so if we continue to double the amount of compute that we're throwing at these systems for just the next few years we can kind of already predict what progress will be taking place in the capabilities that they have and it was a pretty amazing capabilities um now how should we think about these systems more broadly we talk to people there are kind of two camps on the significance of these large language models there is one camp that I want to call the Camp stochastic parad or Advanced autocomplete and that camp essentially says well these systems they are just dumb statistical engines they recognize uh word frequencies and they can auto complete things but they have absolutely no inherent understanding the second Camp is oh wow these systems are on the path towards human level artificial intelligence and you can see Fierce discussions between these two camps well I think uh it's easy to both over and under the estimate these systems at the same time the interesting thing is that their capabilities just work fundamentally different from ours and in some applications we can produce really amazing outputs and in others they can fail horribly embarrassingly and it's so difficult for us to relate to that because our minds function quite differently so people say they have a different capabilities surface so I want to continue uh my discussion of these two camps and instead of saying well the truth lies somewhere in the middle I actually want to suggest that both camps are right at least up to a point uh if we look at the camp stochastic paroids they're absolutely right that these uh highly sophisticated statistical systems have significant limitations uh their length of their prompt is limited the drift in the outputs they produce their training data is kind of outdated then they also tend to hallucinate and sometimes they're really not grounded in our ethical values on the other hand the camp this is the path towards human lovely eye observes their capabilities there are for example CEOs and record that states that their cognitive ability is significantly higher if they use these systems and I'll read a quote here anybody who doesn't use this will shortly be at a severe disadvantage according to the CEO of Coursera there is also a growing number of academic studies that point to the productivity gains that these systems deliver the points to gains of 20 50 or more for certain categories of cognitive workers so that's really quite impressive now what are their capabilities and for that I want to draw on a recent NBR working paper of mine in which I have kind of summarized 25 use cases that I have grouped into six categories and they range from ideation to writing assistance to background research coding data analysis and even math and let me tell you gpt4 has really made significant influences in its mathematical capabilities compared to the previous versions in the interest of time I will jump over the list of 25 but let me refer you to this NBR paper of mine if you're interested it's simply entitled language models and cognitive automation I applied to economic research but it really applies to any cognitive work that you may be performing so I'll jump over these um let me highlight perhaps the importance of prompt engineering um so this is actually a really important new task that we were all not aware of six months ago but now we suddenly have a new job that's called prompt engineer and it's in some ways a mix between programming and using natural language it is in fact programming in natural language and the performance and capabilities that we get out of these systems depend a lot on how good we are at this prompt engineering so one way of thinking about it is that it induces the model to shift into the kind of desired latent state of its World model to produce the outputs that we want to see now what are the short terms lessons of this cognitive automation so right now and yesterday we just had an increase in those capabilities they are highly useful as assistance and assist tutors they can help us automate micro tasks like little things here and there that we do throughout our work days if we are cognitive workers and they can deliver significant productivity gains so what's the economic advice in this kind of world David Ricardo taught us already more than a century ago to focus on our comparative advantage and that means we will really have to change our workflows to optimally take advantage of these new systems uh these new generative AI systems are really good at generating content so in some ways that may be devalued whereas we humans still seem to be better at discriminating content that's complementary and that will allow us to succeed and we can also provide feedback in organized projects and so on what's the medium Outlook medium term Outlook well these systems are becoming better and better and we could just see that yesterday and they will probably also be adapted to a lot more specific use cases so for example openai announced yesterday that it already has Partnerships with 10 different companies on specific use cases for gpd4 and they range from Morgan Stanley to Recon Academy to the government of Iceland and incorporating these new systems into the economy will take a lot of time but it has the potential to significantly restructure how the economy is functioning and then the role of humans in many of these cognitive tasks will likely decline and in a lot of things we may increasingly turn into rubber Stampers with a human veneer so I think that's a development that we should expect in a lot of the cognitive tasks over the next uh five years perhaps decade um so this really creates the potential for cognitive Automation and perhaps one last question in which I want to spend a slide is how does this new cognitive automation differ from the traditional physical automation that has been uh basically all their rage for the past 200 years since the Industrial Revolution so what's new we have been automating tasks and talks ever since the Industrial Revolution well one significant difference is that it affects a new category of workers cognitive workers a second really important difference is that it produces outputs that are non-rival rise that means it can be rolled out very fast this is something that can be copied and used by 100 million people after one month as we observed after the original rollout of child GPT and then finally it also chips away at our last comparative advantage versus machines so having said that this will raise lots and lots of new interesting questions uh this new area of cognitive automation will have big effects on labor markets education technological progress and ultimately social welfare and as of right now our human brains perhaps enhanced by these language models are still the best technology available to answer these questions and with that I want to hand over to our panel but to make the job for them a little more interesting I actually want to briefly demonstrate the new capabilities of charge GPT powered by gpt4 and I want to ask the system to suggest a few questions for our panel so let me write here to illustrate how these systems work please suggest a few questions that I could post a panel of world renowned experts uh at a Brookings event on chat GPD and the future of work and so when I press generate here we will see what the system says and it immediately comes up with a whole bunch of questions I will just let that run and hence the microphone back to sunshade actually back to Alberto okay well thank you very much Anton for the Fantastic uh introduction um we I'm super excited about this wonderful panel before we get started with the panel let me just introduce the the panelists starting from uh Susan 80 uh Professor 80 is the economics of Technology professor at Stanford GSB she's an elected member of the National Academy of Science and is the recipient of the John basic art medal awarded by the American economics Association to The Economist under 40 who's made the greatest contributions to thought and knowledge as one of the first tech economists who served as a Consulting Chief Economist from Microsoft Corporation for six years and has served on the boards of multiple private and public technology firms she was a founding associate director of the Stanford Institute for human centered artificial intelligence and she's one of the founding directors of the goal of capital uh social social impact Lab at Stanford GSB Susan is currently the chief Chief Economist of the antitrust division of the Department of Justice but she's here in her professor capacity thank you very much Susan for being here with us second we have David otter David is the full professor in the MIT Department of Economics he is a scholarship explores the labor market impacts of technological change and globalization on job polarization skill demands earnings level and inequality and electoral outcomes David has received numerous awards for both his scholarship and his teaching most recently David received the Heinz 25th special recognition award from the Heinz Family Foundation for his work Transforming Our understanding of how globalization and technological change are impacting jobs and earning prospects for American workers thank you very much David for being with us and third we have President is an associate professor of operations information and decision at the Wharton School at the University of Pennsylvania his research focuses on the economics of technology and labor Recent research projects focus on understanding how firms compete for software developers how software Engineers choose Technologies in which to specialize and how AI is transforming HR management much of his work analyzes data from online job sites career platforms and other labor market intermediaries well thank you so much Rosanna for being with us as well now before we get to the questions I would love for each one of you my baby kind of talk a little bit about your background and your research interests around ml Ai and the future work maybe we can start with Susan great thank you so much and thanks for that kind introduction so um several of the things you mentioned in my bio um are really the origin story of my interest here um I was first exposed to Ai and ml starting in in 2007 when I was working on the search engine at Microsoft which itself was in a very nascent form and it was really a chance to kind of see the canary in the coal mine for what was coming um as the search engine was both one of the first really large scale production and effective AI systems um but also it was something that was having a big impact on society where we needed to think about how to guide it measure it and make it better um you know the search engine at the time was kind of a composite of a lot of different relatively simple prediction algorithms that were each accomplishing a relatively narrow task and then they're all composed together to produce the search results um that that you get and maybe I'll talk a little more about that later in the panel but what I learned from there is that one of the big weaknesses of AI and ml especially at the time was that it was really doing mostly pattern recognition and as a social scientist you know most of the empirical work that that I had done in other social scientists did was about counterfactuals and about doing cause and effect what would happen if a firmed raised price if we raise minimum wage if we change a policy if we give different information to our citizens if we rank results in a different way those kind of counter factual questions about alternative worlds and so that really sparked a research agenda for me that was trying to both make machine learning and AI better and smarter and in behind the hood really understanding some of the things we we already knew in the causal inference literature about mistakes you can make uh when when trying to accomplish tasks if you don't think about biases and the data that's that was generated in the past and you can draw incorrect inferences about the future and I also worked on trying to bring in some of the amazing computation additional tools and all of the amazing pattern recognition it's you can think of that as as an incredibly important helper to drawing inferences about counter factuals and cause and effect so that's really been the core of my research agenda then in terms of like preparing for this new Chachi BT um phenomenon at Stanford we have this institute for human-centered artificial intelligence and one of the projects led by my colleague Percy Lang was to understand Foundation models and Chachi BT is a foundation model it's a general purpose technology that is going to impact and people are going to build on top of it it's going to impact lots of things around it so we've been trying at Stanford for a few years to understand both the basic science of it but also think about how to guide it in an interdisciplinary perspective so one of the things that I've been working on is because we can't it's hard to work with these really big models I've got a little laboratory where I built a foundation model around um worker Transitions and job Transitions and careers so that was a way to study the future of work um but also study uh Foundation models at the same time so I built a foundation model out of 23 million resumes and then did what's called fine tuning to a representative survey data set to make it less biased and I've been working on methods to try to see how we can in the training and in the application of these models reduce bias and it's a tiny model compared to the huge models that we're thinking about but because it's a more manageable laboratory it also allows us to explore some of these questions and then finally you mentioned my gold capital social impact lab I'm really trying to translate some of these Technologies for social impact applications the social sector really doesn't have the capability and the p people and then the the money to to figure out how to adopt a lot of these Technologies so we've been trying to put them into the field at the same time trying to advance the science of measurement and safety and ethics and impact as we go along so that's that's what I've been bringing to this so far thank you so much for uh the Fantastic introduction so so we move on to David uh sure I um I'll uh I'll try to uh you know anyway let me not spend time saying well try to do uh I uh so I I've been I actually have been a sort of computer geek my whole life I started I taught myself programming with us in grade school and I did work in that field but then after college I transitioned to the kind of uh NGO educational sector I re I directed the educational program for several years at a non-profit in San Francisco that did computer skills education for for kids and adults and that was early in the computer era in the computer in a sort of modern computer I was in the late 80s and early 90s and I was really interested in the impact of how the technology was changing what skills people needed uh who could you know advance and and uh what what skills were Amplified and which ones were made less relevant and I I did that also overseas and then I started studying that academically and so that I've been very focused on the interaction between technology and work and how it affects opportunity and uh and how it it both compliments and substitutes for the skills that people uh you know possess and bring and learn and I think that's what's so much influx uh right now and I think you know for the last couple decades we've had a pretty clear road map of the way the technology worked and what things it could do and what it couldn't do and what it would take to get from here to there and I just think that road map has been you know kind of blown off the map uh and uh and so it's a it's a very exciting time it's also I think a lot there's you know I think there's understandable anxiety and my degree of confidence in any prediction I make is is much uh less than it was a couple years ago so looking forward to this conversation thank you David uh sunny uh yes thank you so much for uh inviting me to participate in this panel on I'm Sunny uh the Wharton School at Penn um also um like David have been a a long time a computer geek I was a software developer for a few years uh coming out of school and so developed this interest in in those particular labor markets looking at how developers kind of surfed uh skills how they chose employers and in the converse of that of course which many people are interested in today which is how employers are attracting developers how do they stay on the frontier uh how do they get the uh get top tier uh Talent so I've been interested in quite a while in understanding some of these ecosystems and understanding the matching process and the unusual Supply characteristics of this market and what that means for how well and efficiently and quickly firms can adopt and Implement Solutions like AI Solutions um I'm also the co-director of Wharton of a new center on AI analytics analytics for business and very we're very much newly interested in understanding this collaboration piece which I'm course is a a big question now that's going to that is already emerging that's going to continue to emerge around these new new tools uh third thing I'll mention is I teach two related classes one's on applied data science and that class we very actively have been using chat GPT to generate code and I've been asking students to understand okay what's taking longer what's more efficient trying to understand what's actually going on that's been a fascinating process and so I'm looking forward to this discussion quite a lot perfect thank you very much so just to kind of start the the overall panel what we'd like to do is is to focus on the cha GPT on the technology uh so starting um with with Susan so Susan you're clearly a huge expert in ml methods can you give us a sense I mean Anton touched on it a little bit but can give us a sense of what ml methods generally do and why and how revolutionary cha GPT is so in other words like why is charge apt so much better than all the chat Bots we've been interacting with on Expedia or on some of this Airline websites where they were clearly not working very well why is this working instead so well and uh um kind of what are the the difference in methodologically compared to the other Bots that we've been interacting with yeah and I think like similar to David um I this is like sort of the first time that I was really surprised um in the last you know year or so I mean and I kind of I was I had a pretty front row seat to this stuff but I was still just surprised by the performance so just to step back a little I mentioned earlier like the search engine and most of the the you know productive large-scale uses of machine learning and practice have really been very simple algorithms doing very simple tasks and it's just pattern recognition and I think one of the things I we've started to get hints of this in the last few years that sometimes um just putting together pattern recognition in a in a more sophisticated way or or piecing together lots and lots of pattern recognition can be really surprising so you can be very dismissive the stochastic parrot or the prediction which I've been more on the skeptical side frankly um but now we're seeing that somehow combining things um can give you something unexpected but just to step back like so what was what was this what was my you know 10 years of speeches about like why you shouldn't be worried about this and why this isn't that transformative you know if you think about the most successful applications for many years they were things like um just prediction and classification so you know you have a data set you have some pixels of images you have labels it's a cat or a dog and you know the fact that the computers could be really good at telling cats and dogs apart it's not like transformative it's just we got some better functional forms that were that work better with things like pictures and we got more compute we got some better optimization algorithms and we were but we were basically just doing the same conceptual thing we'd done before and you're fundamentally limited by the size of your data set like if you have you know and we got some bigger data sets more cat and dog labels and we got some implicit labels from things like you know people clicking on stuff on the web that gave us some big data sets labeled data sets But ultimately you know we're limited and very few people have access to that data and at large scale and you know you're just limited by the data size but it's fundamentally pattern recognition um and and so and then when you talk about like the chat Bots that were used actually whenever people tried to use AI chat Bots they failed and so almost all the chat Bots that you used before like a year ago were completely pre-programmed so just a big decision tree if you do this then it'll do that um and that's just because that was all that worked it was it was safer and better and generally people didn't need to do that many things and so you just had a decision dream so that you and then and the few attempts to put them in the wild were were kind of spectacular failures so were the Breakthrough really is that it's actually gotten it's it's gotten the the AI has gotten good enough that it sort of passed the acceptable threshold and so part of that I think that the end times overview um helped understand the difference when I said like you can take think about just one prediction problem I have a set of images and a label but even just predicting a paragraph of text in one paragraph you have lots of prediction problems you can use the first word to predict all the rest of the words you can use the second two words to predict the rest of the words you know you can predict take the first 10 words and predict the next 10 words so you have lots of X's like prediction predictive variables and outcomes all in one paragraph and then if you have all the text and that's out there on the web you have actually a whole lot of data if your goal is to take an initial set of text and to predict what comes next so that's pretty powerful still people were trying to do that for a while and they were sort of limited by the functional forms they were using and so some of the recent breakthroughs have just been we finally figured out some functional forms that were tractable and computable but that really took advantage of how the words related to each other in a sentence so fundamentally all along since you know the last 20 years the the basic thing in doing this text modeling has been to take text and find a lower dimensional representation of it you think about all the pairs of words in the English language that's a lot if you think about all the paragraphs of words in the English language that's a whole lot and we try to but what we try to do is find lower dimensional representations of text that is useful and this of context aware representations have worked really well but the second part of the Breakthrough has really just been in the engineering that you know if you the dimension at the true dimensionality is very high and so you need a really really big model with lots of parameters in order to do a good job and we've just figured out the engineering and put the money in to buy all the compute we've gotten the hardware and everything else to be able to build a really big model so conceptually it's still the same thing it's just better functional forms better engineering better compute and lots and and just very very very very very very big and then being what is doing very very big do it's it basically makes things very contextual so it can factor in the style like write something in the style of an HBS case or of a news article and so that context you know if you have a big enough model like that that context can produce different output for those different scenarios and I think that's kind of what's been so magical now I still though I mean at some level so it's like putting together pattern recognition but it still is pattern recognition like it's still not smart like you're not the architecture of this is very very simple and it's still just a Big Blob um you're not putting intelligence into it you're not putting structure into it everything is just learning parameters of a big pattern recognition model so the mistakes it makes also were kind of predictable like if it learns from Reddit chats it's going to sound like a Reddit uh chat if it you know if it learns from romance novels and then you start talking like a romance novel it's going to spit back what's in a romance novel so you know it's only only it's still only pattern recognition you know it'll if you ask it for Papers written by me it'll put things in the format of a reference and then the words sound like words on my CV but they're completely made up um and that's because you know the words it understands this is a very likely set of words like you could fool somebody else it looks like Susan CV it's just not my CV um and and so you know but one of the things that's kind of interesting and we can maybe talk about that a little bit more later as well is that you know when you go to say the search engine Bing and ask the same question um partly I think it's using a more advanced model so we still have to test this out today but the difference is but the you know the search engine actually knows what a article is because it's got a very complicated set of heuristics and and decision trees so it knows what references are and it knows where you find them and it actually can give you factual information that it references so one of the things I think that's going to be really interesting going forward is how people take the strength of chat GPT which is that it it can like summarize information and get pause possible sounding text in a style and put it together with other kinds of Technology either on on either in in kind of helping people with prompt engineering very complicated prompt engineering that forces chat GPT to answer the question in a better way or post-processing of the stuff that comes out to correct errors like it's not that hard to recognize a reference in a chat GPT answer to have a classifier that says that looks like a reference and then to do a function call to look up is that an actual article or not and then not return it if it's not an actual article so I think that's really part of the frontier we have this big black box Foundation model which um is that is doing pattern recognition incredibly well but it's a it's it's not very easy to get inside and tweak something you know this thing works at scale exactly because it's a general purpose thing and so how we put stuff on the around it on the in inside on the outside how do we fine tune it to different circumstances will really I think be a part of the determinant of how pattern recognition together with other technology can lead to an interest to a really powerful result well thank you yeah so this is great but in terms of kind of the task I think Anton didn't really have time to get into it so what are the kind of the Practical examples that the chaji PT can currently do I mean we know that they can form sentences it can can it understand images can it what are the sort of things that it can do and what instead are the sort of things that it's not going to be able to do for quite some time like according to your prediction yeah and so let me not just talk about Chad TPT but just like this like category of Technologies um you know we they we already see this in like little apps on our phone we can generate images you know we can generate images in the style of something we can generate text in the style of something and we can summarize and those are really really powerful tools I think the one that's easiest and there's already some research papers out about this um copilot the GitHub co-pilot has been there for a little bit and it's incredibly powerful and anyone who hasn't tried you know you can it's just such a it's it's such an accelerant to be able to um not have to stop and think about the syntax and you know be able to get code written out now the interesting thing is it's still you might imagine as a user or just if you read about it in the newspaper that what this is doing is you've you've encoded the rules and the syntax of of programming languages and it's following those rules and and you kind of somehow communicated with those rules were um certainly the chat GPT version is is not doing that it's it's just generating code that it predicts would be a likely answer to your question like from things like stack Overflow and other code that it's seen in the past so it's it's going to make mistakes and it's really interesting when you use it because it'll give you code and then you paste in the error message and it says oh I'm sorry you know that that you know you need to do this this and that and you think well why did it give me code if there was that it knew was wrong but it does it didn't know it was wrong and it might have taken a long time for it to check and see if it was wrong it gave me code that seemed likely but then when I give it the error message it's sort of got doing pattern recognition for what people have said when people have asked questions about that error message and so it actually gives a really good answer and it seems smart but it's just a sequence of of patterns if it wasn't it wouldn't give me buggy code now you can imagine over time on top of that it's very expensive to actually test code like that's going to be a function call it's going to take compute it's going to take memory it's going to take time but you could piece these things together a little bit um and in the future I can imagine that you will improve once you get the basics down from this like pattern and recognition stuff that you can figure out some some efficient ways to start making it have less mistakes and and self-correct its mistakes or even like sometimes you see this now it'll write something then it'll delete it and then write something again and you sort of get the idea that it's got a layer of more expensive checks that it's doing that slow things down a little bit and then if something's triggered it does some more compute and gives you a better answer and again that's sort of like what the search engine does it kind of triages uh you know whether or not it's worth it to go and do an expensive computation and so over time that triage piece of it may get better but I think the coding is really good and we've seen estimates that it's really improved productivity I mean all my students are using it I've used it you know it's just it just really um picks things up and I think there's a lot of these research tasks that have been incredibly repetitive and Incredibly frustrating like where you have to click on something and look at it and you click on something else and it basically has the same information and you're trying to get unique information and you set the key go you know digging through all this stuff and the ability of chat GPT to summarize information and not show you redundant information um I think is just super charges any kind of research process and then also like just this kind of generating um rough drafts um and and getting better Styles and so on I think it's just it's just kind of game changing but I think a lot of this has been covered and written in Anton covered a lot of this as well so maybe I'll stop there just because I think there's a lot been said on this topic already okay well thank you so the the second thing I wanted to touch on was like the effect of chat GPT on this labor markets and I wanted to switch over to to David so from a labor market perspective what if anything makes AI different from this this AI this different from prior generations of I.T is it just more of the same faster or does it change the nature of like human machine substitution or complementarity and and more importantly if we don't know the answer to this kind of what kind of data would we need to or evidence we need to gather in order to answer this question yeah I think it is qualitatively different it's not just more and let me just step back a little bit to sort of both complement and substitute for what uh Anton and Susan said um so I think it's it's a little bit dangerous to make analogies about the number of neurons and so on and sort of say if it has the same number of neurons as the brain then it's the same I don't think that's what Anton meant but others may have heard that and that's not correct you know there it's a it's different from our brains and I think it has capacities we don't have and we have capacities that it doesn't have and that's been true for all of the tools that we've made for a very long time um what makes AI different from prior generations of software uh is that all software that you know historically was followed the following you know kind of cookbook you decided to ask you want to automate you figure out all the rules you specify them completely and then the machine executes that series of steps without judgment or without you know adaptation or learning and carries them out now it's great for calculation that's great for playing games that's great for storing data that's great for many many things but in sometimes the the machine did not make inferences it did not learn from Context and didn't generate new things uh it was in some sense you could think of it as uh uh whatsoever I want to use it it just it's just uh executing now of course AI is also just executing it's all deterministic right it's all you know atoms and molecules but that's not a useful way to think about it anymore any more than is useful to say you know when I go home for dinner at night that's just molecules moving from it to my house in Newton it's true but it's not a useful analogy uh so what's different here is that uh AI is able to effectively learn instead of us saying to get from here to he to there you follow this through your steps you say you started here you end up there you figure out how to get from here to there and you can make inferences on that and and that means that the range of things that are subject to this tool is much broader than the set of things that we could that we had to hard code to accomplish uh those tasks were routine in the sense they were codifiable this can do many many non-routine things and so the word generative I think is very very useful uh because it really is evocative of what's going on it's sort of creating stuff that we didn't we don't think ourselves of having programmed into it it's also very hard to predict for the same reason um and so I think what we want to think about then that we make lots of tools we all use tools all the time most of what we do is accomplish with tools whether it's getting from place to place the computer you have in front of you the writing instruments the clothes you're wearing we're surrounded ourselves by tools and we are completely feasible without tools and this is a tool as well but the question is what type of tool is it and is it and in particular is it a tool that complements our expertise and makes our skills more valuable as many tools do right you wouldn't take a you know a Pneumatic hammer away from a from a roofer you wouldn't take a scalpel away from a surgeon you wouldn't take a computer away from an economist right those are all their tools and so in many cases tools make our skills and expertise and knowledge and creativity more valuable but then there's some tools that do the opposite they actually commoditize what we have that was valuable right so if you were a London taxi cab driver and you had exhaustively learned all the streets and byways of London and all of a sudden GPS came around well it's great for consumers but your scare skill is no longer scarce and and that's what we need to understand that relationship and also uh direct right so this technology is incredibly malleable right it can do many things right you can use it to build the Great firewall of China you can run the world's largest surveillance state you can do surveillance capitalism if you want to or you could use it for Education you could use it for medicine you could use it to make people without medical degrees effective as doctors and so to say what AI will do kind of Misses our agency in the entire operation AI will do what we invest in to make it do that thing and so we ought to think be thinking hard about what we want to get out of it now of course lots of people are going to do lots of things and it's cheap and lots of Bad actors and you know people with profit motives that may be good or bad many many things will happen but we have a shared interest in directing the technology in a way that will be complementary to us uh and therefore more advancing societal goals helping us solve some of our hardest problems like you know climate change uh you know nutrition health and uh and and spending and doing less of just replicating human capabilities because let's face it we have human capabilities those aren't scarce let's figure out uh something something uh the best way we can use it huge challenge thank you okay yeah but always continue on this uh kind of a labor market aspect so one thing that uh we've seen is this kind of an acceleration of the trends we were observing uh kind of over the last 20 years right in terms of kind of um the kind of exacerbation of wealth inequality or income inequality so David like given that you spent the last couple of decades trying to understand how technology affects labor market dynamics what are the biggest trends that you've observed over the past 20 years that have changed in the recent past before because of AI yeah absolutely so that the the thing that's you know what what we've been doing a lot of with automation of various forms over the last several decades is codifying routine tasks whether they're office clerical administrative tasks whether they're repetitive motion tasks on a factory floor and a lot of those really have changed enormously the office of today is completely different from 40 years ago when people had typewriters filing cabinets uh and and calculators and Staples and those were basically Stables those were their main tools they didn't have spreadsheets they didn't have searched they didn't have word processing they didn't you know many many tools and as a result there are many fewer clerical and advanced workers than there used to be per capita they do a very different job they don't do any more typing and filing and sorting and copying right they they fix problems they resolve travel issues they handle those you know god-awful receipts and so on uh so it's a different problem and this has led to a lot of bifurcation of Labor markets actually a lot of those were middle school jobs they've been hollowed out as a consequence we have a lot of people who are highly augmented who are doing professional Technical and managerial tasks whether it's research whether it's medicine whether it's law whether it's create creative tasks and then we have a lot of people who are doing in-person services that are hard to automate but also are hard to make more productive as a consequence are pretty low paid and use relatively generic skills uh many people could be a good waiter or security guard or cleaner with you know a little bit of training and that means those will not tend to pay well and so the the challenge is great is well what is the new terrain that will be opened up what will be complemented now and what will be substituted will it be the case that basically all thinking will be done machines and all people will basically be building stuff assembling you know handling the physical tasks of the world or how quickly will in fact AI make robots much much more capable because a lot of the challenges in robotics is not the actual movement it's the calculation around the movement um so I think you know if I had to put this in terms of two questions that I think simplified although probably too simple one is does AI what skills does it complement does it like for example Susan mentioned coding I have some students she had Noy and Whitney Zhang who did a really nice study about writing tasks and showed that basically professionals who do kind of you know in industrial scale writing uh they save a substantial amount of time using chat GPT and it kind of brings the bottom up it makes the less good people uh somewhat better uh and no I don't think that's a universal I don't think if you gave it to someone who's illiterate it would make them better right but I think there's a complimentarity so one is who is complemented is it only the best the people who are the top of the field is the people who are in the middle can enable people with a good skill set to do be more effective in many things that's a very good scenario and then the second question you have to ask is well what is this sort of now you sound use jargon what is the kind of elasticity of demand for all those things so if everybody gets 50 more productive at everything right that's good unless we saturate the market with that and all of a sudden things that were valuable when only a few people could do them are now so cheap and commoditized that in some sense they don't they don't command much value anymore now again it's good for productivity it's not good for distribution uh and so I think too so the questions are again just let me rephrase them and then uh pause one is what human skills are complemented and given more leverage made more effective you know having some fundamental skills then this tool allows you to do a broader range of set of tasks and that I think is the good scenario right where we take people who have some expertise and make them more effective at a broader range of things and the other question is how fast do we saturate the market with those things such that even if we get you know you can do a broader range of things they're now just Dirt Cheap uh or water cheap which is the way we use water uh and uh and suddenly that's not such a good living yeah okay perfect yeah and kind of moving on the other side of the labor market not so like thinking about those people that actually developed these tools uh prasanna I know that a ton of your work centers around understanding kind of the labor market for AI skills so a question I have is like where do companies acquire the talent needed to deploy these AI tools within the firms which companies are more successful which ones struggle the most uh is there like is it very difficult to acquire talents to design this kind of uh large AI tools that are VPC becoming more more popular yeah so thank you uh so as we certainly know that AI Talent has been scarce for a while top AI Talent particularly and these most recent round of tech layoffs notwithstanding we don't expect that to change substantively in the short run I mean top AI Talent still is incredibly hard to find it always strikes me um when I talk to technology managers not just HR managers but just Managers generally how many of their decisions are focused on really the hiring piece you know not not just the obvious ones but decisions they make around for instance how much to participate in open source communities and things like that a lot of it has to do ultimately with the way they want to be able to compete for some of these developers really and of course they want to get paid but they also want to be at the Forefront of the community that uh developing these when we we've talked about how quickly they're evolving and rapidly and how rapidly they're evolving so it is uh clearly critical uh for for these developers to be at at the Forefront um at least in my mind it raises kind of interesting question which is and some of my colleagues putting some of the ones on this call have written about you know things like Superstar firms but you have this sort of Matthew effect where you have these developers were attracted to the top companies and what we well this Talent is scarce it's it's hard to understand how if you're behind the frontier how how to attract and compete uh with these workers um so that's one thing I'd also like to mention this might be obvious for people on this call but the broader audience perhaps that I speak spoken to it's perhaps not so not so much the first step I mean one step of course is acquiring AI talent but it seems like firms are settling into the reality more that that's there's really a long pathway that a lot of successful firms have been going through right in terms of complementary assets I mean labor of course is matters but data which has come up several times in these calls business process transformation all of these things are difficult and I think we're now hitting a phase where firms are saying okay this is hard and this is expensive so uh even if you even if you can find a way to attract this Talent it's no easy thing um to have the type of of data you need to train these models and some of the firms that we see to be successful at it have been working at this for for years and even decades really you know starting from collecting consumer data um in the in the dawn of the internet so to speak so so this is really a long pathway that goes well beyond AI Talent as one piece of the puzzle that you have to have in place to get things right um I think the other Force last Force I'll mention here which is an interesting one maybe pushing against this is the the uh a lot of a lot of uh resources are are being expended um by Tech firms big Tech firms particularly at lowering the barriers for using machine learning right so there's a lot of interesting action on maybe automated pipelines going back to the release of deep learning Frameworks uh for for public consumption and then automated pipelines and so on so I still think we're in a world you need to have ai Engineers you need to have data scientists but it's going to be interesting I think to watch how this progresses and how much the lowers the cost for firms to enter into this if they happen to have the the data on hand I see and so in terms of like the international landscape so how competitive is the US compared to other countries right now do you expect this to change in the future and if you think that over time we may kind of lose our Competitive Edge do you think what what policies can the US Implement to remain competitive yeah that's an interesting question you know I think I think if we had this conversation 18 months ago the the general tone of that answer might have been different in the public at least I think there was uh some amount of uh public anxiety or angst um especially with respect to Machine Vision tests and those sorts of things about America's position particularly with respect to China and their Tech ecosystem and that there are large Tech firms have been have been doing I think maybe two things have changed since then um one is large language models of course I think that I think I think it's pretty I don't think it's a controversial opinion to say that us is the US is viewed as a leader in large language models and uh there is certainly other there are other language models in other parts of the world that have done well but I think there is a conversation now happening about how far how the US gets so far ahead and what can we do to catch up so I think from a large language model perspective that's been changing the conversation and then of course uh the the relationship between the Chinese government and their Tech firms has been changing in ways over the last year or two which I think has been challenging um the the the the relative balance I guess in terms of rate of innovation in those two those two sectors um you know in terms of the uh your the other part of your question about what we can do I think most of the top AI Talent is probably still produced here probably because of the complementary resources you need to to learn these skills there's an evergreen question of course about our ability to retain that Talent right those don't go away um there's certainly investments in basic infrastructure I think that can go a long way for uh top AI Talent the last thing I'll mention is there's there's a sense in which it certainly works you don't have to look much further than a place like Toronto right where you've seen like industry government academics kind of come together to develop a Powerhouse in terms of an AI or AI ecosystem and so it certainly seems like there is a role here for um for these these different pieces fitting together I see you perfect so we have a number of questions that are coming in from our audience and I would like to throw them at um all the panelists one uh I think it's kind of related to ethics so the question goes as follows um how can organizations ensure that they are using chat GPT in an ethical and responsible Manner and what steps can be taken to prevent biases or other unintended consequences from arising so I don't know if any of you want to to take it I mean this is related to you know a kind of ethical consideration algorithmic biases yeah maybe I can start with that and it was a while ago but to pick up on one of the things I mentioned in the introduction this is something I've been thinking about um there's a actually a close relationship between thinking about causal imprints and thinking about fairness and bias um because if you have a a really high quality model of the world that will tell you what would happen if something changed um then that high quality model is less likely to be biased in a lot of the ways that we understand it a lot of you know biased algorithms can come from sort of using shortcuts that are proxies for for True underlying causes and in CL if you one of the reasons that you see these problems in like classical big um prediction models is that you're kind of throwing in a soup and that's the beauty of them you can just throw in a soup of predictors and and let the model work and then you get an outcome but it it and it's it's actually been hard to kind of muck with the interior of those black boxes without sacrificing performance both computational and predictive performance um but then you get you know correlation and not causation and models that don't transfer well to different environments or that it's not performing well if the world changes so these are pretty closely connected and I have some writing about this for example an overview paper and nature machine intelligence that's just about prediction problem now it gets all that much harder when you come to these large language models because um you know the state space is just so large so at least like if I'm reasoning about bias and image recognition I can sort of talk about the features and yes you're using a neural net to represent the features but there's like a pretty close connection between what you can see and like whether there's an ear or whiskers or what's the color of your skin if you're talking about humans when you go to these large language models the representations of the text are so complex and so high dimensional and and we don't really like understand the ground truth well enough that it can be very very difficult to even start to up your head around it and then I also want to remind that like we the beauty the reason you can take this architecture and apply it to just all the text in the human language and it's out there and just let it go is because it's not sort of carefully crafted in a way so um when you go to try to think about debiasing it um it's like how do you even start and that's also a problem by the way that like a government that wanted to censor or even if you wanted to you know try to make things safe for kids or safe for schools or you know safe for a company to use in their chatbot it's very hard to like make it behave reliably in any way or make it withhold information because it's not like there's like just a little button you can push like it's it's just a big representation of language and nobody including the people that make it understand it the only way you understand it is to test it and the and the space of the testing is so large so it's very very hard so I've been playing with this in these labor models and I've been looking at things like the the gender wage Gap and so that's like a very simple problem because I want to make sure if I want to say that there's a gap between men and women's wages I want to make sure that that Gap is real and not just a a an artifact of the fact that I used a low dimensional representation of people's histories but that low dimensional representation would say more accurate for men than women or you know gave a different answer for men than women in some way or didn't fully account for Salient aspects of of the women's histories so I'm trying to make sure that my my what my model tells me is a wage Gap is really a wage Gap so we've been actually tinkering with the architecture of it's the same architecture that's behind the llm models to try to de-bias them in just one dimension at a time and we've made some progress and we have some theorems and we have some algorithms that kind of make sure that it's not creating artificial wage gaps in one dimension but that's a single binary variable gender imagine now like you're trying to like debias an entire language model in I mean in just ways that you just can't really comprehend and and the performance I can get I get like only small more performance gaps if I just want to be de-biased in one single dichotomous variable but trying to like do it globally you know is just very very hard um I'll mention one other element of that so part of it is like changing the architecture and changing the objective function but that's going to be hard to scale a second thing is this fine tuning and I alluded to that but let me talk a little bit more about it now because it really affects some of the applications you would do so there's hundreds of papers this is relatively easy to do what you do is you take a trained model somebody spent a whole bunch of money and a whole bunch of compute and a whole bunch of engineering time and trained a model that produces a low dimensional representation of the English language once that's trained you can put some words in and it'll spit some stuff out what you can do next is take that to a much smaller data set maybe it's a proprietary data set of data with confidential information um you know personal information something confidential for your firm but you can take that big model and fine tune it for your smaller data set and that's going to do a couple of things it's going to make it give answers that are much more Salient to your environment in my case I took a large resume data set that's biased towards rich people and high educated people and then I fine-tuned it on a representative survey of of American workers and that's going to de-bias it in another way it's going to make it fit better for the whole set of individuals we have in the workforce um and and so that's another way to de-bias it's a different approach and it does has different properties but you can you can sort of find a corpus that you think is okay in the dimensions that you care about and then fine tune the model in that direction then a final thing that people have done and this is happening in chat GPT is that you you incorporate human feedback so you get people the rate like is this a good or bad answer so you could have people rate is this biased or unbiased and then you add that into the objective function so it's that human feedback component so those are those are basically like you know three different ways you change the architecture to like in in a way that you anticipated de-biasing that's one two is you fine-tune to a more representative data set and three is you incorporate human labels but all of those are inherently not scalable fully because you're limited by human feedback you're limited in the number of Dimensions you can try to debias you're limited by the size of the Corpus of unbiased data and and all the ways you might have conceived of whether that data is biased so there this is going to be an ongoing research area and an ongoing corporate um Innovation area I would say this is like we've got 20 years of work ahead of us to get this right yeah perfect so in and also kind of very related to what you were mentioning so we know that a lot of these AIML tools are trained on customer data employee data and so a lot of people are concerned about uh data privacy and security issues so in the context of kind of a new technologies in the workplace what steps can organization take to protect their employees or customers personal information does anybody want to take this so I can I can start and maybe I you know others can layer in but what I expect to happen is that people will produce a service like look I can make the fine-tuning programs you know with a PhD student in my compute so basically that's not a big impediment and there have been people already fine-tuning these new models on smaller compute so I think that problem is almost already solved technically and it just needs to be productionized so one service I expect to come up imminently is Services where companies can use this on their internal data their email their documents they create and everything else but part of the service will be that it's not shared back and people will buy the one that doesn't share the data back um now there's ways like there's Federated learning and so on that in privacy preserving stuff so you could share the data back but there's going to be I think some organizations that just don't want to take a risk and so they're just not going to want to share but that's going to be perfectly possible they're going to be able to basically fine tune on their own data and then it won't Escape shape now there's some still some problems like suppose you do this in an e-commerce site and you train this in all of your data and then somehow people start doing hacking of your algorithm to try to figure out somebody else's purchase history and we really don't have the full protections on that yet I think like Bing has kind of hacked some of those by by limiting the number of queries in a row just because you can you can test five queries in a row you can't test 100 queries in a row the dimensionality is too large so I think we're going to find some hacks in over time again that's going to be part of the research and part of the Innovation is to make it more safe but generally I'm expecting that that this is something that companies will do using their internal data without having it Escape now in some cases where the data is less confidential you might find the prices are lower if you share your data back or at least if you share you know parameters estimated or gradients estimated from your data back with a common model and companies will try to do that because the their their software will work better if they could share data across multiple firms that have similar data so that's something else you might expect but it might you know depends on the nature of the data the attitude towards privacy and so on which which way that will go okay I think we've touched upon a little bit of this but I want to get deeper into it because we're receiving so many questions related to this so one one of the key kind of theme in the questions that are coming into the chat is how will chat GPT change education like the the question is everybody's thinking that uh there's going to be some upskilling that is needed and is kind of our University is going to be the one that should provide this upskilling are they private corporations are going to are they going to do it on them by themselves and uh also in terms of the actual classroom dynamics how can we teach where a lot of the kind of our tests and trying to understand how individuals have learned from our classes is by kind of asking them questions and maybe writing essays but now they are kind of people are getting access to these tools that are becoming more and more powerful uh so how do you think the education system is going to change and how what are your universities doing in this context and how do you see see this kind of evolving in the future I'll be happy to chime in so I mean I think it presents both Challenge and opportunity the the challenge is pretty clear uh which is you know how do you determine if people are learning uh if if machines can do a lot of their work for them I mean you know they're used to it used to be very controversial by the way to use calculators uh in the classroom and so on we've gotten used to that and we've gotten used to the idea that uh you know testing making people memorize times tables is not necessarily the best use of their time I do think this makes it a little harder because the uh the range of tasks that you could sort of consult the machine for are so Broad and I I think people are adapting to this in real time uh you know you may say well maybe it's like times tables you don't need to know you just have to have an idea and you know uh feed it into chat DPT or you know a successor and uh see where that goes but so I think you know this is this is a a major Challenge and I think an issue even at the degree that we think we know how to deal with this is educating Educators on how to deal with it they don't have a lot of time to you know to spend on this so I think it's gonna it's a responsibility to you know kind of figure out practices and and and then work to enable people to uh to learn them and use them well um let me serve but let me say the other side of this which is not on sort of the testing but on the production of education and there I think there's great potential you know education is an incredibly expensive slow customized activity it consumes a larger and larger share of child societal resources not just in terms of teachers and buildings and Equipment but in terms of the time people the fraction of people's lives that they spend in classrooms right people used to go to you know work at age nine and then at age 18 and now if they do a PhD at age 40 or so and uh so it's a i and of course a lot of adults we have we have an incredibly abysmal record actually of skills retraining for older adults not for for young people but you know people who are further in mid-career and further and the question is can we use the technology not chat GPT exclusively but many of these Technologies to make education you know more engaging more immersive more accessible cheaper faster right you could you know one thing we know about adults is they learn really well in situ rather than you know from watch people on blackboards can't we create simulated environments where people certainly you know learn in some sense in the field from doing what they're doing where the stakes are lower but the the feedbacks are very similar so I do think there's I don't think this eliminates the need for teachers and people seem to be uniquely motive motivating for one another that's part of the reason teachers are effective they just seem to be more have more power to motivate than videos more power motivated than books and so on um but it certainly can do a lot more customization and a lot more support and then a lot more simulation and simulation is something we can't do well in the classroom without technology so I think there's this is a fabulous application when I say you know we should guide the technology of course useful things that's a very useful thing that we could uh we could be investing in um let me check it out just a couple more things um so I I've also um you know even this morning I was hearing students say hi it's like having the best ta ever right in terms of of Education the way they're learning uh how to particularly on coding tell us how how how to uh how to code using this tool is amazing and there's a couple of meta comments one is that um the diffusion path for this um we're all AI targets as employees or customers we have customers we have been for a while but I think with the reach of these companies like Microsoft Google slack and notion we know that they're going to be able to roll these tools into their productivity uh software so quickly it's going to be really interesting I think to watch uh people become AI users really rapidly so the scaling process we're talking about the timeline might be a little bit unpredictable uh relative to what we've seen before and the last thing is more of a question you know I I love to Grapple I'm interested in um Anton mentioned before this notion of being a rubber stamps for discriminating you know there's this it sounds like everyone here has used this for either python or some other tasks like that and you know it's I think it's fundamentally true that uh at least at this point in time you have to know python to use it for Python and so what this means for that complement substitute calculus something that I find really interesting in a question mark I'd love to know more about so let me maybe pick up on some of those points and I thought Sunny made multiple excellent points um actually one of them maybe is a bit of a side point but the fact about this adoption um I think one thing that's makes me whenever I give like the scary case for work for jobs is in the current environment is it over the last 10 or 15 years firms have been adopting more modern software architecture they've been using software as a service they've been moving as a cloud they've been doing things much more modularly and they've also gotten pieces of automation into things like customer service where there's like a you know AI assisted answers for example that makes people more productive in that environment where you're already like partially automated and you've laid a lot of groundwork and you have figured out your data and you've figured out how to pass it around and figure out how to give everybody access to software as a service if you get something in that makes everything like eliminates the need for a certain aspect of customer service the adoption at this Point can be very very fast because people are already using providers so I think that you know and that goes across a bunch of services you mentioned slack and email and so on so that a bunch of people have adopted a cloud software product where you can update that product instantly for everybody and then everybody adopts overnight so it's like we've like been building factories to be ready for electrification and like now the lights are turning on you know and so even though it's been really slow to see some of the impacts that may happen faster then I'm circling back to another couple of comments that Sunny made I totally agree with the best ta ever um you know I was trying to make graphs and and I like them to be really pretty um but I I get I just don't have the patience to do all of the syntax to make them pretty and you know you just I just copied the top line of my Excel of an Excel spreadsheet so I exported my data set got the top line of excel get the the list of variable names you know you copy it in and you say like write me an R script that's going to load in the data set you know aggregate it this way do it that way make me a graph you go back and forth and suddenly I'm getting this beautiful graph um and you could really do that in a very natural language way and I think there's just a few steps to get that to be really usable so I needed our studio installed or Python and stuff like I you need you needed to know something to be able to get started um but after like if you can make a user interface that simplifies that you can be running our studio in the cloud um you know you can set up a server that that abstracts that so nobody has to install any software and then and you can make it easier for them to just you can ask it what is your variable names you know and then you can ask it what kind of chart to make and then you can just iterate I mean it's just kind of amazing of course it has to have the data to actually make your chart so there's there's a little bit of work to be done there but you know it's it's it's it's so close right like it's so close to just not needing any syntax at all that I just I just can't imagine it's not going to be there in you know a year or two it's like I feel you know if I dropped everything I think I could write it you know so I think I think this is just going to be close so this is going to be really interesting going forward and then a final thing this is very unrelated but I actually did a research project very recently where we were trying to get women into tech jobs and so we come up with a program that only costs 15 a person that increase the probability of women getting a job intact from like 0.2 to 0.3 um and you know it was very successful um and what we did there is we we got people who had the poor like kind of aptitude and then we we figured out we interviewed employers and figured out what what they could could the workers do to demonstrate their skills and then we put them into groups and they work together in groups um and used a technology platform to work together to create a portfolio which then they could show to employers and I think that kind of model is really has a lot of potential here because you've got you've got people with General aptitude part of the thing is that you know people that you could sure you can do stuff on Coursera sure you can Google it and find it but people don't really have confidence that if they make those Investments it's going to translate into a job so you need some curation you need some teamwork you need to make sure that the things people are learning or things the employers actually want but if you just put a few of those pieces together and and you know we're used doing this for Polish women and Ukrainian refugees and you know it's working and it wasn't that hard like basically we built this with like a couple of people and very little money so I feel like we can solve these problems with a little bit of thought in a in a scalable way but but but you do have to have multiple ingredients um to make it actually work let's see so one thing one question that just came in which I thought was a relatively interesting is um what like the question is answer is asking uh what are the foreseen and unforeseen regulation on assistive API tools like do you think that uh the regulator is going to step in and kind of determine what is kind of how this tool should be designed or this is something that instead is that the government is going to be staying out of okay I mean I think that I don't think the government will be so involved in design but the question is when they're used in high stake settings especially for decision making then there's a role for policy this comes to you know credit issuance job screening all kinds of things like that and this is where this issue as Susan was speaking of earlier of kind of the you know the biases or the you know unin you know unknown properties of these models now I mean we have lots of decision makers in the world and and they have lots of unknown properties it's not that people do this well and without bias right there's lots and lots of room for improvement on human decision making uh the difficulty and and I used to be actually very optimistic that but using machines it's just you can sort of we you can tune out the bias right you have it's much more educable than the average decision maker uh but it is the case that the machines they're so opaque to us now that uh it's that makes it harder to do there's you know I I used to you know refer to pilani's Paradox which is that we know many we know many things that we don't know how to explain what we do right we we do them but we don't understand them and now we have the opposite problem which we have machines that understand many things and can't tell us what they're doing this is sort of pilani's Revenge uh and so it makes us hard it actually makes it hard neither neither we we cannot communicate our sort of tacit knowledge of machines and now they can't communicate their testing on to us so it does actually create this real challenge for uh you know kind of Regulation or even testing right what does testing mean like you know the notion date well I'm gonna drive a Tesla in on average safe but every once in a while it's just going to drive off the San Francisco Bay Bridge uh you know into the into the into the bay and like well we just have to learn to live with that because on average it's fine that's not so exciting um so I do think the regulate the regulatory challenge is extremely difficult here Regulators don't understand this uh and no one you know uh really has a complete handle on it and so figuring out what are standards right what is a standard that something has to pass I suspect a lot of things will look like that what do you have to prove to certify that this thing is acceptable for making these types of decisions and another really fruitful area of research something uh Senator melonathan has worked on with co-authors is how do machines and people interact in decision making because we're going to have a lot more machine supported decision making but you know which whose authority should you know Prevail when should you accept the machine when should you accept your own intuition how do you virtuously combine them will we just be you know rubber stampers for machine you know we'd be like those Pilots that have forgotten how to fly and so when the autopilot fails we crash into the sea uh hopefully not so this is going to be an important area for training and research yeah I should mention I have a little research paper that that models formerly the trade-offs that David was just talking about about kind of falling asleep at the wheel and the the um incentive problem that's created by having a high quality AI so um you know trying to figure out it's really like an almost an organizational design problem where you know people were motivated to pay attention because they were making decisions and if you take that away then it changes the returns to your Investments and you need to reshape your incentives and structure to to get good outcomes and it's it's very hard to anticipate just to Circle back on something you know something I saw in the search engine as well like you know a typical engineer even a manager of an algorithm doesn't understand doesn't necessarily understand how or why the algorithm works and in certainly is not trained to understand the unintended consequences you know how do you know your algorithm works well you a B test it you tweak it a little bit and you look at you know 10 numbers in the A B test and they look good then you go forward that that doesn't necessarily lead to understanding and some of my research actually was about you know trying to take the results of a B test and help Enlighten people about what the mechanisms are because it was I was observing that people weren't really understanding what they were building and that's certainly the case for the people building the these new large language models I mean certainly the people working with them have a little more intuition than the rest of us but roughly everybody's learning about we I mean you can write down the math we can all stare at the math it's not enlightening so then you know your your looking at the outputs and and that's all we know are the outputs and it's not like any particular individual has had that much time to play with the output so far so you know I think sometimes like oh we need to talk to the person who built this and they'll explain it to us and no like they I mean they can they know they've run a few more tests and they've gotten some intuition from those tests they've run but the space of tests is is very large relative to the tests that have been run today and mostly people have just been focusing on getting the things to work better and making them bigger engineering them to work bigger and just seeing like generalized performance outputs so that's kind of where we are right now and it's not like somebody has this magic Insight that if we could just find the sage they'll be able to tell it to us yeah perfect yeah this is I think this is we we are out of time I would like to take a moment to thank Susan David sunny and also Anton for the Fantastic event I think that this was incredibly insightful and uh you know GPT 4 was just released yesterday you know being and all the other tech companies are jumping in so I think that in another three months what we said today is going to be old and probably not relevant anymore we're gonna have to have another event and we're going to keep on going and keep on learning but thank you so much for the insights it was absolutely phenomenal and everyone enjoyed the rest of your day thank you so much this was a pleasure being kind of with all of you thank you

Channel: Brookings Institution

Views: 9,048

Keywords:

Id: CzjRhcCuc0I

Length: 89min 15sec (5355 seconds)

Published: Wed Mar 15 2023