Prediction Machines: The Simple Economics of Artificial Intelligence

my name is Nick bime I am a partner at Venrock and I have spent much of the last five years trying to figure out the economics of artificial intelligence and I have to confess I am not yet there I'm still working on it but thankfully our our speaker today a J agrawal has spent a lot of time on the subject has published academic papers on the subject has written a book on the subject that he's going to talk about today I've had the opportunity to work with the Jay on multiple projects and I think was particularly interesting about him is he's both a distinguished academic and a successful serial entrepreneur in the field of artificial intelligence so the academic side he is the Peter Munk professor he's a Peter monk professor of entrepreneurship at the University of Toronto where he focuses on the economics of artificial intelligence he's also a research associate at the National Bureau of Economic Research in Boston and on the entrepreneurial foot front he's founded three entities all associated with artificial intelligence that have been really interesting one of them is about to open up operations in New York so the first the one that's gonna open up operations in New York is called creative destruction lab and I have been an advisor to the organization it started in Toronto it is a an accelerator that is home to a hundred and fifty AI driven startups which for those who haven't been keeping track is the largest concentration on the planet so really interesting achievement and as I said they're gonna be expanding operations into New York the second is next AI which is a program for young AI entrepreneurs in Canada and the third is kindred which is a startup that builds machines that with human-like intelligence and was recently named to the MIT tech reviews 50 smartest companies and to CB insights AI 100 and due in part to his work on AI and in 2017 The Globe and Mail which is a national Canadian newspaper named Jay one of the 50 most powerful people in Canadian business and today he's gonna talk about his new book prediction machines the simple economics of artificial intelligence this will all be simple and easily understandable to everyone by the end of the talk and it was published worldwide this week by Harvard Business Review press so here's how the logistics will work today a Jay will speak for about an hour then we'll have about 15 minutes of Q&A and then we'll have coffee and discussion for everyone together outside afterwards and that you probably saw on the way and there's a big stack of books in the back feel free to take one on your way out and with that as background please join me in welcoming a Jagger wall okay thanks very much Nick thank you and thanks for hosting this and the folks at app Nexus thank you very much for hosting here and I don't know if Kelly's here but she coordinate all my logistics and so I like to thank her and also thanks to Jim Levine and Beth Fisher who were the are literary agents and helped us bring this book to fruition thank you for being here just so I can calibrate how many people here have an AI company suspect there's a few okay that are entrepreneurs and then how just in terms of familiarity if you had to just stand up now or who would feel comfortable standing up and then a few sentences in their own words defining what AI is okay three or four and if those that organizations if you were to characterize how you would where you would place your organization in terms of having an AI strategy imagine three categories category one is we're fully prepared we have an AI strategy in the works category 1 category 2 is we are we're putting a team together to start working on that and category 3 is we have no idea where to start those that are in category 1 just by show of hands ok 1 2 3 people category 2 the sort of medium category okay and the category 3 people all right ok so hopefully by the end of this at least the four the first question which is define a I everybody will leave the room feeling very comfortable that you could define it for anybody and also people most people hopefully will feel comfortable in the second one that it you have at least a sense of where to start okay so I don't need to tell this audience you've all self selected to be here that AI is I think the reason there's a fair amount of enthusiasm about this technology is that it's what economists would call a GPT at general purpose technology that it seems to be everywhere it's now hard to find any market where there's not some AI applications that are rolling in to if they have the productivity of whatever they're doing at the same time there's a fair amount of just angst people trying to figure out what it is and what the implications are how they can deploy it and what that means for humans and so my role this morning and what we try to accomplish with the book is to remove some of this anxiety by way of providing some clarity how can we think of a eyes particularly those people who don't have a computer science background and to do that what we do is we take a topic that is really the domain of computer science and put a layer of a different lens on it from a discipline that we often don't associate with clarity and that's economics and so by putting an economics layer on top of a computer science background in this case we think we can provide a fair amount of just insight into how we can think about development of AI so my background is I'm an applied you can professor at the University of Toronto and teaches Business School and the reason that this matters is because two blocks down from my building is the computer science building and due to some serendipity and in fact a hiring mistake 25 years ago Toronto became really the epicenter of the recent renaissance in AI and so today several of the most powerful industrial AI groups in the world are headed by people who were 10 years ago at Toronto so the person who was the of who originally led the AI group at Facebook here in New York was 10 years ago neuron at University of Toronto the person who leads the AI research at Apple and Cupertino 10 years ago University of Toronto the person who leads the AI research for Elon Musk open AI the billion-dollar opening I Institute 10 years ago University of Toronto and so the reason that that's relevant is that I'm the founder of a program at Toronto that Nick mentions opening up here under the leadership of Professor Deepak Haggai who's here called the creative destruction lab and the focus of the lab is to transition science projects into massively scalable companies and so about five years ago what started as a trickle and turned into a flood was graduate students coming out of this lab this computer science lab the companies are coming from around the world to creative destruction lab but out of this particular lab right on our campus the first one that came was a graduate student who said I'm going to use this new technique for predicting which molecules will most efficiently bind with which proteins for the purpose of drug discovery and then right after him his name was a Heifetz right after him another one came in so I'm gonna use this technology for predicting when which credit-card transactions are fraudulent versus legitimate and then another one came and said I'm gonna use this technology to look at in medical images and predict which tumors are benign versus malignant and then another one came so I'm going to use this technology to predict which automobiles have defects before they roll off the production line and so on and so as we were sitting in the lab myself and two of my colleagues avi Goldfarb and Joshua Gans you know we didn't take a rocket scientist to figure out there's something unusual coming out of this computer science department that the same underlying technique which we now refer to as deep learning and reinforcement learning the same underlying technique is being used to give a material lift to this very wide set of applications and so after a while we just started to document what we were seeing and and then we compiled our notes into what are the general lessons to be learned that are applicable across all of these different cases and that resulted in this book that hit the bookstores yesterday and so what I'm going to do now is just give you some of the key the key points from the book ok so I don't need to say to this audience that a lot of people are feeling that today in AI feels a lot like 1995 felt with the internet Heidi I wonder if we can just give me a little bit of echo if so it feels a lot like 1995 felt with the Internet so most people will remember that 1995 was a real transition year for the internet so we had the internet for at least a couple of decades before 95 mostly used by the military and academics bitnet ARPANET and so on and it was growing over time and then in 1995 there was a big jump that was the year that Bill Gates wrote his famous Internet tidal wave email Microsoft launched windows 95 NSFNET was decommissioned which removed the last commercial limit last barriers to carrying commercial traffic on the web and in August of that year a Netscape went public so a three billion dollar IPO for a company that really had generated almost no profits and right after this so ninety-six early 97 the language around the internet started to change and we stopped referring to the Internet as a new technology and instead began referring to it as a new economy and the point here was that it had permeated so many parts of our lives that we stopped thinking it at the technology and started thinking of it as a different way of interacting in the economy and so CEOs and journalists and entrepreneurs and investors politicians all started referring to this as a new economy everybody referring to a new economy except for one group of people and that was economists an economist said wait a minute this isn't a new economy this is exactly the same economy we've always had in fact we won't have to change a single word or a single page in an economics textbook all the underlying economic models still hold everything's still driven by supply and demand production and consumption prices and costs it's all the same the only thing that's changed is that the relative costs of a few key inputs have fallen dramatically the cost of digitally distributing goods and services the cost of search the cost of communication and that's the way that economists view the world the the thing that economists are quite good at is to take a new technology and strip all the fun and Wizardry out of it and resolve a tech down to a single question and the question is what does this reduce the cost of and surprisingly that often gives us some great insights so for example the heart of Silicon Valley of course is a semiconductor industry and if you ask a computer scientist or an electrical engineer can you please describe to me the rise of the semiconductor industry in their heads they'll have an image like this and they will describe to you the underlying science behind cramming more transistors onto a chip so they'll explain to you you know the number of transistors and chip doubling every 18 months and Moore's law and the implications of more law that's what the technologist will explain to you whereas if you were to ask the same question to an economist can you describe to me the rise of the semiconductor industry this isn't the image they they'll have in their head they'll have this image and the reason that economists will say that semiconductors are so foundational while there's many things happening in the bay that semiconductors are the engine of the innovation economy in in the Silicon Valley it's because the thing for which the cost fell was such a foundational input and the case of semiconductors that thing was arithmetic so economists think of semiconductors as a the rise of semiconductors has dropped in the cost of arithmetic and how many people saw the film hidden figures does anybody remember what the the job title of the women who are the protagonists in that film computers right so they were called computers because what they did was they came to work and they computed and there's a scene in the film where they roll in the big machine the big new computer and everyone's trying to figure out what that means for them okay so when the cost of something Falls three important things happen and by understanding these three things we gained great insight into figuring out how this will affect the economy so we start off by going back to your economics 101 and just the very first thing everybody learns is downward-sloping demand curves and the downward sloping demand curves the key insight is that when something gets cheaper we use more of it and so for example things for which we already used a rhythm take things like you know the types of calculations that they were doing at NASA in the film and the Census Bureau and the military where we were using a lot of arithmetic we just started using a lot more and so it was better faster cheaper arithmetic we started doing more of it so the cost Falls we use more of it we also and this is where things become interesting is we use more of it not just for things that were traditionally arithmetic problems but we start taking things that weren't arithmetic problems and convert them into arithmetic problems to take advantage of the new cheap arithmetic an example of that is photography photography used to be a chemistry problem we solve photography with chemistry and made film but as arithmetic became cheap we transitioned to an arithmetic based solution to photography and music and communication and banking one thing after another we transitioned to an arithmetic based solution okay now on to AI if we were to ask a technologist an engineer a computer scientist can you please describe to me the rise of AI what they would describe to you is the rise of the science and the statistics underlying neural networks and they would talk to you about inputs and outputs in the nodes of the network and the links between the nodes and the weights of those links they would explain to you a development of a statistical process called back propagation and so on but if you were to instead ask an economist can you please describe to me the rise of AI they want to have this image in their head they would have this image now this point people will be thinking wait a minute economists are reasonably single-minded they see everything the same and in some level that's true but this is the key to understanding why economists think of AI as in a category of its own so if you go to CES the Consumer Electronics Show in Las Vegas and you see all sort of rainbow of new technologies in robotics and drones artificial intelligence Oh on you see so many different areas of tech and you say look there's so many things why is AI so special the reason economists will say AI is in a different category that all those other things and the reason is because the thing for which the cost falls is a foundational input into such a wide range of activities that we conduct and of course in the case of AI that thing's prediction so we can think of AI as a the rise of AI as a drop in the cost of prediction and what I find is just a useful exercise is anytime you're reading a magazine article or something about AI replace it as you're reading it with the words cheap prediction and the article will suddenly seem less magical and more practical and you can sort of make sense of exactly what are they trying to do so first of all how do we define prediction here prediction we define is taking information you have to generate information you don't have so that includes what most people would traditionally call prediction like demand forecasting so taking you know the last five years of sales to predict or forecast next quarter sales so that's an obvious form of prediction but a less obvious form that we would still call prediction is something like classification so for example I mentioned looking at a medical image that's the information we have are the pixels in the image and the information we don't have is whether the tumor we're looking at is benign or malignant and so the AI generating that classification we would call prediction okay so prediction is taking information you have to generate information you don't have so here we go we have a substantive drop a a plummeting in the cost of prediction what does that mean what are the implications for business and for society okay so three big implications implication umber one is that downward sloping demand curves when the cost of something Falls will use more of it so that means first of all in all the things where we currently use prediction like demand forecasting supply chain management in Durin's all of these things will simply use the new super-powerful prediction better faster cheaper so we'll just start seeing a is coming in and replacing the traditional statistical techniques we use for doing prediction at the same time we'll also and this is where will become interesting this is where in my view may I on the business side separate from the computer science side becomes an art more than a science which is we will start converting non prediction problems into prediction problems so we'll start using more prediction by converting things into prediction problems an example of that so just like we did an arithmetic an example of that error is driving this is the one that everyone's most familiar with so driving we've had autonomous vehicles for a long time thirty or so years depending on how you count but we always deployed our autonomous vehicles in a controlled setting so a factory or a warehouse and the way we did it simplified version is that an engineer would have the floor plans of let's say a factory or warehouse and they would program a robot to move around the factory floor and then they'd give the robot a bit of intelligence they put a camera on the front and then they would tell the robot if somebody walks in front then stop if the Shelf is empty then move to the next shelf if then if then so a series of logic of rules that gave the robot some intelligence the problem was you could never take that robot out of the controlled environment and put it in an uncontrolled environment because there were too many ifs in fact there's an infinite number of ifs if it's dark if it's raining if a child runs up to the edge of the road if a car comes with oncoming car with it left to turn blinker on if-if-if and all the ifs are interacted and because there's an infinite number of ifs in an uncontrolled environment as recently as six years ago the experts in the field were saying we will not have an autonomous car on a city street in our life time because we simply cannot program all the IFS until people in the in the machine learning field reframe the problem and said rather than programming an infinite number of ifs what if we changed the problem to instead making one prediction and that prediction would be what would a good human driver do and so the simplified version of how a autonomous vehicle works is a way to think about it is imagine a car and in the car you put a human and human puts a human in the driver's seat and imagine putting an AI in the passenger seat and so what we do is we tell the human who's sitting in the car drive just drive drive for a million miles and so the human sits in the car and starts to drive and as they're driving they have data coming in through the cameras on the front of their head and in the microphones on the side of their head and as the data comes in we process the data with our monkey brains and then we take an action and the actions are very simple it's a very small set of things we can turn left we can turn right we can break and we can accelerate that's it we have many many ifs coming in and we have a very small number of events okay now on to the AI so imagine the AI and sitting beside the human the AI doesn't have its own eyes or ears so we give it its own sensory inputs cameras radar lidar around the car and the way to think about it is imagine the ifs are coming in the data is flowing in as you're driving and every fraction of a second the AI is looking over and trying to predict what will the human driver do in the next second and so in the beginning the AI is a not a very good predictor we they have big confidence intervals on their predictions meaning they're not very accurate and so they say I think she's gonna turn left I think she's gonna go straight I think she's gonna break and then something happens either she turns left where she doesn't turn left and every time they makes a prediction and then some and then they get to the AI observes what the human driver does if they were right they double down on their model if they're wrong the take their model and and then they make a different prediction potentially the next time okay and so as they're driving in the beginning the a ice confidence intervals are very wide they're making a lot of mistakes but as they drive and they learn and they correct their mistakes the confidence intervals get smaller smaller smaller smaller and smaller until at some point the AI is such a good predictor of what the human driver would do that we say the AI can just do it itself and so the AI is become a prediction machine for driving and that is really where I think a lot of the enthusiasm is it's converting problems that were traditionally prediction problems into prediction problems to take advantage of the new cheap prediction so driving is an obvious one but we've done it in so many other areas translation translation used to be a rules-based problem we had linguists who were experts in the rules of translation and all the exceptions and they would do translations but we've converted translation into a prediction problem and now for those of you who for example used Google Translate even between a year ago and today the improvement is significant and it feels like it's not too far away that will have a commercial-grade translator that's based entirely on prediction rather than rules okay and our lab a creative destruction lab Sanok mentioned that you know this is where we have all these AI companies have come in the last five years and as far as we know the creative destruction lab is now home to the greatest concentration of AI startups of any program on on earth we they're coming in from all over the place and these AI companies each one is you know working on solving a prediction problem and now we're having a lot of corporates large companies come in who want to better understand AI and the an interesting group are people who are heads of HR and the in it so the the common conversation we have is a you know head of HR of this large corporation or that corporation will fly to Toronto and they'll say you know we're trying to learn more about AI I need to know for recruiting what types of skills should we recruit for how should we train our staff in order to prepare them for AI we need to know for the other parts of our company the sales department the manufacturing compartment the design compartment but not for my compartment department because I work in HR HR is a very human business that's what they say to us HR is very human but for the other parts of the company we need to learn about AI for recruiting now most of you know where this is going one by one AI companies are transforming the HR process into a series of predictions so what do HR people do well the very first thing they do is hire they recruit recruiting is effectively now a prediction problem we get a series of resumes and cover letters and interview transcripts and then we predict from a set of applicants which ones will be best for the job once we've hired people the next thing we do is promotion what's promotion a prediction problem we have a set of people working the company and we need to predict of those people who would be best in the next level up then what our next issue is retention well you let's say we're at a 5,000 or 10,000 person organization we need to retain our best people or what's that it's a prediction problem we need to predict which of our stars are most likely to leave and what types of incentives would be most effective to keep them and so on one by one these positions are being converted into prediction problems so that a eyes can tackle them okay so item number one when the cost of something Falls we use more of it and we use more of it for both traditional things and we also convert new things in this case into prediction problems to take advantage of the better cheaper faster prediction here's the number two and number three when the cost of something Falls it affects the value of other stuff in economics language we call that cross price elasticity basically what it means is you can think of things as being if they're related to the focal thing as complements or substitutes so think of coffee if the coffee the cost of coffee were to fall these compliments to coffee the things we use with coffee like cream and sugar the value of those things would go up because of the cost of coffee Falls will start consuming more coffee and because we're consuming more coffee we'll also consume more cream and sugar and therefore the value of cream and sugar will go up or if the cost of golf clubs Falls the value of golf balls goes up alright and so those are complements the value of complements go up the value of substitutes go down so in the case of the cost of coffee falls at the margins some people switch from tea to coffee the value of tea Falls because the demand for tea diminishes tea is a substitute cream and sugar are complements now how does that work in AI we can take any task and break it down into these components any task and so it doesn't matter what part of the organization you work in there'll be a series of tasks that you do and every task can be broken down into these bits so an example let's say that as I'm giving this talk right now I banged my knee on the lectern and three days from now my knee is really sore and so I go to the doc and I tell her my knees sore and so she says okay she asked me a bunch of questions maybe she sends me for an x-ray that's input she's collecting input then she makes a prediction and her prediction might be I think with 90% probability you've bruised your knee with 10% probability there's a hairline fracture then she applies judgment and her judgment is the way to think about it is how costly would it be for this patient if they actually have a bruise but I mistakenly treated as a fracture versus if they actually have a fracture and I mistakenly treat it as a bruise that's her judgment she's taking all the things that she knows about me and is trying to figure out the cost of a mistake that's judgment then she takes an action so her action might be saying okay I've decided I think it's a bruise I mean treated as a bruise put some ice on it raise your leg and if it's still hurting in a week come back and see me and that's the action then there's an outcome the outcome would be a week later let's say my knee is better I'm good to go and so we've learned that she was right and that becomes feedback data which we use that in this case strength strengthens the model or let's say it was wrong a week later my leg is worse and then that's that's feedback data and we update and change the model for the next time okay so any tasks we can break into these components and we find this very useful for designing strategy around AI and for thinking through what are the implications for jobs and the economy and so on here's how if you look at this diagram it's very clear what is the substitute for machine intelligence so as machine intelligence increases as the cost of machine prediction Falls what's the substitute for machine prediction it's the Box in the middle human prediction the value of human prediction will fall as the capability of machine prediction increases so we are quite poor predictors we're slow we're noisy we have all sorts of systemic biases in our prediction it's being very well documented in books like Danny Kahneman Thinking Fast and Slow Airy Ollie's predictably irrational all these things document how terrible humans are at making predictions but we still make them all the time and so as the capabilities of machine prediction increase the cost of machine prediction Falls the value of human prediction will plunge so our human prediction capabilities become less and less valuable because the machines can do it so much better faster cheaper and that's the part I think the press has been fascinated with and that's led to a mischaracterization of a complete wiping out of the role of humans what they miss when they focus on that are all the other boxes the other boxes are the complements they're the cream and sugar they're the things that will increase in value as the price of machine prediction Falls the one that the press has talked about is the first one input so how many people heard the phrase data is the new oil okay most people that is effectively talking about the first box the input we've always had data that data we've had a lot of that data we've had for a long time why is it the new oil what makes it new what makes it new is it's way more valuable the same data is way more valuable today than it was 10 years ago why because the cost of prediction has fallen and so the value of the data has gone way up hey it's a compliment the data is a compliment it's more valuable now because we can do more things with it because prediction is better faster cheaper so the press has done a good job of talking about the input box but the press has not done a good job so far of describing the other bits human judgement ai's do prediction they don't do judgment they don't know what to do with their predictions we have to give them guidance on what to do with the predictions we decide that's judgment so for example the doctor who's deciding what's the cost of a mistake she's using her judgment and so what's interesting is that we're always applying our judgment but the value of our human judgment goes up as the cost of prediction goes down as we start getting better faster cheaper predictions as the fidelity of our predictions increase if machine predictions increases the value of our human judgment goes up because we're applying our judgment to much better predictions okay the next one is action that predictions are generally used to inform an action what action should we take companies you know operating companies have owned very often now with the incumbents they only action that's a valuable asset it's not just the input that the data that matters it's not just data is the new oil your actions are valuable because now you'll be able to take better actions because you're basing those actions on higher fidelity predictions so in our lab where we have all these AI companies one of the issues is that they they build and sell predictions but they don't only actions and so they have to sell their predictions to someone who does own the actions because without the actions their predictions are worthless so actions are a compliment the value of actions goes up where we have the most amount of negotiations with all the AI startups a lot of them who are selling their predictions to larger enterprises is on feedback that's the gold dust when we take an action we find out later whether that action was a good one or a bad one and that's how we learn to update the AI so all our AI startups that are trying to license their predictions output sell their predictions to enterprises are all trying to get their hands on the feedback data and the company some of them realize some of them don't that that's the gold dust that data in very is to some extent much more valuable than this data this data you're using the oil analogy you use it and it's you burn it it's gone in other words you use it to train your AI the first time but once you've trained your AI there's a few caveats here but the main one is you broadway think of it is you use it to train your model and then it's done the value is gone the ongoing value comes from the feedback data that allows the AI to continue to learn okay so the key points here is in thinking about strategy and thinking about the implications of jobs is that the value of human prediction Falls but the value of all these complementary assets go up and a useful way to think about this an example is think about when spreadsheets rolled into town and accountants the accountants originally you know you could think of them as having two broad skill sets one of them was that they would you know type in the numbers and and and so they had a you know we valued their ability to type fast and to add fast and then the second skill they had was to ask good questions so let's say they were doing a model to estimate the present value of some asset and then they might ask a good question by saying well what would happen if interest rates went up by 1% or what would happen if a sales went up by 4% in in the fourth quarter and so they would ask a question but then they'd have to as soon as they ask the question they have to start at the beginning and retype everything and REE add up the whole set of numbers in order to test that new scenario when spreadsheets rolled into town the value of the human ability to type fast and add fast went down because the machine could now do that but the value of being able to ask good questions of your data went up because it became much quicker and easier to ask a question so if you were good at asking questions of you know doing scenario analysis your the value of you as an accountant went up so if you were an accountant where your key skill was asking good questions your value went up if your key skill was adding fast your value went down okay so if AI is just prediction if the current renaissance and AI is really all about lowering the cost of prediction and it's not dolores from west world or c-3po then why is there so much fuss why is there so much fuss about AI if all it is is a prediction tool the answer is because prediction is a key input to decision-making and decision-making is everywhere it is riddled throughout our business lives and our personal lives and so this is the structure of the book it's in five sections at section one we talk about prediction we explain you know what's so interesting about the new method of prediction compared to traditional prediction tools and statistical techniques section 2 is about decision making and while a is new decision theory is old we have about 50 years of a well-developed decision theory and so we can actually have a pretty good sense of what happens when we take this new prediction technology and put it inside a process of decision making that we've understand quite well that has been studied for a long time so that's section two you can think of them as the two academics parts of the book then we get into the practical parts section 3 is on Tools which is the actual building of AI tools and in that section I'm not going to I guess I'll get to give it a couple minutes on the tools is the basic idea here is that we can take any any workflow so a workflow is inside an organization turning an input into an output that's a workflow so you can think of a line of business as a workflow and we take the workflow and break it down into tasks and every task is predicated on a decision or a couple of decisions and a is do tasks they don't do workflows they don't do jobs they do tasks that are predicated on a decision so an example this is an article many people here would have read by the the time the CFO of Goldman and in this article they open up with this dramatic statement by saying at its height back in 2000 the US cash equities trading us at Goldman employed six under traders today there are just two left okay that's the dramatic opening but later down in the article they have or more interesting sentence so they talk about now they're working on more complex areas of trading like currencies and credit they want they're trying to emulate as closely as possible you can replace that with predict predict what a human trader would do goldman has already mapped 146 steps taken in any IPO okay so the idea here is that they've taken the IPO workflow and broken into 146 different tasks and so what we do and we're when we are building a eyes is we take each task and we estimate the return on investment the ROI for building an AI to do that particular task and then we the way to think about is we just stack rank order the tasks which ones have the highest ROI for building an or buying an AI we start at the top and work our way down so when right now this is not for the you know the googles and Facebooks who are already well into this but you know the 99.9% of other organizations were just getting into AI this is a approach we used to getting started we map out the tasks we've ranked them and then we start at the top start working our way down okay and so sometimes there are companies that that come to the creative destruction lab large companies and say hey we've got three AI pilots at the company are we at the frontier of AI and so just to calibrate you know Google now has almost two thousand a AI tools under development and they're probably the high-water mark or possibly by do so we've built this thing it's in the book it's called the AI canvas and we have found this a very useful tool for getting companies started with AI and so what we often do is let's say there'll be 50 people in an off site and they come from all parts of the company they're usually vice-president level and above and they sit in tables of four and halfway through the day they go into these breakouts and they fill out this page and very often there's a there are the whole set of people not a single person in the room has ever written a line of code so they don't have to have any technical background but they simply go through their workflows and they pick out tasks and they say okay what if we built a prediction machine and this is the prediction so they specify the prediction and they say with that prediction here's the human judgment that would be applied to the prediction this is the action that the prediction motivates and then here's the outcome that were results from that and then the training data the data we use to run the AI and the feedback - so the AI can learn and so people with no technical background fill out this thing and by the end of the day we have an organization has a list of twenty a eyes that are at being just designed at a very conceptual level by you know senior people of the organization and that's a we found a useful way to help them get started and just create a map of the organization of where they can start building a eyes okay so the last thing I'm going to talk about is strategy so most of the time when we build a is we are building tools that are very specific and their role is to just make a process more efficient in the service of executing against the organization's strategy so just like Microsoft Word or Excel an AI is a tool to make you more productive executing against a given strategy but occasionally an AI can so fundamentally transform the economics of a process and that they changed the strategy itself so this is what the popular press often refers to as disruption and so what I'm going to show you now is the process that we use for helping to give us guidance on which a is might lead to a disruption a strategy change for the organization our view is that the main thing when would think going through what I'm about to show you is to develop a thesis on time so in other words the thinking about the time it will take for what I'm about to describe to happen will very much influence the investment decisions made today in other words whether something will happen in three years or ten years as very different implications for the investments you make today so if I were giving this lecture even two years ago most of what I'm about to say would be if you know if this were to happen then wouldn't this be interesting or if that were to happen then imagine the possibilities over the last 24 months as most people in this room probably know already we've had way too many proof of concepts to be thinking about this as an if any more in other words we've had these envision so you can think of that as basically machines being able to see giving them eyes in natural language processing so allowing them to come you know comprehend words and language effectively predict what those characters are trying to communicate in motion control robotics being able to do things so this is no longer a discussion of if it is a discussion of when so we know it's possible now it's all about turning the crank and getting those predictions up to commercial grade levels of accuracy so we're just we are in most cases now in the turning the crank mode as opposed to in the wondering if it's even feasible so here's the thought experiment we we use a process we call science fictiony and but it's not science fictiony in the sense that you can sort of sit behind a desk and blue sky and think a eyes can do anything it's a very specific kind of science fictiony the thought experiment is imagine a radio knob and you can turn the radio knob but instead of turning up the volume when you turn the knob you're turning up the prediction accuracy of an AI so that's the only thing that the only parameter you're allowed to move is turning up the dial on the prediction accuracy so here's the here's the thought experiment everybody's being shopping on Amazon so we use this as an example that will everybody will build to imagine and when you shop on Amazon you know you are faced immediately introduced to an AI that's the recommendation engine and Amazon the where they recommend oh we think you might want to buy this you might want to buy that and right now for myself and my co-authors on average that recommendation engine is about five percent correct meaning out of every 20 things it shows us we end up buying one of them which five percent accuracy might sound lousy but it's not too bad when you think that there's millions of things in the Amazon catalog and it's going out and pulling out 20 of them showing them to us and we're buying one so as you know we go into Amazon the recommendation engine shows us some stuff and we shop around and we see things we like we put in our basket we pay for it and then the order is arrives on somebody's tablet in the fulfillment center at Amazon and the Kiva robots are moving around they bring the stuff up to the human the human pulls that out of the shelf and puts it in the box puts a label on the box ships it to your house it arrives at your door and you bring it inside and that's how will you shop at Amazon and when you can we can summarize that by calling that method shopping then shipping you shop for the stuff and then Amazon ships it to you okay so here's now the thought experiment is imagine that recommendation engine which most of us just breathe by we don't put a lot of thought into it when we see it on the website imagine that you know when we don't have to imagine this is just happening is every day at Amazon the people in the machine learning group are working on turning the knob and so let's say right now that knob is at a 2 out of 10 and they are working they're improving their algorithms they are testing some you know different approaches they are acquiring data assets like buying whole foods so they can learn more about how you and I shop offline each time they do that cranking the knob so maybe there are two maybe they only get it up to a 3 and then a four and then a five and they are working entirely focused on turning the knob increasing the prediction accuracy as they increase the accuracy they don't have to get up to spinal tap levels maybe it's you know a 6 out of 10 or 7 out of 10 there's some number where when they reach that number let's say it's 6 out of 10 they get right somebody at Amazon says you know we're so good at predicting what they want why are we waiting for them to order it let's just ship it and so just go through the thought experiment as you turn the knob you get up to some level where Amazon says let's not wait anymore let's just ship it so let's say they ship you a box of stuff and you the box arrives you open it and you take out the things you want so let's say you want 6 out of the 10 things and you leave 4 and you might say well wait a minute why would they ship you things if they know that you're not going to take all of them you're only going to take 6 in return for from their perspective they are potentially significantly increasing a share of wallet so whereas they might have sold you two of those things and you might have bought 4 of those other things from their competitors either online or offline now they're selling you all those six and maybe they would have you would have only bought five of them and the sixth one is something you kind of wanted but not really but now that it's right on your doorstep you think you know I might as well keep it furthermore now you've put four things back on your porch in the box that things that you didn't want and now Amazon does it's in their self-interest to invest in a fleet of trucks that are going to drive down your street once a week and pick up all the things that they deliver to you and your neighbors that you didn't want in order to lower their cost of handling the returns now I'm not sure whether Amazon will ever do this it's not like they've never thought of it this is patent they filed on an approach they called anticipatory shipping and they've already started testing it in some markets with clothes okay so we just with clothing items in some markets but the point here is that the thought experiment is very powerful the thought experiment is what happens remember the only parameter we moved was turning the knob and what's so interesting about turning the knob is that as you turn the knob it's like nothing happens nothing happens nothing happens in other words we see it is getting a little bit better in terms of recommending stuff as we're shopping but we don't even really notice nothing happens nothing happens nothing happens and then it hits some threshold where all of a sudden everything changes they change the entire model from shopping than shipping to shipping then shopping we shop on our front porch instead of on the website and they vertically integrate into us into a fleet of trucks and so on and you can go through that thought exercise with imagine if you could predict insurance claims or bank loans or acceptances into MBA programs if you could crank that knob up to certain length it there's some number where you cross the threshold and it creates opportunities for a completely different approach a so called disruption okay so when Google announced last year that they were moving from a mobile-first strategy to an AI first strategy is that just marketing or does it mean something it means something so when they said before they were mobile first what mobile first meant is not just they want to be good at mobile everybody wants to be good at mobile mobile first means that they will be mobile at the expense of other stuff in other words they will sacrifice their website in physical stores or whatever in order to be mobile first that that's their number-one priority so what does it mean to be AI first it means putting that dial at the very top of your strategy priorities it means that's your number one priority that you will trade off other things in the case of Google they will trade off short-term user experience they will trade off revenues they will trade off privacy in order to crank the dial when asked this question on Quora what does it mean for Google to be AI first Peter Norvig is the research director responded and the essence is down at the bottom he says with information retrieval that's when we do a query on the Google site with information retrieval anything over 80% recall and precision is pretty good not every suggestion has to be perfect since the user can ignore bad suggestions with assistance there's a much higher barrier you wouldn't use a service that booked the wrong reservation 20% of the time or even 2% of the time so an assistant needs to be much more accurate and thus more intelligent more aware of the situation that's what we call AI first so from a computer scientist perspective that's how he defines AI first from an economics perspective we would add to that the trade-off when you make AI first you make other things second and third and fourth and so they're putting this as the priority turning the knob one example of making it a priority is reshuffling the deck putting moving people who were outside the CEOs office somewhere else and moving the people who are working on AI right in the same office as the CEO so a year ago the Google brain team of mathematicians coders and hardware engineers sat in a small office building on the other side of the company's campus but over the past few months it's switched buildings and now works right beside the lounge like area where the CEO and other top executives work ok so to conclude here when people come to our lab what we notice is this distance and the distance is they arrive at the lab and say I get it I get the Amazon recommendation engine and you know Siri remember Sirius is just a prediction machine Siri doesn't understand what you say when you're speaking into Siri Siri hears an audio signal and predicts the vector of words that you're saying and then predicts the response that you want from what you said and so people say I get it these are very clever they're amazing but they're not transformational these are not transforming any business or any economy but on the other hand this is a chart of venture capital into AI you know a very steep curve in the last two quarters of the Obama administration the White House released for reports on how to prepare the American economy for what was coming around the corner with AI as far as we know what we can find it's the only technology where the White House has released four reports in two quarters since the Second World War then there was the Google announcement followed by a series of other companies announcing moving to an AI first strategy then last July the government of China announcing their strategy incredible amount of resource to compete in AI and with the goal of by 2020 catching up in some fields of AI by 2025 dominating in a few subfields of AI and by 2030 dominating across every field in AI was the aspiration then in September the President of Russia announcing AI is the future not just for Russia but for all of humankind and the country that leads an AI will rule the world then later on in that month we hosted a Toronto what I think is still the greatest gathering of economists to meet on the economics of AI so the former Treasury Secretary Larry Summers the former chair of President Obama's council the Economic Advisers Austan Goolsbee the chief economist of Google Hal Varian the chief economist of Microsoft Susan Athey professor Danny Kahneman a Nobel laureate and others gathered to talk about the economics of AI because computer science that gotten so far ahead of economics on this subject at the end of the meeting Danny Kahneman who's the author of the Thinking Fast and Slow concluded with this he said I want to end on a story a well-known novelist wrote to me some time ago that he's planning a novel the novel is about a love triangle between two humans and a robot and what he wanted to know is how would the robot be different from the people I proposed three main differences one is obvious the robot will be much better at statistical reasoning the second is that the robot would have a much higher emotional intelligence so we think of us as being we will always be better than machines and emotional intelligence professor Kahneman says no not so a eyes are already better in some domains they're better at detecting minor facial changes to detect changes in moods they're better at detecting minor changes in audio to detect when someone's voice is reflecting that they're getting happier or sadder or angry or or jealous the third is that the robot would be wiser wisdom is breath wisdom is not having to narrow a view that is the essence of wisdom it's broad framing a robot will be endowed with broad framing when it has learned enough it will be wiser than we people because we do not have a broad frame we are narrow thinkers we are noisy thinkers and it is very easy to improve upon us I do not think that there is very much that we can that we can do that computers will not eventually learn to do so on the one hand we have all of we have these things that we these AIS that we see that look neat but they're not transformational they're not disrupting industries and on the other hand we have all these learning people and powerful people who are making claims and implying that AI is going to have this spectacular effect on the economy how do we reconcile these two things and in our view the single thing that reconciles these is time it is having a thesis on time that the dial in most applications is sitting there at a two out of ten and people are working 24/7 on cranking the dial to three to a four and it's moving faster in some domains and slower in others it's not just doing the last time we had a big technological revolution some companies had a good thesis on time some had a underestimated how fast things would move in this current revolution it's not just the googles and Facebooks and Tesla's and apples it's also older economy companies that are making bets as everybody here already knows companies like GM John Deere and so on we had one company what one of the fellows at the creative destruction lab built a company in less than a year less than 20 people had virtually no revenues and was acquired by TD bank for just over 100 million bucks and so to sum up there was this nice quote that came out not that long ago from Robert work who's a former Deputy Secretary of Defense and he's referring to the race between US and China Nai and the and the incredible amount of resources the Chinese government's putting behind winning in this and he uses the phrase this is a Sputnik moment of course referring to the spate the Soviet launch of Sputnik satellite and they kicked off the the space race and the you know the creation of NASA and so on and what I think here like my co-authors eye is that this is not just a Sputnik moment on the for defense it is a Sputnik moment for all of us the for the people who are right now in the position of leading organizations or running organizations you know we get one of these once in a generation they don't come every year every few years is once in a generation something like this comes along that has the kind of potential that this has for the for the the kids in the audience I think to a large extent you know you will be the ones that will have the creativity of how to really apply prediction machines and ways that that the rest of us don't even think of but the main point here is that in our view this creates an opportunity like most of us will never have again in our professional careers and so for some people that will pose a you know it's a set of of opportunities to into seize and pursue at the same time of course there will be challenges the biggest one will be to make sure that as a society everybody benefits because it has the potential to you know to be to be to change a lot of structure socially that's it I'll end it there thanks very much so I can take you to some quiet Niki's questions questions I'm happy to take yes please oh sorry there's a mic coming and just because they're recording it and if you don't mind just introducing yourself my name is Elmo I'm Patrick Slattery and my question is where do you if we go back to the one diagram you're presented on prediction and judgment and action do you see a role for innovation in the prediction process itself eventually or is it something that's more constrained to judgment and action well and if there is a role for prediction in for judgment and prediction will machines be best at doing that or is that a human complement to prediction so can you repeat the first part I didn't follow what you meant by innovation in prediction so if I think of an innovation or create you think of it as creativity I can I can see a role for creativity and judgment and inaction do you see in the long run a role for creativity in prediction so I don't but I might be missing something in the sense that I think of that out in the world in nature there are probability distributions that describe the phenomenon around us and the AIS will simply be better than we are at understanding what those probability distributions are so that's not really a matter of creativity it's a matter of getting higher resolution on understanding the probability distribution yes please hi thank you so much this is super neat really enjoy it hopefully I'll get to read the book soon so clearly you're a fan of Danny Kahneman so I assume that you buy into behavioral economics right so that kind of like cuts with Oh human beings are rational maybe they're not in fact they definitely aren't and to me there's kind of a parallel and this is where the question comes like I want to hear kind of your opinion cuz to me right now we make very strong assumptions about AI or deep learning it's like okay if you have enough data and particularly if it's a vision problem or if it has particular correlations then boom you're gonna have really good precision recall and you're gonna have like a magical black box but I just my sense is that we are kind of in the rational world like these machines the assumption is we have these like neural nets and we have enough data and boom it'll work out and to me it seems a lot like the classical economics like this is just gonna work out and I wondered what were your thoughts around maybe you know these neural Nets only work a certain way because we have this magical vision data that happens to correlate spatially and things kind of like correlate spatially and it works properly and what are your thoughts around if that's true then what would be that analogous behavioral economics or or and a little more concretely art being a little over exuberant about the possibilities around deep learning sure so I definitely think there's room for being over exuberant in the sense that first of all people and people imagining a eyes can do some things other than prediction or effectively that's all they do furthermore a eyes can only really do good predictions where they have good data and so I think that was part of your other the other part of your question is that when you're in domains where we don't have good data the a eyes can't make good predictions the one thing I would say is we're a I sometimes feel magical is because they are able to find patterns and data that traditionally we we either couldn't use or didn't have access to so with just the falling cost of sensors we can censor up so much stuff image data audio data in other words is not just the lines of data we had in an Excel spreadsheet that we used to use for making predictions of things we can use so much more multimodal data of all kinds of types that can complement the process of making a making a prediction and furthermore in the old methods we used to have - before we started making a prediction we used to have a have to create a model in our head of what data we're going to use to predict what outcome now we can virtually take a kitchen sink approach where we give it everything and we let the the Machine figure out what relations you know what's related to what to make a prediction so I think on the one hand we probably have much more predictive capability than we many people realize because of the way the prediction machines work but on the other hand they're certainly not a silver bullet and there's you know one of the things that we in that first section on prediction is we have a whole section on all the limitations of the types of things that prediction machines are poor at because they they don't have a model they're basically statistical correlations and they rely on the underlying data I'll take two more questions and I know people have to get going and I'm happy to stick around for offline questions afterwards so in your diagram you've labeled a whole lot of boxes as complements to prediction or inference right and it's clear for me how some of those complimentary boxes you can establish monopolies around and extract rents for but there seems to be absolutely no reason why you should expect that those boxes will be you know uniquely retained by human beings right every single one of those boxes is subject to research in a subfield of AI generally and as they become more valuable relatively you can expect the research effort to go into them right so you know a lot of people say everything will be great and human beings human judgment and some will become more valuable but you know what reason really do you have to expect that that's true apart from that it's like comforting ok so that's a great question I mean I will I will attack it with just one example which is the judgment box because it's some sense that's the place where we seem to take the most refuge in ok we're safe from the machines because we have judgment and they don't and so the question was largely you know are we just dealing the first inning and AI research is moving into these other boxes so at least in the moment as long as AI is prediction that's all it is is just prediction but the point is if a eyes see enough examples of a particular type of judgment they can learn to predict that judgment so an example of that is driving when when we're driving the AI is is is effectively baking in the prediction of our judgments so that when we are approaching a yellow light the AI is learning inferring our judgment of how we look whether we're gonna step on the gas or step on the brake depending on whether it's raining and how far we are and how from the light and how fast we're going and our judgments being baked into there to the decision and so it's true that things today that we that we call judgment some of those things a eyes will eventually learn to do because they'll get enough examples of us making that judgment in order to predict them that you know I think of this to some extent you know like those Russian nesting dolls that once the a eyes are good at predicting a certain type of judgment we now can focus on an on a new set of problems a new type of problems where our judgment is useful in the domain you know in that new don't domain the question is you know at some point you run out of nesting dolls and there's nothing left your guess is as good as mine but what I would say is the thing I think that we're very poor at like that our monkey brains are not good at is in an is in anticipating things we've never seen so just the same way that if we would have asked people 200 years ago when 47% of the population or even almost a hundred years ago forty seven percent of the population working in agriculture if we would have said imagine a world where forty five percent are no longer in agriculture they're doing something else that only like less than two percent of us are working in agriculture what are those other forty five percent going to do nobody would have raised their hand and said well they're going to be at a company called AppNexus or they're going to be game developers or they're gonna like nobody would have done that because nobody would even imagine that such a thing would exist and I think that's what we're very poor at we're very port imagining that when we have high fidelity predictions what are all the other things that we could be doing for example most people in this room would say our healthcare systems no it's not great but it's not bad I think 10 years from now people will go back and say I can't believe people were living in those two horrible conditions with that terrible health care system that they had back in 2018 because we'll be able to do so much more so much more efficiently so much more intelligently because the machines are able to handle a lot of the same a space exploration same with all sorts of things where we feel like we're at the frontier and in fact we're just scratching the surface take one more question wrap it up please the costs from so my question is in around picking problems that we want to subject to AI now I mean we've we've dabbled a little bit and looking at deep learning and so on to solve different problems there are of course lots of constraints and so is the assumption that no matter what the problem is that eventually we will be able to turn that dial or are there some ways in which meet we can make better decisions as to what problems to go after and what not to go out yeah sure so that's the process of estimating they return on investment for building an AI so in other words you know how long will it take how much will it cost how much data do we need to get the to turn the dial far enough for this AI to give us you know some lift in performance so in the book we have described some processes of how to do that it's I think we go far enough in the book to help people get started just far enough for non-technical people to get started to then know when to be able to bring in the technical people to provide more definitive answers in terms of like how much data will we need in order to do this or you know what types of sensory information will we need to collect in order to do that and so there's no single answer to your question but there is a reasonably step-by-step process for coming up with the answer of where to start and which a eyes to focus on first okay great thanks very much [Applause] [Music]

Channel: The Artificial Intelligence Channel

Views: 71,585

Rating: 4.8528733 out of 5

Keywords: singularity, ai, artificial intelligence, deep learning, machine learning, deepmind, robots, robotics, self-driving cars, driverless cars, OpenAi

Id: Q4o56nufXTw

Length: 72min 33sec (4353 seconds)

Published: Tue May 29 2018

Reddit Comments

[u/RavlaAlvar]

Isnt decision just another prediction process?

[u/Stormbane]

Agarwal attests that judgment and learning are complementary products of AI prediction. As the cost of prediction decreases he believes the value of human judgment and learning will also increase. Yet he also claims any task can be deconstructed into some form of prediction.

To me it seems that judgement and learning are simply a more recursive form of prediction. Am I being naive in this belief? Agarwal runs the largest AI startup accelerator in the world yet it seems to me that he displays severe cognitive dissonance. As I talk to more people about AI I see this same cognitive dissonance from many well studied intelligent people. Why is this?

[u/Laserdude10642]

I was skeptical when he said he was an economics professor but I found the talk very down to earth and interesting

[u/Cherubin0]

In the background: OpenAI next to the apple of abuse.