AI & Logical Induction - Computerphile

today I thought we would talk a bit about logical induction paper out of the machine intelligence Research Institute very technical paper very mathematical and can be a bit hard to get your head around and we're not going to get too far into it for computer file I just want to explain like why it's cool and why it's interesting and people who are into it can read the paper themselves and find out about it the things about logical induction as a paper is that it's not immediately obvious why it's in a nice safety paper but like my perception of the way that Mary the way to the machine intelligence Research Institute is thinking about this is they're saying they're thinking like we're going to be producing at some point artificial general intelligence and as we talked about in previous videos there are hundreds of ways really weird subtle difficult to predict ways that these this kind of thing could go very badly wrong and so we want to be confident about the behavior of our systems before we turn them on and that means ideally we want to be making systems that we can actually in the best case that we could actually prove theorems about right things that are well specified enough that we could actually write down and formally prove that it will have certain characteristics and if you look at like current machine learning stuff older like deep neural networks and that kind of thing they're just very opaque do you mean by that because we don't necessarily know exactly why it's doing what it's doing yeah and also just that the the system itself is very is very complex and very contingent on a lot of specifics about the training data and the architecture and like there's just yeah effectively we don't understand it well enough but it's it's like not formally specified enough I guess and so they're trying to come up with the sort of mathematical foundations that we would need to be able to prove important things about powerful AI systems before we were talking about hypothetical future AI systems we've got time to print more stamps so maybe it hijacks the world's stamp printing factories and when we were talking about the stamp collector it was really useful to have this framework of an agent and say this is what an agent is it's a formally specified thing and we can reason about the behavior of agents in general and then all we need to do is make these fairly straightforward assumptions about our AI system that it will behave like an agent so we have idealized two forms of reasoning like we have probability theory which tells you the rules that you need to follow to have good beliefs about the state of the world right and we have like like rational choice theory about what what rules you need to follow and it may not be it may not be like actually possible to follow those rules but we can at least formally specify like if the thing has these properties it will do this well and all we're trying to do is they're thinking if we're building very powerful AI systems the one thing we can expect from them is they're going to be able to do thinking well and so if we can come up with some formalized method of exactly what we mean by doing thinking well then if we do if we reason about that that should give us some insight into how these systems will behave that's the idea so let's talk about probability theory then this is where we should have a demo pretty sure backgammon has nice yeah no this is a fun one how's it - I don't know how to play backgammon anyway here's a dime so basic probability theory right I roll the die now it's under the cup and I can ask you what's the probability that that is five one and six right one in six because you don't know it could be anything there was a time when people reasoned about the probabilities of things happening in a very sort of fuzzy way you'd be playing cards and you'd be like oh you know that card has shown up here so I guess it's less likely that he has this hand and people would yeah and like there was some sense of like clearly we are doing sort of what we're doing here is not random it's meaningful to say that this this outcome or that outcome is more or less likely by a bit or by a lot or whatever but people didn't have a really good explanation of how that all works until we had probability theory right we didn't it was an assistant right right and like practically speaking you often can't do the full probability theory stuff in your head for a game of cards especially when you are trying to read people's faces and like stuff that's very hard to quantify but now we have this understanding that like even when we can't actually do it what's going on underneath is probability theory so that's one thing right straightforward probability now let's do a different thing and now I'm get it now I'm gonna need a pen now suppose I give you a sentence that's like the tenth digit after the decimal point of the square root of 17 is a 5 what's the probability that that's true well it's a much more difficult problem one in 10 right one in 10 seems like a totally reasonable thing to say right it's gonna be one of the digits we don't know which one but if I you've got the paper here you know you can do like division you you could do this if you had long enough like if I gave you say an hour with the paper and you could sit here and figure it out and let's say you do it and you you come up and it seems to be like three I don't know what it actually is I haven't done this calculation let's do the calculation about expression oh I've don't run way around on 2 3 4 5 6 7 8 9 10 it's a 6 oh it's a 6 ok I picked that out of nowhere so suppose you didn't have a calculator you were doing it on paper I gave you you know half an hour an hour to do the like long division or whatever is you would have to do to figure this out what's the probability this statement is true now what do you say I'm gonna say zero right but you've just done this in a big hurry on paper right you might have screwed up so what's the probability that you made a mistake you forgot to like carry the 1 or something so it could be 1 so it's not 0 this yeah there's like some smaller probability and then if I left you in in the room again still with the piece of paper for 100 years a thousand years infinite time eventually assuming that was correct eventually you say 0 right you come up with some like formal proof that this is this is false whereas you imagine if I leave you for infinite time with the cup and you can't look at it you can't look under the cup it's 1/6 then it's gonna stay 1/6 forever right when you're playing cards you've got these probabilities for which cards you think depending on the game and as you observe new things you're updating your probabilities for these different things as you observe new evidence and then it maybe eventually you'll actually see the thing and then you can go to 100% or zero whereas in this case all that's changing is your thinking but you're doing a similar sort of thing you are still updating your probabilities for things but probability theory kind of doesn't have anything to say about this scenario like probability is how you update when you see new evidence and here you're not seeing any new evidence so in principle whatever the probability of this is it's one or at zero just as a direct logical consequence of things that you already know right so your uncertainty comes from the fact that you are not logically on this yet in order for you to figure things out it takes you time and this turns out to be really important because it's so most of the time what you have is actually kind of a mixture of these types of uncertainty right so let's imagine a third scenario right suppose you were like an AI system and that's your eyes because you have a camera I do the same thing again and now I ask you what's the probability that it's a five you would say one sake because you don't know but on the other hand you have recorded video footage of it going under the thing the video is your memory of what happened right so you're not observing new information you're still observing the same thing you observed the first time but you can look at that and say hmm you know it looked like it with the amount of energy it had and the way that it was rotating and which numbers were where it looked like it probably wasn't going to land on a 5 if I asked you on a millisecond deadline you know what what was it what's the probability that it was a 5 you're going to give the same number that you gave at the beginning here right 1 in 6 but you can look at the data you already have the information the observations that you've already made and you can do logical reasoning about them you can think okay based on the speed it was going the angle it was turned out on which which pieces which faces were where and so on it seems like you know maybe you can run some simulations internally something like that say it seems like actually less likely than one in six right if before I thought it was like 0.16 recurring now I think it's like 0.15 because all of my grand a million simulations and it seems like that the longer you think about it the more precise you can be maybe you keep thinking about it again and you get it down to like you think it's actually 0.13 right something like that but you don't you can't you don't actually know right because there's still things you don't know like you don't know exactly the physical properties of the paper or what exactly the inside of the cup is like or the weighting of the dye like you still have uncertainty left because you haven't seen which way it landed but by doing some thinking you're able to take some of your you're able to like reduce your logical uncertainty the point is that probability theory in order to do probability theory properly to modify your beliefs according to the evidence you observe in a way that satisfies the the laws of probability you have to be logically omniscient you have to be able to immediately see all of the logical consequences of everything you observe and propagate them throughout your beliefs and this is like not really practical in the real world like in the real world observations do not form an orderly queue and come at you one at a time and give you enough time after each one to fully integrate the consequences of each observation because human beings are bounded right we have limited processing ability and limited speed with this kind of logical uncertainty it feels very intuitive that we can do probabilities based on our logical uncertainty that we can we can think about it in this way it makes perfect sense to say that 1 in 10 is the probability until you've done your thinking one in 10 seems like a perfectly reasonable number but like why is one a good answer and like 50% is not a good answer because you might look at it and say well this is a logical statement it's either true or false that's two possibilities you know 50% why why is one in ten more sensible and in fact you had to do a bit of thinking to get there right you had to say oh yeah it's going to be some digit there are 10 digits I don't have any reason to prefer one digit over the other so 10% so you are like always doing reasoning to come up with these probabilities but the thing is that according to standard probability theory this is like kind of a nonsense question because if you imagine like from the perspective of probability theory this kind of statement is equivalent to saying what's the probability that one equals one right or what's the probability that like true equals false they're all just it's just a logical statement it doesn't have a probability it just is true but if you can't think infinitely fast then you need to have answers you need to have estimates it estimates you need to have estimates of your answers before you've thought for infinite time about them this is like an important thing for actually getting by in the world because as I say like the observations don't just line up and wait for you to reason all of their consequences through so when it comes to empirical uncertainty we have probability theory which has these axioms and these are these sort of the rules that you need to follow in order to do probability well and they're sort of things like that probabilities can't be negative they have to be between 0 and 1 the if you have a set of mutually exclusive possibilities and that that constitutes everything that can happen then they all need to add up to 1 and then if you're if you're doing probabilities about like logical sentences then there's all these extra rules that make perfect sense like you have statement a and statement B you also have statement a and B then statement a and B has to be less likely than A or B or the same it could be the same but like if a has twenty percent chance of happening then a and B couldn't possibly have more than twenty percent chance of happening right even if B is guaranteed to happen it can't so that's like a rule if you find in you're doing your probabilities you have a and B that has a higher probability than either A or B and you've made a mistake that kind of thing you have these rules it would be nice if we could find some way of doing of dealing with logical uncertainty some some similar set of rules that is useful and so um and so related to that we have these things like like Dutch book arguments which are about I don't know why they called Dutch I feel like the English language just like is so harsh to the Dutch for no reason unjustly maligned but anyway we're like if you if basically there's there's theorems which which you can prove that if your beliefs don't obey these rules then there will exist some set of bets by which you're guaranteed to lose money some some set of bets that you will happily take by looking at your probabilities that you cut you can't possibly win whatever happens you lose money and that seems like a stupid like you don't want that in your beliefs so that's like an argument that says that your beliefs should obey these rules because if they do you're at least never gonna end up in a situation where you're guaranteed to lose by betting and that's like one of the things we want beliefs for we want some kind of equivalent thing for logical uncertainty and that's what this paper is trying to do it's trying to come up with a rule so yeah you could you could say if if you are doing your probability theory stuff in such a way that there are no Dutch books that can be made against you then that's good and you're doing well and so when we're talking about advanced AI systems we're gonna it's like a good assumption to make that they're at least gonna try to not have any Dutch books against them in the way they do probability so they will probably be a banging probability theory and it would be nice if we had some equivalent thing for logical uncertainty that said if you are satisfying this criteria and in the way that you do this then you're not going to be like doing obviously stupid things and that's what this paper is trying to do when you're talking about probability there's you can kind of think about what properties you would want your system of deciding probabilities to have people have written down various things they would want from this system right there would be a system of like assigning probabilities to statements logical statements in and then being able to like update those over time so one thing you would want is you would want it to converge which just means if you're thinking about a logical statement you might think of something that makes it seem more likely and then think of something else like I know actually it's less it should be less likely and whatever you can imagine some systems for some statements would just think forever and constantly be changing what they think and never make their mind up that's no good right we want a system that if you give it infinite time to think will eventually actually decide one of things so that's one thing you wanted to converge secondly you obviously want it to converge like to good values so if something turns out to be provable within the system that it's using then you would want it to eventually converge to one if something turns out to be disprovable you want it to eventually converge to zero and then the rest of the time you want it to be like well calibrated which means if you take all of the things that it ever thought were 80% likely to be true of those about 80 percent should actually end up being true right it should when it gives you an estimate of the probability that should be right in some sense another one is that it should never and this one's like a bit controversial but it seems reasonable to me that it should never assign probability 1 to something which is not proved or that can't be proved and it should never assign probability 0 to something that can't be disproved they call that non dogmatism at the end of all of this again at infinity after thinking forever probabilities that it gives two things should follow the rules of probability right the ones we talked about before those rules right it should at the end of it you should end up with like a valid probability distribution over all of your stuff their criterion that they've come up with which is the equivalent of there are no Dutch books is based on this algorithm that they've made which satisfies this criterion and therefore has a whole bunch of these nice properties that you would want a logical inductor to have so the way the algorithm works is it's one of those Aeneas tale rhythms I've ever seen it is it's weird and wonderful and if you're if you're a person who is interested in interesting algorithms I would really recommend checking out the paper it's very technical because we don't have time in this video and I may go into it deeper on my channel at some point if I ever come to actually understand it which to be honest I don't fully right now but it's based around the idea of a prediction market which is a super cool thing that's worth explaining in financial markets as they currently exist you can have futures right and a future it's futures contract it's a contract which says I promised to sell you this amount of stuff at this price of a particular thing right so you say take a date a year from now and you say I'm gonna sell you this many gallons of jet fuel for this price at least that's the way it used to be done in practice the actual jet fuel doesn't move if you just go by the price of jet fuel and you pay the equivalent as if you had and then they can go and buy the jet fuel themselves but this is like a really useful financial instrument to have because is let's say you run an airline your main cost is jet fuel prices are very volatile if prices go up you could just go under completely so you say okay I'm going to buy a whole bunch of these jet fuel futures and then if the price goes up I'm gonna have to pay loads more for jet fuel but I'll make a load of money on these contracts and it if you do it right it exactly balances out the cost of that is if the price of jet fuel falls through the floor then you don't get to save money because you're saving money on jet fuel but then you're losing all this money on the contracts so it just sort of like balances out your risk lets you lock in the price that you're gonna pay a year in advance and that's just like super useful for all sorts of businesses really really good in agriculture as well because you don't know what your yield is going to be in so um so the point is that the price that you can buy these contracts for has to be a really good prediction of what the price of jet fuel is going to be at the time when the contract ends and you can kind of treat the price of that as a combination of everybody's best estimate because if you can predict the price of jet fuel a year in advance better than anybody else you can make like almost arbitrary amounts of money doing that and in doing so you bring the price to be a more accurate representation right if you think the price is too low and it's gonna be more then you're gonna buy a bunch and when you buy it that raises the price of it because you know supply and demand so these prices end up representing humanity's best estimate of the price of jet fuel year from now but the thing that's cool is you could build a bit like this is just a piece of paper with stuff written on it right and these days it's not even a piece of paper in principle you can write anything on there right so what if you wrote a contract that said I promise to pay a hundred pounds if such-and-such horse wins the Grand National and zero otherwise and then that's on the market and people can buy and sell it right so if this horse is like guaranteed to win the grand national then this contract is effectively a hundred pound note right it's it's a sure it's money in the bank as if why is that a phrase banks banks aren't that reliable but anyway if the horse is like guaranteed to lose then it's worth zero and if the horse has a fifty percent chance of winning that thing is going to trade for fifty pounds and so by making these contracts and allowing people to trade them you can get these good like unbiased and as accurate as you can hope for predictions of future events and that's what prediction markets are for you can make money on them by being good at predicting stuff and anybody else can just look at the prices things are trading at and just directly convert them into probabilities and that's you know in a spherical chickens in a vacuum kind of way obviously in practice there's various things that can go wrong but like in principle this is a really beautiful way of effectively doing distributed computing where you have everybody is doing their own computation and then you're aggregating them using the price mechanism as communication between your different nodes in you yeah like super cool so that's what logical induction does it like simulates a prediction market where all of the contracts are a logical statement I think in this way it's not a hundred dollars it's it's one dollar so if it's trading at one dollar then it's for sure it's it's certain saying you know this is currently trading at sixty cents and I think it's more more than sixty percent likely to be true so I'll buy some and that seems like a good investment and so all of the traders in the algorithm are programs they're computer programs and it turns out that if you run this and this is computable it's not like tractable in a practical sense because you're running vast numbers of like arbitrary programs but it is in principle computable it ends up having the market it's as a whole ends up having ends up satisfying all of these really nice properties is that because it comes into balance yeah as their as the traders trade with each other the ones who are like good at predicting end up making more money and the ones who are bad at predicting like go bankrupt effectively you might imagine some trader in this system which is a program that just goes around looking for situations in which you have a and B and also a and B in the system and the prices don't work and it's like arbitrage right so and there are people who do this currently on the stock mark there's different things you can do where you can say oh you know anytime the anytime the price is different between two markets you can make money by buying in one and selling in the other and stuff like that you can do arbitrage and it's the same kind of thing you can have so some of these programs will get rich by just saying oh I notice that this statement for a and B has a probability that's actually higher than the probability of B so I'm gonna sell that you know I'm gonna I'm gonna buy some beet array so you know I'm gonna like respond to that in a way that makes me money and because there's all of these traders they will they will eventually cause everything to line up right and so the thing will converge this is the equivalent of saying there's no Dutch book for your probabilities the logical induction criterion the market satisfies it if there's no efficiently computable trader that can exploit the market which means that it can like find a reliable strategy for making loads of money without risk as long as you have that then it satisfies the logical induction criterion which then means you get all of these other properties for free some of the properties that that it has are kind of crazy like it's okay with self-reference it like doesn't care about paradoxes there's all kinds of really cool things that that don't trip it up bringing us back to what we started talking about how does the paper relate to the a Oh safety thing right right so what we're trying to do reasoning and possibly prove theorems about very powerful AI systems and that means that we want to be able to think of them just in terms of being good at thinking and we've got a lot of good theory that pins down like what does it mean to be good at empirical uncertainty we have all of probability theory and like statistics and we can we can say these are the things you need to do to be good at probability and we have like rational choice theory and we can say this is what it means to be good at making decisions and so then when we're reasoning about AI systems we can think well it's probably going to be good at reasoning about answer making decisions but we have to also assume that a very powerful hypothetical future AI system would be good at reasoning under logical uncertainty because it's going to be a physical like bounded system it is going to need time to think about things and it probably is going to need to like make decisions based on things that it hasn't logically thought through every possible consequence off yet so it's probably going to be good at this too and we need some like formal framework with which we can think about what it means to be good at reasoning about logical uncertainty and that's like what this paper is trying to do there erasable yeah okay I do also have I had sharpies yeah I do have sharpies yeah they squeak on the paper people don't like okay fair enough in fact all pens do but these are slightly better right friction

Channel: Computerphile

Views: 254,512

Rating: 4.9107895 out of 5

Keywords: computers, computerphile, computer, science, Rob Miles, AI, AI Safety, Logical Induction, MIRI, Machine Intelligence Research Institute, Artificial Intelligence, Artificial General Intelligence, AGI, Computer Science, Probability, Prediction Market

Id: gDqkCxYYDGk

Length: 27min 48sec (1668 seconds)

Published: Thu Oct 04 2018