The Search for Randomness with Persi Diaconis

it's it's remarkable to see you all here this place is absolutely full which is very cool and I see community members here I see faculty members the faculty are easy to find because they're looking really serious that the freshmen are easy to find because if you just look at the freshmen they just smile when you look at them look at this right see you're giggling I just looked at you and you're giggle don't stare at the freshmen please but this is precisely why we're all here and why we why we did this this is our sixth year of this common book event I want you to appreciate just how challenging and interesting is to to put together a committee around a a single task which is to find a book that will bring together community a single book that people can hold in their in their hands that will inspire ideas inspire us to think perhaps bring us together around the talk maybe even forum community when freshmen were coming in I saw them sitting in the lawns I saw them sitting out by Husky Stadium with Richard Feynman's book in their hand and that's precisely the point to come to events like like this forum community around I ideas this year the committee chose fineman's books the meaning of it all thoughts of a citizen scientist by this Nobel laureate winning physicist and what a fascinating choice in light of the other selection that we've chose from the beginning the first year was was mountains beyond mountains book about Paul Farmer written by Tracy Kidder and we invited both Tracy Kidder and Paul Farmer here had a wonderful wonderful talk fascinate the next year we invited Elizabeth Kolbert around the book called field notes from a catastrophe that climate change last year we the committee chose the selection of poems we were talking about poetry last year on this stage and this year we're talking about a little bit of everything and to include science art magic it's a chance to bring faculty together and a chance to tie to a theme this is our sesquicentennial at the university our 150th year and when the committee came together and chose this book it was a chance to tie back to a series of lectures in 1963 where Professor Fineman came and spoke and so this book is a compilation of of lectures of Fineman himself it is a fascinating read and it's a little bit of Fineman it is curious it is smart it's a little egocentric just like all of us I look forward to this talk tonight I you're going to learn how to lie you can learn how to cheat and you can learn how to steal and if we do this right maybe we'll learn a little magic as well so welcome Thank You Shelby thank said we're so excited to have dr. diaconis with us tonight one of the things about a place like the u-dub is that you have an opportunity to hear from a range of leading thinkers on different topics and then you can even ask them questions and since this year's common books theme is so much centered around questioning after the talk there's going to be some time for all of you to ask questions of dr. diaconis so think of questions throughout the talk today dr. persi diaconis is the Mary V Sunseri professor of statistics and mathematics at Stanford University but when he was 14 he left home to learn magic and from and travel with the sleight-of-hand legend died Vernon I imagine that was pretty interesting heard some good stories in the green room about those times so but at that point he wanted to read a book on probability theory but he didn't understand the math in the book he vowed to return to school and learn math so that he could understand this book he might be the only person in the world to run away from the circus to join a school it's pretty interesting he went on to earn his undergraduate degree from the City College of New York and his PhD in mathematical statistics from Harvard University he used mathematical principles to debunk physics and hamper a Caribbean casino that was allegedly using shaved dice to benefit the house in his research he often brings an interdisciplinary approach to issues of probability factoring in the physics of a coin flip for example to determine the chance of it landing in the same way it began he incorporates magic into mathematics to explain simple questions the answers he finds can sometimes unravel what we thought we knew about probability and chance he received the MacArthur Genius award and has written several papers on randomness and probability and although he is here tonight to talk about the search for randomness there is nothing random about how glad we are that he's here to speak please join me in welcoming dr. persi diaconis hi speaking in the shadow or the glory of Fineman is just wonderful I was I didn't know this little book I read firemen both high and low surely you're joking but also technical papers and and carry it here is the Bible you you can find lots of different things in Fineman that and I'ma do probability and statistics and so I was reading and I found lots of references to things that I'm interested in maybe as you read along these all sing out to you one of the things he said is all of the things we say in science all of the conclusions are uncertain later it is of paramount importance in order to make progress that we recognize this ignorance and this doubt because we have this doubt we can then propose looking for new ideas he then gets a little bit more quantitative and there's one thing to say I don't know anything and there's another thing to say this there are two sources of difficulty that the young man we are imagining would have I think when he when he studies science the first is that he learns to doubt that it is necessary to doubt that it is valuable to doubt so he begins to question everything the question might have been before is there a God or isn't there regard he's not taking any prisoners he's right after it and that question changes to how sure I am i that there's a God he now has a new and subtle problem that is different than it was before he has to determine how sure he is we're on the scale between absolute certainty and absolute certainty on the other side he can put his belief because he knows that he has to have his knowledge in an unsure condition and he cannot be absolutely certain anymore he has to make up his mind is it 50/50 or is at 97% this sounds like a very small difference but it is an extremely important and subtle difference so how do you think about uncertainty and that's what I'm going to talk to you about tonight this talk is called the search for randomness I've got this thing it's good and and and what I'm going to do is to talk as openly as you can in public about the foundations of my subject the the roots of it and for me one of the primordial examples of a random phenomena is that that is if you flip a coin in your own hand your coin and you you know do that heads or tails that seems like 50/50 to most people and I mean it's a it's a primitive image of randomness and we would often understand what are the chances of something else in terms of well it's like flipping flipping a coin now I'm going to talk about flipping a coin for a while so what about that here's a question that Fineman could have asked when I flip a coin it goes up at a certain speed and it turns over a certain number of revolutions per second if I knew how fast it was going up and how many times a second it was turning Newton tells me whether it comes up heads or tails so it's coin tossing random or is it physics that's a question I mean that sort of well I want to claim that it's physics and to make that point I had my physics department build me a coin tossing machine and I used to bring it along to talks but it's a kind of awkward gadget takes about two hours to set up and when you go through airport security now what's that it's a coin tossing okay so I'm sorry you'll have to look at it this way but there it is it's about this big and coin goes in there that's what looks like and then I turn that ratchet and and the coin goes ping up that way sparing no expense I don't know it must have taken 70 shots to get that edge on see that there that's what it looks the coin just spins up just the way it looked on my when I was doing it there and then the coin lands in the cup tada only it always comes up the same way and so you do it heads you do it heads you know 100 out of 100 heads it's very very disturbing viscerally you know this seems random and it's heads all the time now of course if you think about it the coin is being hit with the same force at the same position it's got to do the same thing so coin tossing is physics it's not random so for a second I'm going to look at the physics a little bit Susan here's here's some physics when when the coin goes up this away it's traveling at a certain speed and it's got a certain how many times a second it's turning and each flip therefore corresponds to a speed and how many times a second that's turning and if I can make a graph on a picture like that where this axis is velocity how fast it's going and Omega is how many times a second that's turning and so for example that red point there which is kind of high on velocity but low on Omega the coin is going up like a pizza okay and then it you can imagine if a coin went up like a pizza it doesn't turn over at all right it just goes up like a pizza and so there's a region there where the coin doesn't turn over at all and and up there there the coin has tremendous rotation number but it just didn't get very much oh so it also doesn't turn over there's a region where the coin doesn't turn over at all and that's bounded by a hyperbola and you can you can write down what it is and then there's another region where it turns over exactly once and exactly twice etcetera so now here's what that picture looks like a grown-up this is the phase space of a coin being tossed now the first thing you might notice about this picture is that up there the the strike so there it turns over no times one time you know two times three times so even an odd so it comes up heads or tails up there the stripes are getting closer and closer together so small changes in the initial conditions make for the difference between heads and tails and everybody kind of knows that and and that that helps to explain why you have to be pretty good with your thumbs to control it okay so there's a question when we flip a real coin like this coin where are we on that picture that's a that's a question so there's two parts to that question first is how fast is it going and how many times a second is it turning out how fast is it going that's not so bad I got a friend with a stopwatch you know 1 2 3 flip sort of clicking and you know flipping and if I know how long it takes to go up and down I can translate that into how how fast is it going and so we had a number for that the other question I like to talk about because people have ideas and I'm sure I'll learn something when I do that when you do that how many times is it turning is it turning seven times as a turning 80 times someplace maybe in between you know it's not so easy to think about so I thought I know what I'll do I'll get a high speed slow-motion camera and I'll you know I'll just do it so I went to the unit went all over the university and I wanted a high speed slow-motion camera that would capture a coin being put you have to shoot at about a thousand frames a second to do that and at the time there was one camera on Stanford's campus which would shoot at a thousand frames a second and it was owned by our football team they had a one-week period during the summer before the team came back where they would allow me to use the camera but I had to hire their operator and be about $2,000 a day I wanted to know the number of it I didn't want to know it that badly then I had a first idea which was paint half of the coin black half of the coin white and get a friend with a tunable stroboscope like blink blink blink blink and then have him adjust and I flipped the coin and he adjusts so seven or eight hours later it never freezes it's kind of complicated what coins dot do but we had some idea and then I had my first good technical idea okay here it is brought to you just explain this this is an American half-dollar with I hope eventually some how stuck together could it have could it have become there we are it's got dental floss on in case you were wondering right this is the new kind of dental floss that's flat so I flatten it out like that there it is flat flat I don't want to do that there it is there is Kennedy face-up and it's flat I do that and then I unravel it until it's flat again and I can tell how many times it sorry well okay by hook and by crook we had a pretty good idea of how coins go and I want to tell you the answers for a second from experimentation a one-foot toss or so that well that's a little more but it takes about half a second and that means the coins going up about five and a half miles per hour using the tunable strobe and with the ribbon and that stuff a coin when you flip it is is turning over it something between say 35 and 40 revolutions per second now it's only going up for half a second so that means it's turning over well you know something between you know 18 and 20 that's not so many so much variation 18 19 20 it's not it's pretty set in there right it's not so so wild and so I kind of know where I am now going back to this picture I have some data where does that put me on the picture well in the units of this picture this is the velocity the how fast it's going is a fifth so there's five there's one a fifth it's pretty close to zero but it's 40 units up so this picture says nothing about real coin-tossing but the math behind the picture tells you how serrated the lines are up there and there's a real sense in which I mean approvable sense in a practical sense in which coins flipped the way we normally flip them or slightly biased to come up the way they started slightly by fake started heads they're slightly biased to come up heads again and the bias is about 0.5 1 so about one in a hundred that's a small bias it would take you about a quarter of a million flips to detect that in real experiments but every year the Superbowl time ESPN calls and say could we get you to do a minute about whether the Super Bowl toss is fair or not and I have ducked in the last few years but who knows I want to abstract a little bit where does randomness come from and this image that I'm about to explain is about as clear as I can say for me for where random has come from so imagine that that wall was painted black and white and I was standing there not standing here okay and then I actually had a dart so that's black that's white and I'm not great with darts but I could mostly get it into the black sure and or maybe if I could think of stance a little close to get it mostly into the white so there's nothing terribly random and now imagine that the paint on the wall was rearranged so that instead of being that panel is black in that panel is white there one foot by one foot I'm not bad but I'm not good suppose they go down there one inch by one inch black white black white black white black white it's random right wear it whether it's black or white when it hits the wall right so you could see the randomness coming in I want to that that's such a common cause that is the tiniest little uncertainty about the initial conditions get magnified into making something random I want to make math out of that for a second this is a picture that's supposed to mirror my the angle that I tossed that dart in so the height of that curve is the propensity for the dart to come where the curve is high and where the curve is low my it's not so likely that I'll toss the dart there there's some curve my aim where the dart one's under the curve it's um it's labeled say red or black and a wonderful mathematician on reap encore a proved as a theorem now it's sort of intuitively obvious that as that ruling gets finer and finer the chance that it's one color or another tends to 1/2 this is essentially the only math in the talk if you're allergic put your fingers in your ears it's it's it's it's not very much proof I've got my density I have to figure out what is the area above the red pieces how does that differ from the area above the white pieces that's what I wrote down there these symbols say that is the difference between the propensity for the dart landing at x2 the propensity of the dart landing at X plus h if the propensity function is a nice continuous function that's bounded by what's called the modulus of continuity sorry for using bad words and it's a property of continuous functions that that tends to 0 as H goes to zero if you're grown up sitting here it's also true without assuming continuous because translation is continuous in l1 now if you want to ever know what that means go take some math course it's ok not for that we can sharpen that to make a quantitative bound the chance of read differs from 1/2 by how wide how far apart things are H times this measure of whig leanness the derivative measures how sharply peak the function is and you have to put something like that and because I might be a professional dart player and get pretty good at throwing the dart where I want but as the ruling gets finer and finer the chance of red or black goes to 1/2 no matter how accurate you are that's what this says this idea of quantifying the approach to randomness is called the method of arbitrary functions it was invented by pong crane it was brilliantly developed by Eberhard Hough who treated all kinds of problems that way now rather than tell you anymore math I want to tell you some stories I can put some all down so we can all see them together the same kind of analysis can be applied to any image of randomness so roulette here's a story from the trenches when I started teaching at Stanford we had the idea you know roulette is a physical game probably most of you know roulette there's a outer wheel a ball gets spun on the outer wheel it goes around like that there's an inner wheel with 38 numbers on it numbers 1 to 36 plus 0 plus double zero it gets spun the other way the ball drops down and it lands in one of the pockets and if you bet on the right number you you win relates a physical game and they let you bet pretty late at Roulette and so we had the idea we could clock the wheel and try to figure out where the ball is going to no romp and so we built a gadget kind of a little like a cigar package and it worked like this the coupie a throws the bowl when the ball passes a fixed point on the wheel we tap a switch on our arm or something you don't sit there with a little computer you know and and then the gadget knows the ball just went by it comes around again you tap and the gadget knows so many other gadget knows how fast the ball is going you do the same for the inner wheel tap tap the bad the gadget knows how fast the inner wheel is going and then the gadget figures out where the ball is going to land it's only right to within half a wheel if you're getting 35 to 1 on an 18 to 1 shot it's like having a vacuum cleaner in the casino okay I mean one thing we learned is that casinos don't like it when you start hitting numbers you put $25 chips on a number and you hit a couple of times they really don't like it and when we were building the gadget we I rented a roulette wheel we very carefully leveled it we done all the physics so we you know how do you program the gadget and we tried it out and all the equations didn't work at all and one of the things I did as I did with the coin is we had neglected friction this three kinds of friction this rolling friction this sliding friction and there's air friction and the friction that really mattered was air friction so when they turned on the air conditioners all the equations changed right it wasn't a waste of time because we knew the shape of the equations and we just had to tune a few parameters and so what we did is go to the casinos watch for half an hour tune the exact parameters it's a little bit of the use of statistics and then we we got we got pretty pretty good if you want to read about this we made money with it and so we didn't write a book but there was a group at Santa Cruz that never got their act together so they did write a book the book is pretty good it's called the eudaimonic pie you Dumanis was one of Aristotle's good demons and I knew that they had actually done it actually no the guys who wrote the book when they said you know there's three kinds of friction sliding rolling and air friction really matters as they tried it ok that's good there is one other thing I want to do which is this it's one of the reasons I like this transparency projector there's another way of flipping the coin and it's this okay you know spinning it on this edge no that doesn't show so well but that's good Sosa coin spun on its edge so if somebody asks you you know what are the odds that a coin spun on its edge comes up heads or tails I don't know there's a right answer to that question you don't know because you don't know it turns out anything like 50/50 s way way off coin spun on their edge have very very distinctive biases when I was a kid I learned that 64 D penny spun on their edge come up tails 80% of the time when spun on a pool table you know you could make money from somebody it has to do in a very delicate way with where the center of gravity is it works better with coins with unmilled edges pennies and nickels are more bias than than others here's some data that I I gathered this is I had the class of 103 kids and I tortured them first exercise was find a penny and toss it a hundred times so that's a lot of times to toss a penny and all of the kids toss the penny and tell me how many heads you got and this is a histogram and what you can see it's about 50 but then plus or minus you know something and that's just what 5050 coin tosses should be this kind of a coin toss then I had them take the same penny and spin it on their edge and then I collected the penny okay and this is the spinning data so each kid spun the coins 100 times and a given coin has its own bias but for example here out of 100 kids you know 5 or so had had a had a coin that came up I guess that's 90 tails tails this comes up more than heads in in spun coins some reason but but you know 90% of the time it came up tails well that's pretty biased right now I'm gonna combine these things here's a question if somebody asked you this before you go into that talk what do you think the odds are about this one guy says I'm going to you know you're going to do this I'm not cheating flip a coin in the air that's one thing the second thing it could be is that flip a coin in the air land on the floor which is fairer now before this talk you would have said is that a question does it have an answer could it have an answer most people when asked and I've asked think that coins tossed on the floor or Ferrer I'd like to suggest that's because when a coin is caught in the hand there's a hand there and you think maybe something can happen you know so let's rule that out you're tossing the coin at home you're not looking and catch it okay which is fairer well having watched a lot of coins when a coin hits the ground before it dies it spins around a little on its edge and some of that edge stuff comes up so there's a real sense in which coins caught in the hand are fairer than coins tossed on the ground what's that f so to be honest for a moment I lied to you I said I was going to explain randomness and if you listen carefully when I got down to it F which was you know was the propensity of it was sort of the probability of the dart landing someplace so it's there really is a sense in which you can't get probability out without putting probability in but what is that F what is it actually well punker a who invented the theory he just didn't care about philosophies I didn't worry about it hop said something which a lot of people like and it went like this if I if I toss the dart a lot it's going to get darker in some parts of the line than others if I flip the coin a lot each time on my phase diagram I put a dot the velocity and the spin the dots will be bigger some places and less less less dark other places hop thought you have this data and then you make a histogram out of it and then you F is a continuous smooth approximation to the long term frequency of the initial conditions and that's okay it's okay but the problem is the argument I gave the careful analysis I gave the analysis seems right for one flip you know I don't need to talk about repeating endlessly also if you try to think about how I can use anything about probability to talk about you know more interesting things like is it going to be raining when we leave the building or does she love me or does she love me not I mean any real application of uncertainty and there every part of life the notion of I have to embed my life in the present circumstances in some imaginary long-term frequency is gaga for me and on the other hand we do have uncertainty in everything we we have and we do and and is there a way of dealing with uncertainty where things can't be repeated and there is a way and I'm not going to tell you very much about it but at least I'll say the magic words it's called subjective probability and Bruna de Finetti Frank Ramsey Jimmy Savage and I think that probability represents his best thought of as representing a person's degree of knowledge or uncertainty for observables so the way I think about it coins don't have little numbers probability written inside them they have force they have mass and coefficients of restitution and centers of gravity and things that you can measure but they don't have little probability people have probability so if you know I'm a very you could guess that I'm a very experienced coin toss er if you had to you know assign odds to yourself for me by your knowledge you might very well think that you know this guy is going to get ten heads in a row whereas I'm probably not going to get ten heads in row that is what you know about the circumstances and situations have to be you know go into the calculations well it seems like a tall order but it's not and the using notions of coherence and exchangeability there's a very beautiful theory that captures what's useful I find it captures what's useful for me in making probability calculations and it protects us from silliness as you'll hear in a minute it's possible to invoke probability and randomness and do the silliest things and and I find that thinking this is supposed to be my opinion you mean mm-hmm that's not my opinion or anybody's opinion I find that this subjective world helps me so here's my answer to a Bayesian answer to what is F F is my best guess for the next for example pair so suppose Edie pops up right now and he's got a water pistol and he says alright diaconis you're about to flip that coin and he said and where do you think it's going to come in your fancy face picture I don't know figure on the trigger it's leaking a little bit and and we'll wait to say I actually know quite a bit about where it's going to land I've done experiments I've done lots of experiments and actually I know how high it is and how spread it is and so f is my best guess for what the initial conditions are and then the math that I develop says that F washes out my specific choice of F doesn't matter as long as the mesh is getting finer and finer any Bayesian will come to the conclusion that it's that it's fair if the flip is vigorous if I'm out out there in left left field right field actually so f washes out giving a subjectivist version of objective chance that's just one example change of subject this is not a question that you hear asked in public so often you know you're a bunch of grown ups and there I am talk to you about flipping a coin and other things like that does any of it matter to people does it matter in the world I mean because it can sit well there are a lot of answers to that question and let me begin with maybe a quite practical one there's the world now so what do I mean does it matter I mean does thinking carefully about randomness matter that is could it could it change your life let me make money or anything or protect you from bad things so a lot of people used to play it's now the FBI stepped in it that couple of months ago it's not so easy to do used to play online poker and I promise you they don't have people sitting there shuffling cards in the Bahamas what they have is some silly computer chip that generates random orders of a deck of cards and and it's nothing like actually shuffling cards and some computer science guys and poker players and people like me ouch figured out that you could crack the random number generator so to give you an idea of how it goes the number of arrangements of a deck of cards is a big number it's more than the number of particles in the universe this is 52 factorial 52 times 51 times down to two down to one it's a big numbers ten to the 67th about 10 to the 68th if okay random number generators the way the computer works are limited by the size of the computer there they are limited the way they they work and they're a much smaller number ten to the ninth about but the way the random number generators are turned on they use a starting seed and if you know the random number generator and you know the starting seed you know what all of the numbers are and you know the order of the deck of cards the way the random number generators are started is to use the number of milliseconds since the start of the day well these people knew when the computers were open so they had a pretty good guess at how many milliseconds they were you know within okay some factor and so that was that many possibilities when the dust settled the actual number of arrangements that could happen in a day of guards were about two thousand and so what happened was that the deck is dealt out this is Hold'em which is a variant of poker and when I see my two cards and the flop goes down three other cups I know everybody's cards well you can play pretty well if you know what everybody else is handed so this is a group of guys that by thinking about hey B's things aren't really random and I can think about it when made money they made money for a year and a half millions of dollars and the way it came out was you can't just I always win you have to have somebody you know taking down the money and so it might be I talked to Ed and said I've got an opportunity nothing you know I'm just going to tell you what to do but you sign on and then you know you play I'll tell you how to play sorry and and and what happened to some guy was being told exactly what the cards were and exactly how to bet decided he was an ace and he wanted half the money and they said give me a break he said well if you don't give me half the money I'm gonna you know go public with this and he went public with it and so that's why we know about it anyway the point is real you know casino operations can be thought about now if that's a little flippant to you here's one that's you know hundreds of millions of dollars bigger the stock market in practice it said on the New York Times the four stock formulas weren't perfect I read a little bit of it move ahead to August 2007 beyond when markets wound on doubts about subprime mortgages stocks that the model predicted were bound to go up when sharply down and vice-versa events that were supposed to happen only once in 10,000 years happened three days in a row so there are these phonies they really are phony who claim they can make equations and it's like physics out of the stock market and somehow they convince people that their equations were were valid and and and they were crazy and and and it matters I mean at least in this way it mattered if you don't like that it's about money after all but that is people try to use probability in ways in which aren't they're not sensible here are some other examples of probability calculations that give me the willies from mouse to man well this is the world of big models you ever wonder how they set carbon monoxide levels or other you know how much sugar should be in something rather well one way is they have these very very spread especially bred laboratory mice if you breathe on them they fall over and they expose them to thousands of times more carbon dioxide than anybody would ever use or see and and they see how they do and then they try to extrapolate from those experiments to rest of the world to us other things in order to extrapolate you need a formula and those formulas are to a large extent completely made up out of whole cloth now I'm a professional I looked hard they're just somebody makes them up we know we can do the experiments you can extrapolate from mice to rats you know it's very expensive experiment to do what they've been done the functions are not standard exponential functions it's very complicated so I'm just telling you that the reference there paper by David Friedman and Hans ISIL in statistical science is somebody saying it very clearly and then the lab scientists coming back and saying well we you know so and so and they Duke it out and I you know suggest you look some time but it's an example of I think silly use of probability modeling in the context of big models there's another one that I put up and they're there many many examples of this sort and that census adjustment you know we just had a census a couple of years ago and and one of the things that happens to try to count everybody in the US and one of the things that happens when they take a census is they miss some people and it's not trivial it's five to seven percent it's you know tens of millions of people or and mostly they miss people in big cities you know people don't want to be counted illegal aliens don't want to be counted sure okay the numbers are big enough so that it can make a difference on whether you get you know four representatives or three representatives on on how much money your state gets I mean they're huge different huge differences for resources that have to do with the with the census and how do they adjust well it's a long story and I'm not going to try to tell it in any detail but to some extent they make up a bunch of equations and they try it you know and they say well it works pretty well another guy say it doesn't work I'm of the school of thinks it's really all made up and if they don't know what they're talking about and and I'm not alone I mean there are lots of us that feel that way if you want it to read again one of these lets Duke it out I'll tell you what I'm doing and you say this census adjustment is is that's a paper by David Friedman and many other commentators statistics has a funny tradition it has discussion journals in which somebody writes an article and then people who don't like it you know get to respond them and you get to respond back and it's pretty you know they're pretty tough they're interesting conversations but the point is this statistics has been taken over by huge models models that are that are so big that no one person knows what the ingredients are models that are so big that if you try to run them again on the same data you always get different answers because of tiny little rounding differences and things like that models where the input comes from other big models and the output is coefficients in some regression equation which can't be compared with reality and they've taken over okay so it makes me very very nervous that's my field statistics and it's just really every place and and now if you see the difficulty that I can have with flipping a coin you could just imagine the difficulty I have with the regression equation that has thousands of parameters in it and linearity and constant variances and all of that stuff now the doctors don't testify against other doctors and statisticians aren't supposed to say bad things about statistics but this stuff stinks sorry colleagues and and you all heard me and you have to make up your own mind but it's something to be aware of now that's a lot of negative stuff and I don't want to end on a lot of negative stuff so I have to remind you and me that the probability can be used for the greater good so there's one more example and here it comes and and this is a story but it's true story it's the story of Gauss one of the greatest mathematicians ever ever ever and Ceres Ceres is the largest asteroid and thus the time is 1801 and Gauss who really is a brilliant man has just finished his masterpiece this christianís arithmetic ax is 21 years old and it comes out and then he realizes he's got to get a job and what to do about that at that time the world was sure because Hegel had argued that there were only five planets everybody knew and and then some guy spotted Ceres and he followed it for about 40 days and he how high above it of the horizon is it how far over is it so he had some measurements and then he lost it in the Sun and this wasn't something of some astronomer in a corner this was on the front pages you know is there another planet that circles the world in it that's pretty interesting people really were excited by it except they couldn't find it and the best mathematicians and astronomers and physicists tried to find series they had the preliminary data and they they couldn't find they couldn't use the data a guy published the data in a newspaper he was so frustrated by it and now saw this date and he said wait a second maybe I can figure out where series is by thinking about it okay now let me explain a little bit Ceres is orbiting going around for anything that's orbiting that way if you knew where it was at some time and you knew how fast it was going so that's three coordinates of position three coordinates of velocity if you know where it was and how fast it was going you would know where it's got where where it is for all time so the unknowns are only six numbers you know at some time where it is and where is it going the observational data that Gauss had were you know he was the astronomers were here so how far above the horizon is it on the first day on the second day and the third day how high you know how high up is it how far over it was it there were certain functions of those six parameters and of course you don't observe it perfectly even with the telescope you observe it with error and that's what that sentence says that is the observation with some function of these unknown parameters plus a little error why I might be the height or the angle on a given day and epsilon I is noise okay Gauss had the idea justified it that so I've got to figure out these six parameters the elements of the orbit Gauss had the idea I know the form of the function and I don't know these Thetas but I could try to choose the Thetas so that the form of the function at that theta differs from my observation by a small amount as possible so he took the observation he took what it should be if theta was the true state of nature and he took the difference squared it added up that up over all of his observations that's called least squares and he had to figure that out and is he right I'm quoting now October 1801 the first clear night when the planet was sought by Zack as directed by the numbers I deduced restored the fugitive to observation ok charming what did he actually do this doesn't go on forever but I'm putting one slide up of what gauss actually did the first thing he had to do was he had to figure out what the equations were they weren't so well understood it was sort of annoying trigonometry in 3d you know three spherical coordinates it was it was pretty hard work that's one thing he did he had the idea of taking that unknown rule fi of theta and linearizing it he set the problem up as we do in our day-to-day statistics work this observations these Y's that I observe and that's X some matrix plus a vector beta beta the differences of theta from an initial guess and then so he formulated the problem that way plus an error because for the observations because he knew the form of the function that matrix X had a known form he invented Gaussian elimination as it's now called to solve for beta he used this beta to give a second guess he had a first guessed theta naught he linearized around it he then corrected it he iterated and he thus invented nonlinear least squares he proved that his estimate beta hat was the best linear unbiased estimator he proved that various other things he was Gauss I mean he was just amazing I was complaining about the use of models in this example there were many many assumptions made the planets don't go in circles you know that is this rather complicated motion we Gauss assumed that the errors were from the bell-shaped curve that's completely made up he assumed that the equations were linear that Y was a linear function of the Thetas at least initially he assumed that the measurement error didn't penned on the day which it might if the sun shining or not or whether it's this kind of measurement or that kind of measurement he assumed that the measurement error was constant and he assumed no bias those a roll wrong wrong wrong nonetheless Gauss found series right and where others failed and it's still where he says it is right you just said it's going to go in this orbit and you can look now and what the other thing that happened is it's now officially a planet you know you know when Pluto got demoted Ceres got Ceres is now one of the planets so what that's just a few years ago Gauss didn't know about that an unfortunate thing happened is he got job as an astronomer and so he stopped doing mathematics at least full times this is an example of good probability model modeling skillful statistical analysis to do a job that the smartest people couldn't do without it it's one clear example it's the first example of this kind of analysis there are thousands of other applications as Fineman says again and again advances of this kind tools of this kind can be used well or badly and Fineman says it very well I won't try to use it I told you they can be used badly in you know mouse to man and census adjustment and many many other misuses of statistics they can also be used well and here's my last slide conclusions I talked about flipping a coin the dart dice roulette I didn't tell you about shuffling cards but I could have in case after case randomness or good analyses are possible but often this species us are lazy and we don't flip the coin vigorously enough we don't shuffle the cards enough we don't brush our teeth enough okay we're lazy and things can be pretty bad and it's not only in theory they can be pretty bad you can go make money in the casino based on the fact that they don't roll the roulette wheels right people make money in the market and lose fortunes by using bad models okay that's one conclusion this is a quote from my friend Amos Tversky it's easy to lie with statistics it's good to remember it's a whole lot easier to lie without them why I gave this talk especially aimed at the freshmen here is the following sentence randomness isn't a black box it isn't a black box we can and must look inside to make responsible use of probability theory whatever you do you're going to encounter somebody using some analyses and you're allowed to think is that true is it reasonable you don't have to figure it out yourself remember there are people like us who can help you we're happy to help you and but but but you can think about things and you should should be skeptical because there just are endless charlatans out there they're what they really are okay this talk was called the search for randomness I think you could tell by now it's been my search for randomness thank you for coming along with me you

Channel: UW Video

Views: 22,533

Rating: 4.9324894 out of 5

Keywords: University of Washington, UW Common Book, Randomness, Diaconis, Feynman, The Meaning of It All, Stanford, science, Nobel laureate, citizen scientist

Id: xit5LDwJVck

Length: 55min 2sec (3302 seconds)

Published: Fri Nov 15 2013