People expect computers to be
entirely accurate and precise with numbers. You know, you grow up with calculators.
You expect that if you type in: "0.1 + 0.2", you expect it to come out to "0.3". And sooner or later
every programmer, as they're learning, tries something like that.
And they type it into a programming language. Something like "0.1 + 0.2",
and they will hit Enter, and what the computer will give them
is something like "0.300000000...1". And at that point, they will have started
to learn about floating point numbers, and they will be starting to tear their hair out. Floating point numbers are
essentially scientific notation. For the uninitiated, let's say
you're an astrophysicist, and you have the speed of light, which is this enormous number with eight 0s after it. And you want to multiply it by a tiny distance. It doesn't really matter what we're measuring here,
it's just a big number and a small number. Let's say, 0.00000015. Let's say you want to multiply these two together.
(I may have made this a bit deliberately easy.) You could try and do some
really complicated maths and work it all out, but you don't need to.
Because in scientific notation, this is 3 times 10 to the 8, and this one is 1.5 times 10 to the -7. So you don't need to do anything particularly complicated to multiply these two together. All you have to do is multiply these,
which is 4.5, and then add these exponents together here. So, 8 add -7, eight minus seven
means that's 1. So we'll have 4.5 times 10 to the 1,
or 4.5 times 10, or 45. Brilliant. That's much easier than
trying to actually handle large numbers. And it also brings us to
the concept of significant figures. Because the speed of light isn't
exactly 3 times 10 to the 8. It's 2.997[something], and there's
a lot of digits on the end of there which don't really matter
for everyday calculations. We round it up. We say it's 3 times 10 to the 8. Or maybe, for being precise,
we say it's 2.997 times 10 to the 8, but we don't really worry
about the numbers beyond that. Hold that thought.
So, we'll come back to significant digits later. The two main advantages to floating point
are speed and efficiency. Speed, because floating point
has been built over many, many years. It is now lightning fast
for computers to deal with. And efficiency, because
it can deal with really big numbers -- massive numbers, "size of the universe" numbers -- and really small numbers --
"size of an atom" numbers -- without needing enormous amounts of space. If you had a format that needed
to include the size of the universe and the size of an atom, you'd need
so many zeroes stored on both ends or you'd need something particularly
complicated for a computer to work out that it would either be really inefficient,
storing one number as a megabyte, or you would need something that would be
really fiddly for a computer to work out as it tries to place
all the zeroes and the numbers everywhere. Let's talk about base 10.
Let's talk about us - people. (Let's talk about the lights being on
all of a sudden. That's weird.) In base 10, in our normal human numbers, so you know ten digits - base 10, we have hundreds, tens, and units.
You learn that in primary school. And over here, you have tenths,
hundredths, and thousandths. So if you want to say "one tenth" in decimal,
you say "0.1", and that's normal, and that's natural. Base 2, on the other hand, binary..., computers..., they don't do that. Because what they see are fours, twos, and ones. And then over here: halves, quarters, eighths and sixteenths. There's no 0.1 in there. In fact, 0.1 in binary is "0.00011",
and that "0011" is recurring. It's actually 0.00011001100110011...
all the way to infinity. Which is interesting,
because 32-bit computers, the ones we generally use,
(we've moved a bit on now) 32-bit computers only store 23 significant digits. They also store where the decimal point is. They basically do scientific notation in base 2.
That's what floating point is. So they say, "All right, so we've got
this long binary number, times 2 to the something." And that's a problem.
Because what it loses is precision. It doesn't understand recursion. I have an analogy for that in base 10: if you try to write "a third" as a decimal, we go: hundreds, tens, units, tenths, hundredths, thousandths, and then, we try and do this and we end up with
0.333333333... recurring. Now imagine you don't understand
recurring numbers, like a computer, like scientific notation
if you take away that recursion, like floating point arithmetic doesn't. So this is where we bring everything together. If you're in base 10, if you're in decimal,
let's say you have 1/3 + 1/3 + 1/3. As humans we go, "Well that's 0.3 recurring,
plus 0.3 recurring, plus 0.3 recurring, which is 1." But say you're a computer
and you don't understand recursion, because floating point math
is essentially significant figures. So a computer would look at that and go,
"0.333333333... plus 333..., plus 333...",
and it will say, "Well, a third plus a third plus a third.
Total it to 0.99999999..." But after a while, you'll stop.
Because you've run out of digits. What you've just got there
is a floating point rounding error. Let's take it down again to base 2.
Let's do 1/10 + 2/10. Which is 0.00011001100110011...
Oh! We've run out of digits! Because floating point arithmetic
on 32-bit computers only stores 23 digits. And then you try and add 2/10, which is
0.001100110011... I'm not going to try and do that maths,
because it's going to take ages but what happens is exactly the same
as back here. After 23 digits in 32-bit arithmetic
(I think we've got 56 on 64-bit computers) it cuts.
It doesn't understand recurring, which means that floating point looks at
1/10 + 2/10, what you just typed in, "0.1 + 0.2" in decimal,
and it says that 1/10 + 2/10 does NOT quite equal 3/10. Because, to its mind, it doesn't. For nearly all cases, it IS close enough. In most cases, you don't need
more than 23 binary digits of precision. If you do, you go to 64-bit. If you are doing a 3D game and you need to know
where to point something on the screen, it doesn't matter if it's 1/100000 of a pixel out because it will be in the right place. And if you're doing big calculations,
it doesn't matter if a tiny, tiny, tiny fraction -- something smaller than a molecule -- is out, because it will look more or less right: right beyond any precision or tolerance
that the real world could possibly manage. But it means that if you're dealing with currency,
and you say that you are adding 10p (or £0.1) to 20p,
every programmer hits this sooner or later, we just go like that. That's easy. A computer will come back and say,
"That's actually... that many pence." And in these particular human things,
we will notice the error because suddenly to us
it's very, very obvious. That's the problem with floating point.
To fix that, by the way, if you're doing currency, you either use a decimal type
in more advanced programming languages or you just store everything as an integer
in terms of pence, cents, or whatever and then divide by 100 at the end. Floating point is something
that really, really confuses a lot of people. It's tough to get your head around
until you realize that it is just scientific notation. It's the same thing that we learned
in maths class at school, only in base 2. It's halves and quarters and eighths, and once you crack that, once you realize that
it's storing a set length of number and then working out where the decimal point could be
and it's just significant figures, it starts to become a lot easier.
And you start to realize, "Oh, that's why if i type 0.1 + 0.2, what I get isn't 0.3."
Round off error
Truncation error
Knowing the difference and importance of these two sources of error is something I learned in my first programming class (and is no less trivial because I learned it then).
I think C# did it right by adding decimal to its number types for when you need to do currency stuff
Neat. It seems this video is targets towards programmers in a specific stage of their development - they understand some basics about how computers work but don't fully understand how all those arithmetic operations are taking place down in the silicon.
I imagine someone without much math/programming language wouldn't follow much of what's being said here, and most programmers who need to program near the metal already know this.
The guy that made this video, Brady Haran, has quite a few other channels focusing on different sciences. His Maths channel Numberphile and Physics channel Sixty Symbols are pretty good.
Here is another (more detailed) explanation of floating point numbers, from JSConf EU 2013: video (and remember to play with the IEEE754 visualization while watching the presentation)