Can You Recover Sound From Images?

This video was filmed without sound. Is it possible to use only these images to reconstruct the sound? that is can you hear pictures? in this video I'm gonna try to demonstrate that it's possible to get sound from pictures but it's not gonna be easy so I'm gonna need some help. this episode was sponsored by LastPass which allowed me to fly to the Bay Area to meet with one of my science heroes so now on with the video. How're you doing? I'm sorry my place of my concern is like threw all this crap over here. So this is the experiment that I came up with. -It's like a crumpled up ball of tinfoil? -yeah like if I had a more powerful camera and I had like the right lens then you know we could do something that looks more like you're spying on somebody. But we should be able to demonstrate that you can recover like a rhythm or a sound from you know whatever camera you have. Now you might think it would be easy to record sound in video because after all sound is just vibrations so the air is vibrating back and forth and everything it hits should vibrate back and forth too. So you'd think all we need to do is video that motion and plot displacement versus time and then recover the sound. But it's not that simple because for one thing I mean these sound vibrations are incredibly tiny. They move objects only about one micrometer and even if you're super zoomed in that is way less than a pixel we're talking a hundredth or a thousandth of a pixel. -We're not seeing something that is at one pixel move to an adjacent pixel. You're seeing one pixel get slightly darker and the next pixel get slightly brighter. -What objects work best for recovering sound? -So the things that work best are things that have a lot of damping but are also very light so that they move very readily with changes in the air pressure so what are some good examples? -well like a bag of chips. you know the initial experiments were very like contrived in a way. You know we had these objects on optical benches we were blasting light at them the sounds were like as loud as we could make them. Mary had a little lamb, little lamb, little lamb I figured we'd try to do a rhythm. This is... shave and a haircut let's put that camera on a tripod - oh yeah that's I mean that'd be great All right, let's give it a shot, this is the actual clip I recorded and I want you to notice two things: first, you can't really see much motion; and second, there are plenty of pixels getting dimmer and brighter because of image noise. I mean it's not a pristine, perfect image so how do you tell the difference between pixels getting brighter and dimmer due to tiny movements versus noise? Essentially you look for edges in the image and then you say well if the object moves by some fraction of a fraction of a pixel in one particular direction then pixels on one side of that edge will get a little bit brighter pixels the other side of that edge will get a little bit darker. And so basically what we do is we sum together all the ones that are supposed to get brighter and subtract all the ones that are supposed to get darker and then that gives us sort of one number right and if you track that number over time then that gives you an estimate of the displacement over time. This is time but in samples and then this is position. -hmm so what do you do now? well we're gonna try to do some filtering on that. it's clipping. You don't say I mean it's not much... it's not much but you can recognize two of the beats. This is what we've recovered from a hundred and eighty frames per second, which isn't really a whole lot within the range of like audible frequencies so that's why you know kind of the most we can hope for here is a bit of a rhythm Now of course the main limiting factor is framerate because we can hear sounds from 20 Hertz to 20,000 Hertz but most cameras only shoot 30 frames per second so they miss virtually all of these sound frequencies. Imagine this is the motion created by a 30 Hertz sound if you try to capture this with a camera at 30 frames per second you would end up seeing the object always in the same position because it's at the same point in the wave cycle. So your perception would be that the object is not moving at all So in order to measure a frequency of sound you actually need to sample at least twice that frequency which is why a lot of music is sampled at 44 or 48 kilohertz - that's more than twice as much as the highest sound we can hear. At any rate if you want to get a something more intelligible you're going to need some higher frame rate camera. so we just went to the camera store and picked up a new camera that should be able to shoot a thousand frames per second or thereabouts. Is that gonna be enough? it'll be enough for something. Hehehe I love that confidence This is one modulation away from dubstep right here. Just a little bit more of a wump wump and then we have the next big track. I've set it to a thousand frames per second now we're talking -yeah okay okay so you have the footage there and you're cropping in a bit, tell me about that. well we're running this on my laptop as opposed to the servers that I had back at MIT and that is gonna mean that if we run it on the full video of my laptop will crash -okay so we're gonna crop it and try it on that - I can see a little bit of motion -yeah I mean I think that one question is whether that's like -resonant motion yeah well in this case would be kind of like the equivalent of a rocking chair like if the if the foil has a like a rocking mode then that's actually not gonna give us a lot of sound information -mm-hmm Can you tell whether this is gonna work or not? I am optimistic. I think because I know what I'm listening for I can hear it in there but yeah you've gotta be careful though really? that you're well just gonna be careful that you're not like confirmation bias sure. let's try that that's about 60 Hertz that seems a little much okay we're gonna try one more time. We have basically put the piece of foil on top of the speaker, we're dialing up the volume to... 11. -Well I mean it's a shower speaker so Cool So what do you think of that image there? it's beautiful, it's gorgeous. it has been like a couple years since I've looked at this code so I suspect I might have forgotten something and I'm using it wrong but I want to point something out here, which is that here's what we recovered and here is here is the piano roll of the signal that we sent. This looks like duh duh to me and that's the duhhh yeah yeah that's it that is the... ---dih duh dud din duhhhh din dih - yeah okay so let's see if we can actually get it to hear that -We can see it, we just can't hear it I think I know what it is. What is it? I don't think my laptop can play those frequencies. hold on a second let me get some headphones We need real speakers or something. What do you hear Abe? hold on. I hear shave and a haircut two bits here, listen this. All right. -here, I'll hold it. thank you. yeah. and... pitch shift [recovered sound of Shave and a haircut, two bits] yep there you go, visual microphone. [laughter] this was a basic proof of concept but Abe showed that with more powerful equipment he could recover human speech from outside soundproof glass. Have you ever considered that your computer is a physical system that gives off vibrations each key on the keyboard produces a unique sound due to its unique location. In fact, research has shown that audio recordings of typing reveal 96% of keystrokes accurately. Now this portion of the video was sponsored by LastPass, an app that stores all your usernames and passwords so you never have to type them in. And this prevents people from stealing your passwords by say recording audio of your keystrokes or just looking over your shoulder. What's even better is the convenience of never having to fill in usernames and passwords or getting locked out again With LastPass you don't have to write remember or reset passwords. You get unlimited password storage and free cross-device sync. When you open an app or site on your computer or on an iOS or Android device LastPass fills in your username and password this saves you valuable brain space so put your passwords on autopilot with LastPass. Now something I particularly like is if you upgrade to LastPass premium you get advanced multi-factor authentication and it works like magic. So click the link below to find out more about LastPass and thanks to LastPass for sponsoring today's show

Channel: Veritasium

Views: 2,769,748

Rating: 4.932106 out of 5

Keywords: veritasium, science, visual microphone, sound, pictures, audio, hear pictures, hear, vibrations, abe davis, lastpass, lastpass login, password, password generator, lastpass chrome, lastpass download, lastpass premium, random password generator, generate password, password manager, recover, lastpas, laspass, lastpass premium features, delete account mobogram, lastpass.com, how secure is my password, last password

Id: eUzB0L0mSCI

Length: 11min 23sec (683 seconds)

Published: Fri Mar 01 2019

Reddit Comments

[u/not_responsible]

Okay, I watched it. I know it's science. But being able to record human voices behind soundproof glass with only video? MAGIC.

[u/razorbackgeek]

They have recovered the sounds of a potters wheels from pottery. Pretty neat stuff.

[u/[deleted]]

Ted talk from dr. abe davis himself

https://www.youtube.com/watch?v=npNYP2vzaPo