Optical Flow - Computerphile

today I thought we'd talk about something called optic flow so this is something that you might not have heard about before but it actually turns up in quite a lot of different things so very broadly optic flow is about understanding how things are moving in an image so the pixel level how motion is happening so how things are moving around across the whole image and this is useful for a number of reasons if you've got a wobbly shaky camera you can use it for image stabilization I wasn't insinuating anything yet so you can use optic floaters see what kind of global motion is happening and sort of take that out drones things like that some of them have got optical sensors that will kind of look at sort of how the world is moving around them so this isn't about tracking sort of individual objects or things this is about seeing how the global motion all the surfaces in the scene are moving around if you can kind of break up the scene by motion it's another way of segmenting an image or labeling an image if you've seen some videos about that before so in addition to things like color or texture you can use how things are moving around to help sort of figure out how to divide up a scene so it's almost like 3d I suppose if you get a parallax like yeah exactly rocks here you can see that you're moving differently to the window exactly and you can either you can pull out some other things there as well so things closer to the camera tend to move faster so you get kind of an idea of depth and there's certain techniques such as something called structure for a motion which does exactly that it figures out how things appear in 3d by seeing how the camera moves around let's have a think about sort of what's going on in an image as you move a camera so obviously an image is composed of loads of pixels potentially millions of pixels and when you move your camera around there's some kind of change happening in those pixels and what we want to figure out is how that motion at an individual pixel level is happening okay so that means pretty much for every pixel in the image we want to try and apply some kind of motion vector to it to understand where moving to okay now this is a pretty hard problem there's many many pixels in the image and to understand exactly how each of them are moving we're going to have to kind of make some assumptions and some simplifications and that's where the optic flow techniques cannot come in so this is a really simple image here where we've just got sort of one shape in the middle of it like this perhaps a ball or something like that and if we're calculating optic flow across this image and the ball is moving what we're going to end up with is remember this is sort of every single pixel we're going to get these flow vectors when we're talking about optic flow really what we're considering is sort of the motion at this level within the object or surface that's moving what's going on okay so here I've drawn some pretty fads optic flow vectors showing that maybe this ball is sort of moving down and if the background was moving as well you know perhaps you'd have lots of vectors over the background showing that the background was moving a certain way so different surfaces in the image can move in different directions but it's important to remember that we're just working at the pixel level so we're trying to pick apart what's happening on individual pixels within the image so this is quite complicated to do because the real world is complicated so we have to make a lot of assumptions in order to try and pull out the motion of essentially these individual pixels that represent something called motion flow and that's something sort of moving through the real world so if I move something around that's moving in the real world but how does it move on the image we've just got this 2d plane of pixels and so how do we understand from that how something's moving sort of in 3d so there's a number of assumptions that we have to make one of the assumptions that we have to make is that the things like the lighting doesn't change okay so if you if you think there's this piece of paper if I move my hand across it and we get a shadow then optic flow techniques might pick up on that shadow as being something actually moving in the image rather than just the lighting changing so one thing we have to assume is that lighting isn't isn't changing too much and obviously in the real world that's quite challenging because we get lots of shadows and and stuff happening that we it's going to confuse these kind of approaches so another sort of simplification that we make is that when we're looking at this flow we only do it over very small changes in time so I'm like tracking where we might be following something through a whole video sequence here we're just looking at two frames so we're take two neighboring frames out of a video and see how the pixels have moved between them as well as having things like lighting changing which can break optic flow another problem we can get sort of the other way round is that maybe there's not enough features on a surface to be able to detect the difference anyway I was going to show you an example with a ball but the only really show anymore like find was this tiny one so if you spin it it's spinning of course but you can't see it spinning because you can't see the texture turning round on the ball especially because it's reflecting lighting which isn't moving so one of the problems we have with optic flow is that we're making all these assumptions you know such that we can actually see things moving and that's not always true one of the key constraints we make when we're writing down how we're going to calculate optic flow is this idea of a brightness constraint and what that really means is if we've got our image here like this we've got the intensity at a particular point so this point here our brightness in our image so our grayscale value so the higher is in the new year is to white the lower is near it is to black is that a position X comma Y so that's our pixel at a particular time T and what we're thinking about is essentially where has the thing at that pixel here moved to in the next frame so it's going to have moved only a little bit even much more than that actually but less to it like that so we've got our new position here that we think this information has moved to whoever's in the real world has moved here in the image and so this is going to be essentially the brightness X plus a little bit Y plus a little bit at t plus a little so we've got some changes happening in space X plus a little bit y plus a little bit and also a change happening in time as well so because we've got these little changes you can use actually derivatives you can use some neat tricks where we're just looking at gradients in the image to figure out this equation we can rewrite it as something called the optic flow equation which just basically uses image derivatives to look at changes in grayscale over very tiny patches of course if you want to calculate derivatives in in an image you can use things like kernels like this a barrel kernel which I think Mike's done a video on before so the math spoils down to essentially being able to do pixel wise calculations to do with grayscale so looking in neighboring regions to see changes in space and looking at changes over time as well so differences between frames I'm looking at that now from my videographer head and thinking for K 60 frames per second yeah this is a loss calculator right so there's a few tricks so perhaps at least nicely onto how a few people have solved this because this is the problem and actually now you've got a load of different optimization techniques that will try and give you these vectors where we've moved to here this is kind of known as U and V so these are our motion vectors that we've got everywhere in the image and you're right it can take a little while to calculate that we've thought about how you can talk about optic flow with little changes in space and little changes in time that means derivatives and say things like Sibel which we have had a camera with a fairly high ISO settings you had a lot of grainy noise that's going to produce it full of low-level edge over everything

Channel: Computerphile

Views: 68,904

Rating: 4.9126902 out of 5

Keywords: computers, computerphile, computer, science, Computer Science, University of Nottingham, Dr Andy French, Optic Flow, Computer Vision, Sobel, Optical Flow

Id: 5AUypv5BNbI

Length: 8min 24sec (504 seconds)

Published: Wed Oct 16 2019