Phase-Based Video Motion Processing

we presents a novel method to manipulate motions and videos by modifying a local face over time in different spatial scales and orientations our proposed approach allows us to reveal imperceptible phenomena not previously visualized in clarity and detail for example man-made structures like the crane in this video are designed to sway in the wind amplifying the changes in this video reveals the swaying of the cranes mast and the undulation of its hook when a person fixates at a point the human eye also make subtle motions these motions due to involuntary header eye movements such as microseconds are amplified in the sequence they may have medical applications as a frequency of eye movement can have clinically useful data for an input video we perform the following processing we use a complex terrible pyramid to decompose each frame and separate the amplitude and phase of each band we then temporarily filter the phases at each location orientation and scale optionally we apply an amplitude weighted multi scale spatial smoothing to increase the phase SNR we amplify or attenuate the process phases and reconstruct the video the result of a motion magnified sequence we demonstrate our method on a 1d synthetic sequence the new phase based method in green is able to magnify objects further than the linear method of blue at all in red we discussed the bound in the paper the phase based method also has excellent noise characteristics as shown here on Gaussian iid noise the linear method amplifies noise with signal in contrast the phase basement that translates rather than amplifies noise now we compare the phase based results with linear motion magnification by Wu at all to illustrate the effect of using phase instead of pixel values we do not apply spatial smoothing to the phases and these sequences on all of the videos we have tested the phase based method has less noise and fewer artifacts we can further improve our results by spatially smoothing the phases although it is not suggested by you at all we also compare results when applying video denoising both before and after linear motion magnification in this case we apply state-of-the-art denoising algorithms after a linear magnification the de noising algorithms cannot do much with the medium frequency noise in the linear result well the phase based result has significantly less noise video denoising is also computationally expensive taking about 10 times longer than the phase based method applying video denoising prior to amplification can help in some cases but it can also kill the motion signal as vbm 3d does on the guitar sequence the Rebbe we use determines the maximum magnification possible with an octave bandwidth pyramid the magnification of the impulse is limited because it is attenuated by its Gaussian window we can deal with this artifact by using a half octave pyramid this results in a wider spatial support for the filter we can widen and further with a quota octave pyramid the artifacts that we see in the synthetic example also occur in 2d natural sequences and are reduced as we increase the number of filters in this side-by-side comparison there are fewer artifacts in the videos processed using the half and quarter octave pyramid representations videos hide subtle changes at different temporal frequencies in this video the low frequencies contain the swinging of the trunk at mid-range frequencies we see the motions of the branches at high frequencies the motion of the leaves is most visible a user can use an interface we've created to sweep through the temporal frequencies of a video for this video the motions of the shoulder and chest are most visible at the lower frequencies as the frequency is increased the motions of the head become visible and finally at the highest frequencies only the motion of the eye is visible our phase based formulation also lends itself naturally to the attenuation of motions and videos in this sequence we show that we can amplify colors without amplifying motion by first attenuating the phases and then applying wu attalos color amplification on the left atmospheric turbulence manifests itself as low to mid frequency jitters on the right we use our method to remove the turbulence motion magnification can and regions of large motions such as those around the jumping boy by automatically disabling the amplification for regions in which the phase change exceeds our bound we can remove these artifacts as on the right the subtle motions of the platform shaking are still amplified at the miniature scales of motion we are after one might ask would our magnified motion resemble the motions in the scene had they actually been larger to answer this question we conduct controlled experiments in the first experiment we induce small motions of a metal structure using a hammer and record the motion with an accelerometer we also record the structure with the video camera we amplify a sequence with an axillary motion of 0.1 pixels by 50 times the resulting video is similar to a sequence with an axillary motion of five pixels which was obtained from a hardier hammer hit in a second experiment we mount a sheet of rubber on a PVC pipe to create a tense membrane we send away from over are choosing through a loudspeaker to vibrate the air which in turn vibrates the membrane we found that the motions of the membrane magnified 10 times by our method looked similar to a sequence where the amplitude of the waveform is ten times larger to separate and amplify different modes of the membrane when we play a composite waveform of 76 Hertz and hundred ten Hertz the modes have different spatial patterns thank you for your attention

Channel: Neal Wadhwa

Views: 98,527

Keywords: Video Processing, SIGGRAPH

Id: W7ZQ-FG7Nvw

Length: 6min 22sec (382 seconds)

Published: Mon May 13 2013