Dear Fellow Scholars, this is Two Minute Papers
with Dr. Károly Zsolnai-Fehér. Today, a variety of techniques exist that
can take an image that contains humans, and perform pose estimation on it. This gives us these interesting skeletons
that show us the current posture of the subjects shown in these images. Having this skeleton opens up the possibility
for many cool applications, for instance, it’s great for fall detection and generally
many kinds of activity recognition, analyzing athletic performance and much, much more. But that would require that we can do it for
not only still images, but animations. Can we? Yes, we already can, this is a piece of footage
from a previous episode that does exactly that. But what if we wish for more? Let’s think bigger, for instance, can we
reconstruct not only the pose of the model, but the entire 3D geometry of the model itself? You know, including the body shape, face,
clothes, and more. That sounds like science fiction, right? Or with today’s powerful learning algorithms,
maybe it is finally a possibility, who really knows? Let’s have a look together and evaluate
it with three, increasingly more difficult experiments. Let’s start with experiment number one,
still images. Nice! I think if I knew these people, I might have
a shot at recognizing them solely from the 3D reconstruction. And not only that, but I also see some detail
in the clothes, a suit can be recognized, and jeans have wrinkles. This new method uses a different geometry
representation that enables higher-resolution outputs, and it immediately shows. Checkmark. It is clearly working quite well on still
images. And now, hold on to your papers for experiment
number two, because it can not only deal with still images of the front side only, but it
can also reconstruct the backside of the person. Look! My goodness, but hold on for a second…that
part of the data is completely unobserved. We haven’t seen the backside…so, how is
that even possible? Well, we have to shift our thinking a little. An intelligent person would be able to infer
some of these details, for instance, we know that this is a suit, or that these are boots,
and we know roughly what the backside of these objects should look like. This new method leans on an earlier technique
by the name image to image translation to estimate this data. And it truly works like magic! If you take a closer look, you see that we
have less detail in the backside than in the front, but the fact that we can do this is
truly a miracle. But we can go even further. I know it is not reasonable to ask, but what
about video reconstruction? Let’s have a look. Don’t expect miracles, at least not yet,
there is obviously still quite a bit of flickering left, but the preliminary results are quite
encouraging, and I am fairly certain that two more papers down the line, and these video
results will be nearly as good as the ones were for the still images. The key idea here is that the new method performs
these reconstructions in a way that is consistent, or in other words, if there is a small change
in the input model, there will also be a small change in the output model. This is the property that opens up the possibility
to extend this method to videos! So, how does it compare to previous methods? All of these competing techniques are quite
recent as they are from 2019. They appear to be missing a lot of detail,
and I don’t think we would have a chance of recognizing the target subject from the
reconstructions. And now, just a year and a half later, look
at that incredible progress! It truly feels like we are living in a science
fiction world. What time to be alive! Thanks for watching and for your generous
support, and I'll see you next time!
