You hear it everywhere. DLSS this, DLSS that. But
what is it? And why is it doing the impossible? This AI-based technology is being talked about
the media a great deal, and it is supposedly a way of dramatically speeding up computer games
and virtual worlds. Does this really work? Well, when Jensen Huang, NVIDIA’s CEO says that
every single pixel on your screen for these games will not be rendered, but will be generated soon,
presumably by an AI, that really got my attention. That sounds insane, and if hearing this you think
that this cannot possibly be true, I don’t blame you for a second. We can’t just create all this
footage, there has to be a proper system that can compute reflections, diffuse interactions,
and all that. Right? Well, not quite. In 2018, I had a little project where I wanted to generate
photorealistic images of material models quickly. Now, generating an image through ray
tracing took up to 40 to 60 seconds, and that was a bit too long for my taste.
So I tried to write a neural renderer that could take just the description of a material
model, and it would immediately, in real time, synthesize what it would look like. And it was
able to do all this not in 60 seconds, but in 3 milliseconds. Yes, it was 20,000 times faster.
However, it was limited to this very scene. So, how did this research field improve in the last
5 years? Do we have anything more usable now? Oh boy, if I could tell you. Well, meet DLSS. Deep
Learning Super Sampling. This is a system where hardware and software works together to
create an incredible experience where games and virtual worlds really run faster
than what should be possible. So, how? Well, as of DLSS version 3, get this, by
generating 7 of every 8 pixel that appears on the screen. Yes, more than 85% of the pixels
are just generated, not computed by for instance, ray tracing or traditional techniques. That
sounds flat out impossible. But, it is possible. The first step is running super resolution in
real time. This means that a coarse image goes in, and an AI finds out if we pretended
that this is a fine image instead, what are the details that are missing,
and it synthesizes those details. Second, optical flow. We can look at two adjacent
frames of the video game, and try to estimate what has happened between them, and where things
are going. With this, entire intermediate frames can be synthesized, creating the illusion as if
the game was running smoother than our hardware can run it. Combining the two together, with
a pool of very few pixels that are computed, both image quality and smoothness can be improved
at the same time. What a time to be alive! But it does not stop there. It gets better!
They have also announced DLSS 3.5, hopefully, all this can be improved even further
through something that they call ray reconstruction. Oh my goodness, this
is going to be so good. But, how? Well, normally, when we perform ray tracing, we
simulate the path of millions and millions of rays, and bounce them around in a scene.
And if we can’t afford to wait for up to hours at a time for an image, we will have to
settle for a really noisy image. Like this. Denoising techniques exist that try to guess
what the missing information could be, but they are not perfect. Not even close. Some important
details between objects can be missed entirely, and still, reflections can be a lot less clear
than in a full simulation. And unfortunately, we have a problem. It gets even worse.
This image will undergo an upscaling step, where these errors get magnified even more.
Denoising and upscaling are two separate steps typically done by two separate models, so
why not do both of them with the same AI model? And now, ray reconstruction enters the scene.
It learned on a ton more information than the previous models, 5 times more training
data was given to it. And look at this step. This one is right on the money. This is
tailored to retain high-frequency information, fine details to make sure that before the
upscaling step, we have the highest quality information available. So, is it better than
what denoisers did before? Oh my goodness, look at that! We finally get that shadowy
region back that was filled in by incorrect information with the denoisers before.
Yummy! So good! And that high-frequency information retaining capability can
also be witnessed here. Loving it. Now note that ideally, here, we talk about
peer-reviewed research papers where we can see all the weak points and failure cases, that
is my home turf. This is not a research paper, so it is harder for me to find the flaws,
so please bear in mind that it may have weaknesses that were not shown here. And in
the meantime, this technology is being handed out to millions and millions of people all
around the world, and it is incredible. So, this will be available only for the
shiniest, newest graphics card owners, right? Well, hold on to your papers Fellow
Scholars, because here comes the best part: no! This comes to all RTX graphics cards, even
the older ones that you can pick up for a couple hundred bucks. This can breathe new life to aging
hardware which is highly appreciated. Note that the particular game or app you’re running has to
be ready for DLSS3.5 for all this magic to happen. Now, I am a light transport researcher
by trade, I dream with rays of light, and ray tracing if you will, so I am incredibly
happy to see this technology make it to the hands of real users in the real world in such
an incredible way. And make no mistake, this is not perfect by any means, there are
games where it does not appear to work very well, and some people prefer to not use it at
all, anywhere. However, note that this is still nascent technology, and it can already
synthesize more than 85% of these frames, which is something that I thought would only
be possible in a science fiction movie. Maybe not even there. I am absolutely stunned by these
results, even with their weaknesses. And don’t forget to invoke the First Law of Papers. The
First Law Of Papers says that research is a process. Do not look at where we are, look at
where we will be two more papers down the line.
