Tech Focus: TAA - Blessing Or Curse? Temporal Anti-Aliasing Deep Dive

temporal anti-aliasing also known as TAA is it a blessing or is it a curse TAA is the de facto anti-aliasing in video games these days and whether you think it's just a blurry mess that is ruining game Graphics or you think we have finally reached the Holy Grail of cheap and effective anti-aliasing well guess what TAA is here to stay for the foreseeable future regardless of what you think so what is TAA why do so many games use it what are the downsides what are the upsides that is what this video is going to be about today going in depth for sure but not too far down so as to get lost in the details to understand TAA I think the best place to start looking is why it exists in the first place at one point there wasn't TAA so what came before it and why did it come to existence for about a full decade from the late '90s to around 2010 the best anti-aliasing you could get in a game was super sample anti-aliasing also called ssaa I can demonstrate what super sampling is with Crisis 3 here from 2013 as you're seeing running without anti-aliasing right now at 1080p it's quite jaggy and there's a lot of instability in the image here especially in any of the finer details like the grass or on the edges of the weapons and hands so how do you go about applying super sample anti-aliasing to clean up this jaggy mess here we do that by running the game at a much higher resolution and then compressing it down from that ultra high resolution back into the smaller resolution so for example 8K resolution down to 1080p is known as 8 times super sample anti-aliasing or adex ssaa for short and 4K down to 1080p is is known as four times ssaa in practice it looks like this in Crisis 3 notice how much more stable the image is now with adex ssaa being applied the grass when swing is sharp without any breakup and when we look at the leaves there's much finer detail and the weapon model no longer has that obvious stair-stepped look to the edges and there's just generally less flickering and shimmering in motion I think the vast majority of people would say the image on the right is much superior to the one on the left so super sample anti- eling creates some great results but why is it so uncommon these days and why am I talking about it more or less in the past tense the reason is due to Performance as I said earlier you are running the game at effectively a much much higher resolution and compressing it down now modern GPU like the RTX 3070 when we run Crisis 3 we can see on the left here at the highest settings in game without anti-aliasing at 1080p that the game is running nearly 200 frames per second without it being CPU limited on the right we have four times ssaa which is essentially 4K and the game's frame rate is heavily reduced 70 FPS at adex ssaa the frame rate is but a fraction 19 FPS there Crisis 3 is more than 10 years old and we're only at 1080p here so imagine how slow super sample anti-aliasing would be in a modern game at even higher resolutions like 1440p or 4K since ssaa is expensive and it's essentially always been expensive a compromise solution was developed called multi sample anti-aliasing also known as msaa the real way msaa works is complex on a technical level but I just want you to imagine it simplified as super sampling on the geometry of a game the idea is that hey games are made up of a lot of triangles and instead of multiplying the resolution of the entire image why don't we just multiply the resolution of the geometry and its edges to get a similar effect multi- sample anti-aliasing was transformative and surprisingly cheap back in the day check out this example from the Elder Scrolls Oblivion running at 8X msaa at600 X 1200 on an 8800 GTX this is the first commercial GPU that explicitly supported that very high level of msaa at a very high resolution at that time here we can see a simil smoothing effect like which we saw with super stample anti-aliasing in Crisis 3 obvious jaggies are cleaned up and there's flickering that is going away and the performance hit is lower than what we saw with super sampling the 8800 GTX is losing around 35% of its raw frame rate here with adex msaa and that is much better than the near 90% reduction in frame rate we saw in Crisis 3 on the 3070 with adex ssaa like super sampling there are quality modes too so you can run a 2X mode or a 4X mode with less reduction of the jagged edges but also less of a performance hit msaa was a great compromise so why do nearly all modern games not use it there are a couple reasons the first is due to the rise of deferred rendering now I could spend a lot of time here but you just need to know that deferred rendering is introduced into game engines to increase rendering complexity while not being performance intensive an example of a deferred rendering type would be deferred lighting which could greatly increase the amount of visible lights on screen versus previous methods this is a huge Topic in its own right but the whole point is that many game engines started switching to these deferred approaches to enable new visual quality Killzone 2 is a famous historical example but with many other examples existing before and after it deferred rendering is largely incompatible with multi- sample anti-aliasing as it has been done in the past msaa could be added to to a game but at a great cost in working hours and it would typically have worse performance so as time went on less and less games would ship supporting msaa and those that did were quite rare for example Crisis 3 that I just showed off that supports multi- sample anti-aliasing even though it uses aspects of deferred rendering turning on msaa to 8X in that game nearly has the frame rate at 1080p on the 3070 and it has a higher relative frame time increase than what we saw in Oblivion earlier on the 8800 GTX at 4K it's even worse the FPS is more than half and the frame time cost of msaa is more than the entire cost of rendering the game alt together without AA on that is a huge cost performance on a very modern GPU I would say in a game that is more than a decade old dovetailing into this was the fact that multi- sample anti-aliasing was getting worse at its own job remember msaa in principle is only going to be working on Geometry for the most part that's perfect for older games like Jedi Academy from 2003 where the graphics are mainly made up of simple textures and just geometry but msa's effect of working on Geometry became less useful when pixel shading started to dominate The Way games looked things like normal Maps as popularized by Doom 3 were making game lighting and object detail being driven by textures instead of geometry alone and many other aspects of shading were starting to be done with new ways yet had stuff like volumetric lighting popping up in many games or screen space Amun occlusion that was growing in use all these new effects had little to do with the geometry that multi- sample anti-aliasing was going to be affecting for example if we go back to Crisis 3 you can see some issues with msaa in this scene I showed off earlier msaa was doing a good enough job it was cutting the frame rate in half on the 3070 but it was doing its work now let's look at this scene here but we can see already msaa is not working very well as the camera is moving we can see a lot of jaggies and flickering present in the metal on this scene here this is called specular aliasing the right side with msaa is flickering almost as much as the left side without any anti-aliasing at all so in a scene like this one you could be potentially having your frame rate with adx MS a but it's not effectively making the game less Jagged or take this scene from the intro of the game and there's a wet water shading effect being applied to the pistol and nearly the entire scene notice how the right side with multi- sample anti-aliasing looks no different than the left side without it so the frame rate is going to be haved here for no visual Improvement this issue of msaa being ineffective due to specialized shading is perhaps its biggest issue games nowadays use even even more complex shading and they're shinier than ever before and msaa is really not built to deal with that as a stop Gap after msaa became less tenable historically developers came up with postprocess anti-aliasing Solutions like FXAA or smaa you don't need to know what those acronyms mean it's just you need to know that these Solutions look for Visible edges in a scene after it's already been rendered so they do it in a postprocess kind of way and they try and blur and soften edges that they detect or try to detect and this still here we can see how they're working we can see some edges are looking cleaner and less Jagged though with FXAA we can see that the entire image is actually being softened as a byproduct how it works it's pretty inaccurate so they look like they're working in this still here I would say and if we look at the frame rate hit we can see that they're cheap to render they're practically free when compared to no anti-aliasing they are lot cheaper than msaa that's for sure the problem is these are still images games actually move and as soon as we start showing some movement we can see some of the issues becoming apparent none of the jagged edges are actually really being cleaned up in motion we still see Flicker and Shimmer on all edges much like you would see with no anti-aliasing these stop Gap postprocess Solutions like smaa and FXAA don't actually help in aggregate as they don't understand how images are moving they're cheap but they're not actually good at anti-aliasing so this is where developers started to come up with temporal anti-aliasing or TAA to replace things like FXAA and SMA the concept for TAA was to leverage the idea of super sample anti-aliasing but instead of feeding that image and getting it information from a larger image like 8K they would get more information from previous frames so on temporal terms over time it is actually super sampling done over time that is why you can sometimes see TAA being labeled as tssaa in game options temporal super sampling so what are the advantages of TAA a big advantage of TAA is that it has similar Behavior to Super sampling the name so it cleans up all jaggies of all types where msaa would break with Shader aliasing as I showed earlier well temporal AA was going to clean that up quite nicely a lot like how ssaa would since it works like super sampling TAA can affect things that are not made of just geometry like in control here R tracing effects in that game get proper Edge smoothing and clean up with TAA which they don't get with msaa at all is that is really only going to meaningly affect G geometric edges the super sampling effect of TAA is really profound and certain forms of TAA like nvidia's daa can actually supersede quality aspects of traditional super sampling check this image here Crisis 3 we're looking at standard ssaa here so super sampling of the old variety when we move the camera backward and forward I want you to notice how in the distance there's these horizontal and perpendicular lines the detail on those lines L is snapping this creates a flickering in Motion daa in comparison it's going to produce a different image but if you look at those lines you'll notice that they're no longer flickering in the same way they are with traditional ssaa this super sampling property was impressive and it was quickly exploited by developers and many games designed their effects like Reflections volumetric leting Shadow filtering Amon occlusion or hair rendering to only look really clean with TAA being being on typically developers run these at lower resolution or lower quality then they have TAA clean them up to look much higher quality than they actually really are thus they save on performance so developers are not just using TAA to clean up Jagged edges like old anti-aliasing they're also using it to optimize game effects and height where they cut corners on quality so as that implies performance is a huge part of ta a check out this example from Deus X mankind divided you can see here just how much performance TAA saves versus old Alternatives TAA here reduces the average frame rate by Just Around 3% versus no anti-aliasing this is completely different than msaa which is otherwise very expensive here with TAA you can get a great High frame rate experience that is not at all possible with msaa especially in the 4X or 8X variety here here on the RTX 370 at 1080p this is a 2016 game on a GPU from 2020 and only at 1080p think about that I think this is a great example to show how the cards were being heavily stacked in ta's favor to become adopted versus prior techniques like smaa or msaa nowadays TAA has been adopted across the entire industry but it's seen a lot of iteration since its first Little Steps it's started off as genuinely poor and distracting in titles like Halo Reach where that game's TAA is so rudimentary that the entire game looks like it's constantly having ghosted blurs in every single frame back then people thought that was the game's motion blur and it wasn't it was just the game's really poor TAA compar these humble and ugly Beginnings in 2010 to something like dlss in 2024 now this is a form of AI assisted TAA it doesn't just smooth Jagged edges it allows entire games to be rendered at a much lower resolution like we see here in control with the LSS on and then be super sampled over time to look like a much higher resolution and the quality is surprisingly high so TAA nowaday has evolved into something which allows us to reduce the entire resolution of an image and then upscale it up to a higher one and it actually looks like that higher resolution image when I talk about TAA like this focusing on its positive aspects it admittedly sounds pretty awesome better performance and better anti-aliasing quality than previous techniques but that's just the positive stuff in isolation if we look at the negative aspects of which there's a laundry list of them we can see that there's some core issues with TAA the one of the first ones is Clarity a key issue with TAA is that it resolves softened images now part of this is is just the nature of what anti-aliasing is even super sample anti-aliasing can be considered soft it's softening edges smoothing them out to make the image less sharp technically and less fizzily in motion and depending upon the method of information and how it's compressed with ssaa the level of softness will be different by linear interpolation as we're seeing here looks a lot softer than say Lanos that's just a technicality but even if one considers how anti-aliasing technically softens an image TAA is still inherently softer due to how it's integrating information from previous frames and it does that by jittering the current frame always ever so slightly to achieve its effect this leaves TAA resolving typically softer than other anti-aliasing techniques by default you may find this pleasingly soft or you may find this annoyingly blurry a reason why you may find it blurry or you may find it good is based upon how far far you are from your screen and how high resolution the game is if you're further away from your screen and the game you're playing on it you may not notice softness and you may just appreciate macro aspects of TAA like how good it is at anti-aliasing but if you're closer to a screen the softness may be Amplified and it'll become more obvious and you may dislike it that softness then becomes even more obvious if the screen is lower resolution check out this shot here of this game running at 1080p without AA then 1080p with TAA and let's compare this with 4K without TAA and 4K with TAA I would say there's a great difference in softness when TAA is turned on with 1080P and it's much lessened at 4K at 4K I would say it's almost a nonissue while at 1080p I would say it could be particularly agrees in a game like this this aspect of screen distance and resolution making TAA seem seem better or worse is key to the general problem with TAA and why some people really don't like it for people who play console games for example they often play on high resolution televisions a few meters away from them the game might be at 30 FPS but due to the resolution and the distance the softness of something like TAA is not so bothersome but on PC many people are playing at much closer to their screens and the resolutions are often lower so if a game developer balances image quality to favor the console experience then they leave a large number of PC users perhaps out to dry as they're closer to their screens and the lower resolution they're playing at might lead to a subjectively softer and worse experience where the flaws of TAA are much more noticeable another issue with TAA is how it blurs on movement check this out Halo infinite with TAA on versus TAA off at 1080p I think the issue here is pretty easy to see in this game when it's moving but let's pause the recording and highlight it so yes TAA is making Jagged edges go away here as it should but it's also softening the entire image I can say that it's softening the entire image by comparing it to Super sampling which is supposed to be a quote unquote Perfect Image we can see the issue if I highlight it here Jagged edges are being softened with this blue but the internal Edge detail here marked with red is being softened as well and that's honestly bad this Behavior makes the whole image look like it's blurred or softened same with this shot here which is a bit more obvious actually if we look at the edge detail on the side of the Pelican here we can see the line is perfectly cleaned up with TAA but notice how the detail and lines on the ground with TAA that are there without anti-aliasing and that are there with ssaa while they're complete complely missing and smooth out of existence with TAA on now Halo infinite's TAA is especially blurry I would say but I could still find this behavior in all types of TAA it's not a bug it's a feature how much you will notice it though is a subjective question if you're lower resolution and closer to your screen you'll notice it more you'll also notice it more the more the screen is moving so Mouse users will definitely notice this more than controller users another aspect of TAA that plays into all this is that it's frame rate dependent it starts looking better the higher a frame rate is you can see this here in this example from cyberp Punk's Benchmark if I run the game at 30 FPS 60 FPS and 120 FPS we can see how the image produced with 120 FPS looks better and that's not just because it looks better because the frame rate's higher if we stop images we can see how there's less errors in the image like less weird ghosted frames here and less artifacts at 120 FPS versus 60 and 30 this is because the previous frames that TAA is using are spatially closer together there's literally less room for error if we look at this shot here from that same sequence we can also see that higher FPS means better anti-aliasing look how much better the edge anti-aliasing is here there's less jaggies at 120 FPS than 60 or 30 if you play games at a high frame rate you may actually like ta's effect here but I would say it kind of biases ta's usefulness it makes it less useful for certain types of games because fast movement is going to look worse and it also makes you want to Target a high frame rate meaning you'll have to turn down your graphical Ambitions another less than favorable aspect of TAA is how it can create visible Jitter in an image let's go back to that shot of control earlier that I showed off if we look at this area this time in the image we can see that the TAA side of the images here well you can see kind of an active Jitter there kind of weird looking it's a small little part of the image but you can find this in nearly all TAA techniques due to the fact that they Jitter to make this super sampling effect and the lower your output resolution is and the closer you are to your screen the more obvious this error will be when it happens perhaps the most infamous negative aspect of TAA is when it really fails hard and it produces a lot of ghosting from previous frames here's a great example from Doom 2016 a game where the temporal anti-aliasing was lotted at release you go into Samuel Hayden's office and yeah tons of ghosting following any and all movement of his hands as TAA is making less than favorable judgments about which stuff to keep from previous frames and which stuff not to keep now this is not the default way TAA should look but nearly every type of ta a is capable of doing this and I've seen this Behavior hundreds of times in my years of reviewing games for digital Foundry at any and every resolution even dlss which is kind of the Apex of current TAA techniques can have this issue and I've showed it off a lot on the channel the last negative aspect of TAA that I want to cover is actually a positive one that I mentioned earlier you know in games like control that I showed earlier where there's Reflections or sh Shadows which only look properly highres with TAA on I said that's a good thing as it allows developers to optimize their games in new ways well it's also a bad thing when you think about it as if you don't want to use TAA and you turn it off you're going to get blotchy effects or low resolution effects and a lot of game menus won't allow you to turn up effect quality individually without TAA on that's not a very good thing so while TAA definitely has a lot of great aspects it has a lot of negative ones as well so what is my opinion here well I kind of look at this in a more holistic way historical I think without TAA real-time Graphics would not have progressed to the point we are at now things like real-time path tracing in AAA games such as cyberpunk just doesn't exist without the concepts that TAA brought to the table using information from previous frames on the counil side of things without ta ta a the difference between the console Generations would actually be more minor than it currently is right now and I'm being serious there without TAA existing a lot of computational power would be spent on cleaning up Jagged edges and not improving other aspects of rendering like Lighting in Geometry it's always a push and a pull here things like the Matrix awakens demo on Xbox series consoles or Playstation yeah that wouldn't be possible without TAA coming into existence it evolved into unreal engines TSR or temporal super resolution and made that possible in the first place on Console hardware and it looks great so as I see it TAA has been a great thing in spite of the negative aspects but I am merely one voice and there's a great plurality of voices that should be heard beyond my own I for example play games at 4K and that forms the basis of my experience ta's issues are less visible at 4K and more visible at lower resolution solons like 1080p this is critical as a huge amount of users still play games at 1080p based on the hardware survey from Steam and since ta's downsides are more visible there and those downsides are ghosting and blur well then guess what the industry I think needs to treat TAA like other options which add blur to games TAA should be able to be turned off much like motion blur should always be able to be turned off in a game as that is an accessibility option for people who get motion sickness furthermore I think there should be a simple standard alternative available for users should they not wish to use TAA in those Spider-Man ports by nixes those games on Console were definitely designed around TAA being there but you can turn off TAA in these games on PC and elect to use alternative methods like smaa sure they're not good actually at cleaning up Jagged edges but they're better than nothing and they're simple to have as an alternative the last reason why I think TAA should be able to be toggled off is for future scaling right now TAA is there because it's convenient but in 10 years maybe you don't actually need it to get good image quality maybe you can just super sample the game in making this video I loaded up a lot of games that were heavily super sampled on big gpus and I thought it looked pretty awesome without TAA being enabled so I think developers should allow TAA or things like dlss or FSR or TSR whatever they're called to be turned off so that the game can be future proofed and you could run it in future Hardware with ssaa enabled and it'll look that much better without TAA being on simply put if people don't like TAA they should be able to turn it off I may love TAA but a whole bunch of other people may not and they should be able to turn it off much like they can with other accessibility options with that being said maybe you and the audience don't love TAA but I hope you did love this video and all I had to say about TAA here if you did like the video covering TAA please hit that like button and subscribe to the channel if you're already a subscriber hit that little bell in the corner to be informed as soon as digital Foundry posts a video support DF on patreon to enable more videos like this Tech Focus Series in the future comment below follow on Twitter and as always this is Alex SP fway Al see

Channel: Digital Foundry

Views: 191,669

Keywords:

Id: WG8w9Yg5B3g

Length: 28min 4sec (1684 seconds)

Published: Sun Feb 11 2024