Video games have spectacular graphics, capable of
transporting you to incredibly detailed cities, heart-racing battlegrounds, magical
worlds, and breathtaking environments. While this may look like an old western train
station and locomotive from Red Dead Redemption 2, it’s actually composed of 2.1 million
vertices assembled into 3.5 million triangles with 976 colors and textures
assigned to the various surfaces, all with a virtual sun illuminating the scene below.
But perhaps the most impressive fact is that these vertices, textures, and lights are entirely
composed of ones and zeroes that’s continuously being processed inside your computer’s
graphics card or a video game console. So then, how does your computer take billions
of ones and zeroes and turn it into realistic 3D graphics? Well, let’s jump right in.
The video game graphics rendering pipeline has three key steps: Vertex Shading, Rasterization,
and Fragment Shading. While additional steps are used in many modern video games, these three core
steps have been used for decades in thousands of video games for both computers and consoles and
are still the backbone of the video game graphics algorithm for pretty much every game you play.
Let’s begin with the first step called vertex shading. The basic idea in this step is to
take all the objects’ geometries and meshes in a 3D space and use the field of view of the
camera to calculate where each object falls in a 2D window called the view screen, which
is the 2D image that’s sent to the display. In this train station scene, there are 1,100
different models and the camera’s field of view sections off what the player sees, reducing the
number of objects that need to be rendered to 600. Let’s focus on the locomotive as an example.
Although this engine has rounded surfaces and some rather complex shapes, it’s actually
assembled from 762 thousand flat triangles using 382 thousand vertices and 9 different
materials or colors applied to the surfaces of the triangles. Conceptually, the entire train
is moved as one piece onto the viewscreen, but actually, each of the train’s hundreds of
thousands of vertices are moved one at a time. So, let’s focus on a single vertex. The
process of moving a vertex, and by extension, the triangles and the train, from a 3D world onto
a 2D view screen is done using 3 transformations. First moving a vertex from model space to world
space, then from world space to camera space, and finally from the perspective field of view onto
the view screen. To perform this transformation we use the X,Y, and Z coordinates of that
vertex in modeling space, then the position, scale, and rotation of the model in world space,
and finally the coordinates and rotation of the camera and its field of view. We plug all
these numbers into different transformation matrices and multiply them together resulting
in the X and Y values of the vertex on the view screen as well as a Z value or depth, which
we’ll use later to determine object blocking. After three vertices of the train are transformed
using similar matrix math, we get a single triangle moved onto the view screen. Then the
rest of the 382 thousand vertices of the train and the 2.1 million vertices of all the 600
objects in the camera’s field of view undergo a similar set of transformations, thereby moving
all 3.5 million triangles onto a 2D viewscreen. This is an incredible amount of matrix
math, but GPUs in graphics cards and video game consoles are designed to be triangle mesh
rendering monsters and thus have evolved over decades to handle millions of triangles every few
milliseconds. For example, this GPU has 10,000ish cores designed to efficiently execute up to 35
trillion operations of 32-bit multiplication and addition every second, and, by distributing the
vertex co-ordinates and transformation data among each of the cores, the GPU can easily render the
scene resulting in 120 or more frames a second. Now that we have all the vertices moved onto a 2D
plane, the next step is to use the 3 vertices of a single triangle and figure out which specific
pixels on your display are covered by that triangle. This process is called rasterization.
A 4K monitor or TV has a resolution of thirty-eight forty by twenty-one sixty,
yielding around 8.3 million pixels. Using the X and Y coordinates of the vertices
of a given triangle on the view screen, your GPU calculates where it falls within this
massive grid and which of the pixels are covered by that particular triangle. Next, those pixels
are shaded using the texture or color assigned to that triangle. Thus, with rasterization,
we turn triangles into fragments which are groups of pixels that come from the same
triangle and share the same texture or color. Then we move on to the next triangle and
shade in the pixels that are covered by it and continue to do this for each of the
3.5 million triangles that were previously moved onto the viewscreen. By applying the Red
Blue and Green color values of each triangle to the appropriate pixels, a 4K image is formed
in the frame buffer and sent to the display. You’re probably wondering how we account
for triangles that overlap or block other triangles. For example, the train is blocking the
view of much of the train station. Additionally, the train has hundreds of thousands of triangles
on its backside that are sent through the rendering pipeline, but obviously don’t appear in
the final image. Determining which triangles are in front is called the visibility problem and
is solved by using a Z-buffer or Depth Buffer. A Z-Buffer adds an extra value to each of the
8.3 million pixels corresponding to the distance or depth that each pixel is from the camera.
In the previous step, when we did the vertex transformations, we ended up with X and Y
coordinates, but then also got a Z value that corresponds to the distance from the
transformed vertex to the camera. When a triangle is rasterized, it covers a set
of pixels and the Z value or depth of the triangle is compared with the values stored in the
Z-Buffer. If the triangle’s depth values are lower than those in the Z-buffer, meaning the triangle
is closer to the camera, then we paint in those pixels using the triangle’s color and re-place the
Z-buffer’s values using that triangle’s Z-values. However, let’s say a second triangle comes along
with Z values that are higher than those in the Z-buffer, meaning the triangle is further away.
We just throw it out and keep the pixels from the triangle that was previously painted
with lower Z-values. Using this method, only the closest triangles to the camera with the
lowest Z-values will be displayed on the screen. By the way, here’s the image of the Z or Depth
buffer, wherein black is close and white is far. Note that because these triangles are in 3D space,
the vertices often have 3 different Z values, and thus each individual pixel of the triangle needs
its Z value computed using the vertex coordinates. This allows intersecting triangles to properly
render out their intersections pixel by pixel. One issue with rasterization and these pixels is
that if the triangle cuts at an angle and passes through the center of the pixel, then the
entire pixel is painted with that triangle’s color resulting in jagged and pixelated edges.
To reduce the appearance of these jagged edges, graphics processors implement a technique
called Super Sampling Anti-Aliasing. With SSAA, 16 sampling points are distributed across a single
pixel, and when a triangle cuts through a pixel, depending on how many of the 16 sampling
points the triangle covers, a corresponding fractional shade of that color is applied to
the pixel, resulting in faded edges in the image and significantly less noticeable pixelization.
One thing to remember is that when you’re playing a video game, your character’s camera view as
well as the objects in the scene are continuously moving around. As a result, the process and
calculations within vertex shading, rasterization, and fragment shading are recalculated for every
single frame once every 8.3 milliseconds for a game running at 120 frames a second.
Let’s move onto the next step which is Fragment Shading. Now that we have a set
of pixels corresponding to each triangle, it’s not enough to simply paint by number to color
the pixels. Rather, to make the scene realistic, we have to account for the direction and
strength of the light or illumination, the position of the camera, reflections, and
shadows cast by other objects. Fragment shading is therefore used to shade in each pixel
with accurate illumination to make the scene realistic. As a reminder, fragments are groups of
pixels formed from a single rasterized triangle. Let’s see the fragment shader in action. This
train engine is mostly made of black metal, and if we apply the same color to each of its
pixel fragments, we get a horribly inaccurate train. But once we apply proper shading, such
as making the bottom darker and the top lighter, and by adding in specular highlights or shininess
where the light bounces off the surface, we get a realistic black metal train. Additionally, as the
sun moves in the sky, the shading on the train reflects the passage of time throughout the day,
and, if it’s night, the materials and colors of all the objects are darker and illuminated from
the light of the fire. Even video games such as Super Mario 64 which is almost 30 years old have
some simple shading where the colors of surfaces are changed by the lighting and shadows in the
scene. So, let’s see how fragment shading works. The basic idea is that if a surface is pointing
directly at a light source such as the sun, it’s shaded brighter whereas if a
surface is facing perpendicular to, or away from the light, it’s shaded darker.
In order to calculate a triangle’s shading, there are two key details we need to know.
First, the direction of the light and second, the direction the triangle’s surface is facing.
Let’s continue to use the locomotive as an example and paint it bright red instead of black. As
you already know, this train is made of 762 thou-sand flat triangles, many of which face
in different directions. The direction that an individual triangle is facing is called its
surface normal, which is simply the direction perpendicular to the plane of the triangle, kind
of like a flagpole sticking out of the ground. To calculate a triangle’s shading, we take the
cosine of the angle or theta between the two directions. The cosine theta value is 1 when the
surface is facing the light and when the surface is perpendicular to the light it’s 0. Next, we
multiply cosine theta by the intensity of the light and then by the color of the material to
get the properly shaded color of that triangle. This process adjusts the triangles’ RGB values
and as a result, we get a range of lightness to darkness of a surface depending on how its
individual triangles are facing the light. However, if the surface is perpendicular or
facing away, we don’t want a cosine theta value of 0 or a negative number because this would
result in a pitch-black surface. Therefore, we set the minimum to 0 and add in an ambient
light intensity times the surface color, and adjust this ambient light so that it’s higher
in daytime scenes, and closer to 0 at night. Finally, when there are multiple light sources
in a scene, we perform this calculation multiple times with different light directions, and
intensities and then add the individual contributions together. Having more than a few
light sources is computationally intense for your GPU, and thus scenes limit the number
of individual light sources and sometimes limit the range of influence for the lights
so that triangles will ignore distant lights. The vector and matrix math used in rendering
video game graphics is rather complicated, but luckily there’s a free and easy way to learn
it and that’s with Brilliant.org. Brilliant is a multidisciplinary online interactive education
platform and is the best way to learn math, computer science, and many other
fields of science and engineering. Thus far we’ve been simplifying the math behind
video game graphics considerably. For example, vectors are used to find the value of cosine theta
between the direction of the light and the surface normal, and the GPU uses the dot product divided
by the norm of the two vectors to calculate it. Additionally, we skipped a lot of detail when it
came to 3D shapes and transformations from one coordinate system to another using matrices.
Rather fittingly, Brilliant.org has entire courses on vector calculus, trigonometry, and 3D
geometry, as well as courses on linear algebra and matrix math. All of which have direct
applications to this video and are needed for you to fully understand graphics algorithms.
Alternatively, if you’re all set with math, we recommend their course on Thinking in
Code which will help you build a solid foundation on computational problem solving.
Brilliant is offering a free 30-day trial with full access to their thousands of
lessons. It’s incredibly easy to sign up, try out some of their lessons for free and, if you
like them, which we’re sure you will, you can sign up for an annual subscription. To the viewers
of this channel, Brilliant is offering 20% off an annual subscription to the first 200 people who
sign up. Just go to brilliant.org/brancheducation. The link is in the description below.
Let’s get back to exploring fragment shading. One key problem with it is that the triangles
within an object each have only a single normal, and thus each triangle will share the
same color throughout the triangle’s surface. This is called flat shading and
is rather unrealistic when viewed on curved surfaces such as the body of this steam engine.
So, in order to produce smooth shading, instead of using surface normals, we use one normal for
each vertex calculated using the average of the normals of the adjacent triangles. Next, we
use a method called barycentric coordinates to produce a smooth gradient of normals across the
surface of a triangle. Visually it’s like mixing 3 different colors across a triangle, but instead
we’re using the three vertex normal directions. For a given fragment we take the center of each
pixel and use the vertex normals and coordinates of the pre-rasterized triangle to calculate the
barycentric normal of that particular pixel. Just like mixing the three colors across a triangle
this pixel’s normal will be a proportional mix of the three vertex normals of the triangle.
As a result, when a set of triangles is used to form a curved surface, each pixel will be part
of a gradient of normals resulting in a gradient of angles facing the light with pixel-by-pixel
coloring and smooth shad-ing across the surface. We want to say that this has been one of the most
enjoyable videos to make simply because we love playing video games and seeing the algorithm
that makes these incredible graphics has been a joy. We spent over 540 hours researching,
writing, modelling this scene from RDR2, and animating. If you could take a few seconds
to hit that like button, subscribe, share this video with a friend, and write a comment below it
would help us more than you think, so thank you. Thus far we’ve covered the core steps for
the graphics rendering pipeline, however, there are many more steps and advanced topics. For
example, you might be wondering where ray tracing and DLSS or deep learning super sampling fits into
this pipeline. Ray tracing is predominately used to create highly detailed scenes with accurate
lighting and reflections typically found in TV and movies and a single frame can take dozens
of minutes or more to render. For video games, the primary visibility and shading of the objects
are calculated using the graphics rendering pipeline we discussed, but in certain video
games ray tracing is used to calculate shadows, reflections, and improved lighting. On the other
hand, DLSS is an algorithm for taking a low resolution frame and upscaling it to a 4K frame
using a convolution neural network. Therefore DLSS is executed after ray tracing and the graphics
pipeline generates a low-resolution frame. One interesting note is that the latest generation
of GPUs has 3 entirely separate architectures of computational resources or cores. CUDA or Shading
cores execute the graphics rendering pipeline. Ray tracing cores are self-explanatory. And then
DLSS is run on the Tensor cores. Therefore, when you’re playing a high-end video game with
Ray Tracing and DLSS, your GPU utilizes all of its computational resources at the same time,
allowing you to play 4K games and render frames in less than 10 milliseconds each. Whereas if you
were to solely rely on the CUDA or shading cores, then a single frame would take around
50 milliseconds. With that in mind, Ray Tracing and DLSS are entirely different topics
with their own equally complicated algorithms, and therefore we’re planning separate videos
that will explore each of these topics in detail. Furthermore, when it comes to video game graphics,
there are advanced topics such as Shadows, Reflections, UVs, Normal Maps and more. Therefore,
we’re considering making an additional video on these advanced topics. If you’re interested
in such a video let us know in the comments. We believe the future will require a strong
emphasis on engineering education and we’re thankful to all our Patreon and YouTube Membership
Sponsors for supporting this dream. If you want to support us on YouTube Memberships, or Patreon,
you can find the links in the description. This is Branch Education, and we create
3D animations that dive deeply into the technology that drives our modern
world. Watch another Branch video by clicking one of these cards or click here
to subscribe. Thanks for watching to the end!
