[MUSIC PLAYING] SPEAKER: Welcome to another
episode of the Flutter Build Show. When we last saw our heroes,
that's us by the way, we were locked in mortal
combat with a custom render object over the layout
of a dynamic chat bubble. Today, we'll explore a very
different way to render things in Flutter that barely
uses widgets at all. But first it's worth
taking a step back to look at the big picture of
how Flutter renders anything. For this I think it's
best to work backwards. One thing we know for
certain is that by the end of this whole
process, our screens draw what we asked them to. So let's call that final
step pixels on the screen. Now, before that, what
color each of those pixels should show is
computed by the GPU in small programs
called fragment shaders. This is a GPU's whole
purpose in life. It's why they were invented and
why your computer or phone has one, to super
parallelize the execution of tiny shader programs that
color your device's screen one pixel at a time. And it's worth reflecting
on what this graph means. Later on, higher
in this flowchart, we'll see different
APIs to render things. But they all funnel down
through this ending. This means even though
everyone in Flutter seems obsessed with
fragment shaders recently, they're not actually new. Every pixel of every
Flutter counter app ever run was rendered via a fragment
shader built into the Flutter framework or into Skia itself. So shaders are
everywhere, whether or not we know about them. OK, back to the graph. Above shaders is a
highly abstracted layer called the canvas API. Now, the canvas API allows you
to make declarative graphics calls like adding
shapes or custom paths and filling them with colors. Render objects across our app
integrate with the canvas API in this same way. Internally the
canvas API's job is to translate code like this
into the appropriate shaders and then shuttle
them off to the GPU. In the ancient Roman world,
it was said all roads lead to Rome. But in Flutter, all roads
lead to the canvas API. One possible road is the
most common Flutter story. Widgets that build
render objects. This is what we covered
in the previous episode on building custom
render objects. Every widget that draws
anything to your screen does so by eventually
adding a render object to the render tree. But you can also
write your own ultra thin widgets that immediately
delegate to a specialized render object of your design. But a custom render object
isn't the only way to get cute. The other way to render
things in Flutter is to use a custom
painter and draw whatever you want directly. Technically the
custom painter also requires a widget
called CustomPaint, but it's so fundamentally
different from normal widgets that it deserves its
own branch in our chart. Now, if you've used a
custom painter before, then maybe you've seen its paint
method, whose first parameter is a canvas object. This is your handle
to the canvas API. Well, you can do lots of neat
stuff in that paint method. And one of them is
to say, hey, look, I got this shader
over here that's prepared to tell you what
color to use for any given pixel in this space. Please ask it any questions. And Flutter will, in fact,
have a lot of questions, like what color should
the pixel at 0, 0 be and 0, 1 and 1, 0 and so on? And it'll ask the
shader over and over and over again hundreds of times
in parallel, because that's a GPU's purpose in life. Be super good at
parallelizing math. OK, so that's a primer. This episode of
the build show is dedicated to that
path on the right where you use a custom paint
widget, a custom painter, and then specifically tell your
custom painter to use a shader. Let's get into it. First things first, shaders
are not written in Dart. They're written in
another language called GLSL, standing for
Open GL Shading Language. Now, the anatomy of a
shader isn't so bad. At the top, you
choose an API version by selecting your desired
SemVer major and minor release without periods. This declaration would
accept any 4.60 release. Then, include this file to get
a few Flutter specific helpers. Next, you declare any
parameters your shader should accept from the outside
world with the uniform keyword. The syntax is uniform then the
type, then the variable name. By convention, such variable
names start with the letter U. And the word uniform
was chosen to indicate that these variables
will have the same value across every pixel this shader
computes for this frame. It looks like this code would
accept two parameters, a size and a color. But GLSL doesn't have
complex types like that, so it's also fair to
think of this shader as accepting six floats,
which it will bundle into two vectors of length two and four. GLSL makes heavy use of plain
old vectors of lengths two, three, or four. Next, declare what variable
your shader will return. This should always
be a vec4, which is a color with full RGBA values. Your only choice here
is the variable name. At last we're ready to write
the actual guts of our shader. Start with a main
method whose job it is to assign a value to the
variable we indicated up top. In this case, frag color. Let's aim for the Hello World
of shaders for our first go and assign the same
color to every single pixel. Now, which color should we use? Well, how about the
color we passed in? RGB and A are helpful
getters on the vec4 class since it's so common
to use them for colors. Now I wrote the line
that way the first time to show the red green,
blue, and alpha getters on the vec4 class, but we
just unwrapped a vec4 only to assign it right
back to a vec4. That means we could
have also written this. So that's the simplest
shader possible. Now let's return to our
Flutter code and plug this in. I'm in a fresh Flutter project
created by Flutter create --empty with my
shader sitting at assets/shaders/my_shader.frag. To start I'll include that
shader in my pubspec.yaml file. Next, I'll replace
everything in my material app with a custom paint widget,
which in turn requires a subclass of custom painter. Thinking back to
our shader, we said we were going to
pass it a color. So I'll declare one in this
custom painter's constructor. Now scrolling back up to our
widgets, let's keep our word and supply a value. How about colors.green
this time to mix it up? Everybody uses colors.blue. I'll select create two missing
overrides in the quick fix menu and we're ready to go. The first method we should
consider is should repaint. To understand this method,
note that a new instance of our custom painter
is created every time this wing of the
widget tree updates. That means custom
painters are cheap objects similar to widgets themselves
that receive initial values and then never change. The job of should repaint is to
compare ourselves, the newest version being inserted
into the widget tree, with the previous
version and work out if we'd even do
anything differently than our predecessor. In our case, that's a
matter of whether or not our color is different
from old delegate.color. If they're different,
we should repaint. As is, the editor isn't
too happy with us. But this is just a wrinkle
of Dart's type system. We can safely change
covariant custom painter to my painter, which
allows Dart to trust us that it will find the color
property on old delegate. Next, we get to implement paint. To start, let's set up the most
basic parts of the structure. This method will end by calling
canvas.drawrect, defining a rectangle that fills
all available space and supplying a paint object. So far this paint
object does nothing, but that is where our
shader will come into play. We can assign our shader
to the paint object by using the cascade operator
and setting it directly. The only issue is the shader
obviously doesn't exist yet. Scroll up to our
painter's constructor and add a new parameter,
shader of type fragment shader, and a corresponding class field. Note, for this to work,
you'll have to import Dart UI. Now main app is
complaining because we've invalidated its build method. Scroll to the very
top of the file and declare a global
variable fragment program. I know, global variables
aren't the best. But the Flutter shaders package
has a nice helper widget called Shader Builder that
abstracts this for us, and we should all be using that. But I want to first show
everyone what that Shader Builder widget abstracts. So in your main method,
use the fragment programs from asset static method to grab
the shader we made available in our pubspec.yaml file. That returns a future. So I'll make the main
method asynchronous, slap the awake
keyword on the front, and we're good to continue. We're ready to scroll back
down to my shader and supply fragmentprogram.fragmentshader
as the value to the required parameter shader. The fragment program is
the pointer to the asset. But what our widget needs
is the compiled shader itself returned from the
fragment shader method. Again, in my real apps, I let
the Flutter shader package's Shader Builder widget
handle all of this. Now the limiter is
happy, but this time it's actually a false positive
coming from the fact that the Dart analysis server
can't see into fragment shaders and understand what
parameters they expect. If we ran our app now, the
shader would immediately crash. Sadly, the next part of our
code won't be type safe. I'm switching back to
our fragment shader to remind myself what
parameters I even said I'd pass. First, the shader expects two
floats for its total surface area. And then, four more
floats to represent the color it should paint. OK, back to our Dart code. The way to plumb parameters
through to a shader is to use methods on
the fragment shader class called set and then
the type of the variable we're passing. In this case, a float. First pass size.width
then size.height. Another oddity is
that leading integer, which orders our variables for
consumption on the shaders end. It's a little unusual, I admit. Next send our color through by
applying each of its values one at a time. Only this isn't quite right
because of an interesting quirk in how shaders think
about some floats. If you've done much
with colors before, you may know them as defined
by three numbers, one for red, green, and blue,
each ranging from 0 to 255. Well, GLSL and the
land of shaders has a bone to pick
with that system. In GLSL, floats for some
data types, including colors, work on normalized
values between 0 and 1. Here's a quick primer for how
to think about all of this. Consider this arbitrary color
written as you'd commonly see it in Dart code. The 0x in the front doesn't
influence the number itself. It merely tells Dart that what
comes next is not represented by standard decimal
notation but instead by hexadecimal notation. Then we encounter four
channels, each occupying two hexadecimal characters. Compare that with how
GLSL represents colors. In GLSL, colors are represented
by vectors, essentially fixed length arrays of decimal
floating point numbers. Not only is the order
slightly different, with the alpha channel,
that's the opacity by the way, moved to the end,
but the decimal floating point values are normalized
between 0 and 1. So with 0, dark colors you're
used to would stay a 0, the middle value 128 in
decimal or 80 in hexadecimal would be close to 0.5, and the
maximum value, 255 in decimal or FF in hexadecimal
would be 1.0. So that's the target format
we're looking to support. But how do we do this in Dart? The conversion is achieved by
dividing each number by 255. There. Now we're correctly passing
our color value to our shader. Let's run our app. And look at that. A big green square. Now this might seem
like going around the block to get next door
compared to writing colored box, color, colors.green. But it's actually the opposite. If you draw a green rectangle
using a colored box, it goes around the block by
using elements and render objects only to make its
way next door to a shader exactly like what we wrote. And that shader really exists. And we can look at it. It lives in the Flutter
Engine Repository at impeller/entity/shaders/solid
fill.frag. And here's its code. Looks pretty familiar. Our version produces
the same visual result as a colored box delivered
by the same system, shaders running on the GPU,
only more directly. And while this might not
look like it's much more direct than the
colored box, it truly is because Flutter does a lot
of work to translate our widgets and render objects into
the appropriate shaders. So when we supply
our own shader, Flutter is kind
of on easy street. Now, does this mean you should
use shaders for everything and stop using widgets? Absolutely not. Widgets exist because shaders
are tedious and confusing. So definitely keep
using widgets. But custom shaders also exist
because they have a purpose. Really cool UIs. Obviously,
plain green squares don't fully capture that. So let's look at something
a little more advanced. Let's write a
shader that renders a gradient between whatever
color we supply and white. Now remember the
size information we passed in but never touched? Well, now is its moment
and another opportunity to further explore that 0 to
1 normalization from earlier. Our first shader couldn't
have been simpler. It returned the same color
for every single pixel and it didn't even
calculate that color. Our Dart code
supplied the value. Well, this new shader and
basically every other shader in the universe
actually needs to know which pixel it's operating on. So typically one
of the first things you see in any
shader's main method is this line, which grabs
the current coordinates. Now this returns a vec2
of raw normalized floats. But shader authors, good
at math as they are, like to make things as easy
on themselves as possible. And it turns out
the math gets all screwy if you combine normalized
and unnormalized values. So we need to perform that
same trick on our pixel coordinates that we applied
to the color values. But what would that look like? It's certainly
not divide by 255. Except it sort of is, actually. What was 255 in that scenario? It was the maximum value. We divided our raw value
by the maximum value to collapse it down
into that 0 to 1 range. For the pixels coordinates,
our maximum values are represented by
the total size, which is the part of the screen that
our Flutter code has allocated to this custom paint widget. Luckily we have that
information available in the uSize variable. I knew we passed
that in for a reason. Thus to normalize
our pixel variable so that it's compatible
with our color variables, we need to divide our current
raw coordinates by U size. Now, if our shader was working
on a small space, maybe only 100 by 50 pixels, and the
current coordinates were 10, 20, that would mean our pixel
vec2 would store 0.1 and 0.4. Converting all our numbers
to normalized floats makes it straightforward to use
them together in calculations. Shaders love to normalize
values between 0 and 1. Next, let's save the color
white in a fresh vec4. And notice how I only
supplied one parameter. GLSL interprets this as
meaning I want the same value spread across all
available parameters. This is essentially the same
as writing 0XFFFFFFFF in Dart. The last thing we're
going to do, just like with our first shader, is
assign a value to frag color. We need something
like colors.lerp from the Flutter SDK. And luckily GLSL has
a built in called mix which does the same thing. For that first
variable placeholder, replace A with uColor,
the uniform parameter our shader accepted
from our Dart code. Replace B with white
and C with a normalized float, just like
the third parameter to colors.lerp in Dart. Luckily we just converted
the variable pixel to a vector of just
such normalized floats. So let's start with pixel.x. At this point, our app
should be ready to run again. Hot restart. And would you look at that? A gradient from green to
white across the x-axis. To rotate this 90
degrees and have it spread across the y-axis, we
can change pixel.x to pixel.y. Pretty cool. We've covered a lot
of ground here, But? There's actually one very
sneaky bug lurking in our code as it currently stands. To see this in action, I'll
flip back to my Dart code and change the
color I'm passing. Let's replace colors.green
with its raw hex values, color 0XFF4CAF50. So far this is identical. But now let's cut the
alpha value in half. Obviously, our gradient
should change, right? Well, hot reload your app and
get ready to see no change. Told you we had a bug. We've talked about
how colors in Flutter contain an alpha channel. And until now, it's always
been someone else's problem to apply that. Well, once you start writing
shaders, that someone is you. Flip back to your
GLSL code and we can fix it by creating
a new vec4 to store our alpha applied color value. For the first three
variables, pull out the red, green, and blue
channels from uColor. And then, multiply them
by uColors alpha channel. This code won't run yet,
as we're only supplying three values to this vec4. The convention is to bundle the
original alpha value in in case something later needs to
unapply the alpha channel for some other magic. Notice this unique
syntax where we've extracted a vec3 from
vec4 and then smooshed it back into another
vec4 with the help of an additional
floating point number. Lastly, pass color with alpha
to mix instead of uColor. Now hot reload
your app and voila. Our color's opacity is honored. So that's fragment shaders. The world of advanced
fragment shaders is broad and deep and full
of math I'm not good at. However, if you like
trigonometry and thought this looked fun, I
recommend checking out the book of shaders
linked in the show notes below for a step by step
walkthrough of all you can do. And if you're looking
for more inspiration, check out the Building Next
Gen UIs in Flutter workshop from Google I/O '23. There you'll find two
pretty incredible shaders to sink your teeth into,
and a special widget called the Animated Sampler,
which applies fragment shaders to other parts of your UI. Be sure to leave comments below
on how you're finding the show, including topics you'd like
to see in future episodes. Every video we make
exists for you. So don't be shy about asking. Until next time,
happy fluttering. [MUSIC PLAYING]
great video!