Building custom fragment shaders

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments

great video!

👍︎︎ 1 👤︎︎ u/GetBoolean 📅︎︎ May 26 2023 🗫︎ replies
Captions
[MUSIC PLAYING] SPEAKER: Welcome to another episode of the Flutter Build Show. When we last saw our heroes, that's us by the way, we were locked in mortal combat with a custom render object over the layout of a dynamic chat bubble. Today, we'll explore a very different way to render things in Flutter that barely uses widgets at all. But first it's worth taking a step back to look at the big picture of how Flutter renders anything. For this I think it's best to work backwards. One thing we know for certain is that by the end of this whole process, our screens draw what we asked them to. So let's call that final step pixels on the screen. Now, before that, what color each of those pixels should show is computed by the GPU in small programs called fragment shaders. This is a GPU's whole purpose in life. It's why they were invented and why your computer or phone has one, to super parallelize the execution of tiny shader programs that color your device's screen one pixel at a time. And it's worth reflecting on what this graph means. Later on, higher in this flowchart, we'll see different APIs to render things. But they all funnel down through this ending. This means even though everyone in Flutter seems obsessed with fragment shaders recently, they're not actually new. Every pixel of every Flutter counter app ever run was rendered via a fragment shader built into the Flutter framework or into Skia itself. So shaders are everywhere, whether or not we know about them. OK, back to the graph. Above shaders is a highly abstracted layer called the canvas API. Now, the canvas API allows you to make declarative graphics calls like adding shapes or custom paths and filling them with colors. Render objects across our app integrate with the canvas API in this same way. Internally the canvas API's job is to translate code like this into the appropriate shaders and then shuttle them off to the GPU. In the ancient Roman world, it was said all roads lead to Rome. But in Flutter, all roads lead to the canvas API. One possible road is the most common Flutter story. Widgets that build render objects. This is what we covered in the previous episode on building custom render objects. Every widget that draws anything to your screen does so by eventually adding a render object to the render tree. But you can also write your own ultra thin widgets that immediately delegate to a specialized render object of your design. But a custom render object isn't the only way to get cute. The other way to render things in Flutter is to use a custom painter and draw whatever you want directly. Technically the custom painter also requires a widget called CustomPaint, but it's so fundamentally different from normal widgets that it deserves its own branch in our chart. Now, if you've used a custom painter before, then maybe you've seen its paint method, whose first parameter is a canvas object. This is your handle to the canvas API. Well, you can do lots of neat stuff in that paint method. And one of them is to say, hey, look, I got this shader over here that's prepared to tell you what color to use for any given pixel in this space. Please ask it any questions. And Flutter will, in fact, have a lot of questions, like what color should the pixel at 0, 0 be and 0, 1 and 1, 0 and so on? And it'll ask the shader over and over and over again hundreds of times in parallel, because that's a GPU's purpose in life. Be super good at parallelizing math. OK, so that's a primer. This episode of the build show is dedicated to that path on the right where you use a custom paint widget, a custom painter, and then specifically tell your custom painter to use a shader. Let's get into it. First things first, shaders are not written in Dart. They're written in another language called GLSL, standing for Open GL Shading Language. Now, the anatomy of a shader isn't so bad. At the top, you choose an API version by selecting your desired SemVer major and minor release without periods. This declaration would accept any 4.60 release. Then, include this file to get a few Flutter specific helpers. Next, you declare any parameters your shader should accept from the outside world with the uniform keyword. The syntax is uniform then the type, then the variable name. By convention, such variable names start with the letter U. And the word uniform was chosen to indicate that these variables will have the same value across every pixel this shader computes for this frame. It looks like this code would accept two parameters, a size and a color. But GLSL doesn't have complex types like that, so it's also fair to think of this shader as accepting six floats, which it will bundle into two vectors of length two and four. GLSL makes heavy use of plain old vectors of lengths two, three, or four. Next, declare what variable your shader will return. This should always be a vec4, which is a color with full RGBA values. Your only choice here is the variable name. At last we're ready to write the actual guts of our shader. Start with a main method whose job it is to assign a value to the variable we indicated up top. In this case, frag color. Let's aim for the Hello World of shaders for our first go and assign the same color to every single pixel. Now, which color should we use? Well, how about the color we passed in? RGB and A are helpful getters on the vec4 class since it's so common to use them for colors. Now I wrote the line that way the first time to show the red green, blue, and alpha getters on the vec4 class, but we just unwrapped a vec4 only to assign it right back to a vec4. That means we could have also written this. So that's the simplest shader possible. Now let's return to our Flutter code and plug this in. I'm in a fresh Flutter project created by Flutter create --empty with my shader sitting at assets/shaders/my_shader.frag. To start I'll include that shader in my pubspec.yaml file. Next, I'll replace everything in my material app with a custom paint widget, which in turn requires a subclass of custom painter. Thinking back to our shader, we said we were going to pass it a color. So I'll declare one in this custom painter's constructor. Now scrolling back up to our widgets, let's keep our word and supply a value. How about colors.green this time to mix it up? Everybody uses colors.blue. I'll select create two missing overrides in the quick fix menu and we're ready to go. The first method we should consider is should repaint. To understand this method, note that a new instance of our custom painter is created every time this wing of the widget tree updates. That means custom painters are cheap objects similar to widgets themselves that receive initial values and then never change. The job of should repaint is to compare ourselves, the newest version being inserted into the widget tree, with the previous version and work out if we'd even do anything differently than our predecessor. In our case, that's a matter of whether or not our color is different from old delegate.color. If they're different, we should repaint. As is, the editor isn't too happy with us. But this is just a wrinkle of Dart's type system. We can safely change covariant custom painter to my painter, which allows Dart to trust us that it will find the color property on old delegate. Next, we get to implement paint. To start, let's set up the most basic parts of the structure. This method will end by calling canvas.drawrect, defining a rectangle that fills all available space and supplying a paint object. So far this paint object does nothing, but that is where our shader will come into play. We can assign our shader to the paint object by using the cascade operator and setting it directly. The only issue is the shader obviously doesn't exist yet. Scroll up to our painter's constructor and add a new parameter, shader of type fragment shader, and a corresponding class field. Note, for this to work, you'll have to import Dart UI. Now main app is complaining because we've invalidated its build method. Scroll to the very top of the file and declare a global variable fragment program. I know, global variables aren't the best. But the Flutter shaders package has a nice helper widget called Shader Builder that abstracts this for us, and we should all be using that. But I want to first show everyone what that Shader Builder widget abstracts. So in your main method, use the fragment programs from asset static method to grab the shader we made available in our pubspec.yaml file. That returns a future. So I'll make the main method asynchronous, slap the awake keyword on the front, and we're good to continue. We're ready to scroll back down to my shader and supply fragmentprogram.fragmentshader as the value to the required parameter shader. The fragment program is the pointer to the asset. But what our widget needs is the compiled shader itself returned from the fragment shader method. Again, in my real apps, I let the Flutter shader package's Shader Builder widget handle all of this. Now the limiter is happy, but this time it's actually a false positive coming from the fact that the Dart analysis server can't see into fragment shaders and understand what parameters they expect. If we ran our app now, the shader would immediately crash. Sadly, the next part of our code won't be type safe. I'm switching back to our fragment shader to remind myself what parameters I even said I'd pass. First, the shader expects two floats for its total surface area. And then, four more floats to represent the color it should paint. OK, back to our Dart code. The way to plumb parameters through to a shader is to use methods on the fragment shader class called set and then the type of the variable we're passing. In this case, a float. First pass size.width then size.height. Another oddity is that leading integer, which orders our variables for consumption on the shaders end. It's a little unusual, I admit. Next send our color through by applying each of its values one at a time. Only this isn't quite right because of an interesting quirk in how shaders think about some floats. If you've done much with colors before, you may know them as defined by three numbers, one for red, green, and blue, each ranging from 0 to 255. Well, GLSL and the land of shaders has a bone to pick with that system. In GLSL, floats for some data types, including colors, work on normalized values between 0 and 1. Here's a quick primer for how to think about all of this. Consider this arbitrary color written as you'd commonly see it in Dart code. The 0x in the front doesn't influence the number itself. It merely tells Dart that what comes next is not represented by standard decimal notation but instead by hexadecimal notation. Then we encounter four channels, each occupying two hexadecimal characters. Compare that with how GLSL represents colors. In GLSL, colors are represented by vectors, essentially fixed length arrays of decimal floating point numbers. Not only is the order slightly different, with the alpha channel, that's the opacity by the way, moved to the end, but the decimal floating point values are normalized between 0 and 1. So with 0, dark colors you're used to would stay a 0, the middle value 128 in decimal or 80 in hexadecimal would be close to 0.5, and the maximum value, 255 in decimal or FF in hexadecimal would be 1.0. So that's the target format we're looking to support. But how do we do this in Dart? The conversion is achieved by dividing each number by 255. There. Now we're correctly passing our color value to our shader. Let's run our app. And look at that. A big green square. Now this might seem like going around the block to get next door compared to writing colored box, color, colors.green. But it's actually the opposite. If you draw a green rectangle using a colored box, it goes around the block by using elements and render objects only to make its way next door to a shader exactly like what we wrote. And that shader really exists. And we can look at it. It lives in the Flutter Engine Repository at impeller/entity/shaders/solid fill.frag. And here's its code. Looks pretty familiar. Our version produces the same visual result as a colored box delivered by the same system, shaders running on the GPU, only more directly. And while this might not look like it's much more direct than the colored box, it truly is because Flutter does a lot of work to translate our widgets and render objects into the appropriate shaders. So when we supply our own shader, Flutter is kind of on easy street. Now, does this mean you should use shaders for everything and stop using widgets? Absolutely not. Widgets exist because shaders are tedious and confusing. So definitely keep using widgets. But custom shaders also exist because they have a purpose. Really cool UIs. Obviously, plain green squares don't fully capture that. So let's look at something a little more advanced. Let's write a shader that renders a gradient between whatever color we supply and white. Now remember the size information we passed in but never touched? Well, now is its moment and another opportunity to further explore that 0 to 1 normalization from earlier. Our first shader couldn't have been simpler. It returned the same color for every single pixel and it didn't even calculate that color. Our Dart code supplied the value. Well, this new shader and basically every other shader in the universe actually needs to know which pixel it's operating on. So typically one of the first things you see in any shader's main method is this line, which grabs the current coordinates. Now this returns a vec2 of raw normalized floats. But shader authors, good at math as they are, like to make things as easy on themselves as possible. And it turns out the math gets all screwy if you combine normalized and unnormalized values. So we need to perform that same trick on our pixel coordinates that we applied to the color values. But what would that look like? It's certainly not divide by 255. Except it sort of is, actually. What was 255 in that scenario? It was the maximum value. We divided our raw value by the maximum value to collapse it down into that 0 to 1 range. For the pixels coordinates, our maximum values are represented by the total size, which is the part of the screen that our Flutter code has allocated to this custom paint widget. Luckily we have that information available in the uSize variable. I knew we passed that in for a reason. Thus to normalize our pixel variable so that it's compatible with our color variables, we need to divide our current raw coordinates by U size. Now, if our shader was working on a small space, maybe only 100 by 50 pixels, and the current coordinates were 10, 20, that would mean our pixel vec2 would store 0.1 and 0.4. Converting all our numbers to normalized floats makes it straightforward to use them together in calculations. Shaders love to normalize values between 0 and 1. Next, let's save the color white in a fresh vec4. And notice how I only supplied one parameter. GLSL interprets this as meaning I want the same value spread across all available parameters. This is essentially the same as writing 0XFFFFFFFF in Dart. The last thing we're going to do, just like with our first shader, is assign a value to frag color. We need something like colors.lerp from the Flutter SDK. And luckily GLSL has a built in called mix which does the same thing. For that first variable placeholder, replace A with uColor, the uniform parameter our shader accepted from our Dart code. Replace B with white and C with a normalized float, just like the third parameter to colors.lerp in Dart. Luckily we just converted the variable pixel to a vector of just such normalized floats. So let's start with pixel.x. At this point, our app should be ready to run again. Hot restart. And would you look at that? A gradient from green to white across the x-axis. To rotate this 90 degrees and have it spread across the y-axis, we can change pixel.x to pixel.y. Pretty cool. We've covered a lot of ground here, But? There's actually one very sneaky bug lurking in our code as it currently stands. To see this in action, I'll flip back to my Dart code and change the color I'm passing. Let's replace colors.green with its raw hex values, color 0XFF4CAF50. So far this is identical. But now let's cut the alpha value in half. Obviously, our gradient should change, right? Well, hot reload your app and get ready to see no change. Told you we had a bug. We've talked about how colors in Flutter contain an alpha channel. And until now, it's always been someone else's problem to apply that. Well, once you start writing shaders, that someone is you. Flip back to your GLSL code and we can fix it by creating a new vec4 to store our alpha applied color value. For the first three variables, pull out the red, green, and blue channels from uColor. And then, multiply them by uColors alpha channel. This code won't run yet, as we're only supplying three values to this vec4. The convention is to bundle the original alpha value in in case something later needs to unapply the alpha channel for some other magic. Notice this unique syntax where we've extracted a vec3 from vec4 and then smooshed it back into another vec4 with the help of an additional floating point number. Lastly, pass color with alpha to mix instead of uColor. Now hot reload your app and voila. Our color's opacity is honored. So that's fragment shaders. The world of advanced fragment shaders is broad and deep and full of math I'm not good at. However, if you like trigonometry and thought this looked fun, I recommend checking out the book of shaders linked in the show notes below for a step by step walkthrough of all you can do. And if you're looking for more inspiration, check out the Building Next Gen UIs in Flutter workshop from Google I/O '23. There you'll find two pretty incredible shaders to sink your teeth into, and a special widget called the Animated Sampler, which applies fragment shaders to other parts of your UI. Be sure to leave comments below on how you're finding the show, including topics you'd like to see in future episodes. Every video we make exists for you. So don't be shy about asking. Until next time, happy fluttering. [MUSIC PLAYING]
Info
Channel: Flutter
Views: 20,810
Rating: undefined out of 5
Keywords: Flutter build show, introducing the flutter build show, what is the flutter build show, intro flutter build show, flutter build show intro, unlock the full potential of your apps, full potential of your apps, flutter developer, flutter developers, flutter latest, latest from flutter, flutter updates, updates from flutter, developer, developers, google, flutter, code, coding, Craig Labenz
Id: OpcPZdfJbq8
Channel Id: undefined
Length: 20min 36sec (1236 seconds)
Published: Thu May 25 2023
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.