#notGDC 2023 - FFT Ocean Flipbook : How to create & sample one using Blender & UE

hey what's up not GDC if you follow my channel you probably know that I've worked on Plenty of projects involving water in some way right Dynamic waves girls now waves flow maps and all kind of other experiments and I'm not even done their school staff coming up soon probably next month so stay tuned anyway today I'm going to explain a technique I used to create the warrior in my tropical drama project now what I was doing back then wasn't technically perfect and there's a few things I explained in that Back Down video that everyone quote unquote correct and that's all the way it was about time I made an actual tutorial there's typically two main ways to render that kind of ocean in a real-time application there's girls now waves and I've made a long video on the subject already so feel free to take a look at it in a nutshell kelsner ways are awesome but they come with some constraints namely you hit some kind of performance barrier as soon as you want to do any kind of realistic ocean simulation because it's just too expensive to compute tons and tons of girls and waves which you kinda need to do to have a realistic water Behavior so the second way and the actual state of the art despite being quite old already is to use fft which stands for fast Fourier transform it's a huge mathematical tool used across all kind of scientific domains and I highly recommend you watching veritasium's video on the subject if you want to know more it's fantastic but then what isn't when it comes to Direct's work anyway I'm going off topic implementing an fft water simulation is not easy there's plenty of papers on the subject so if you really want to dive in and get your hands dirty it's doable alright I've yet to do that myself but one day in the meantime I'm going to show you a technique to kind that sheet it's certainly not perfect right it has its limitation and all but still it might be a helpful trick to know so I'm going to show you how to use blender and utilize its own modifier which actually is an nft ocean simulation to bake a looping tileable fft version simulation into a flip book that you can then use in engine to cheaply trade quite a decent looking water it isn't as straightforward as it sounds though creating and sampling a tailing flipbook comes with some complications but we'll get to that soon enough alright let's get started then foreign first create a plane make sure it's a zero zero zero and add a notion modifier to it I'm going to increase on the resolution reduce the spatial size to 40 and the wind velocity to 4 meter per second then I'm going to create a camera press alt air to reset its rotation some tighten that water plane move it up a little bit like so and change its type to orthographic then I need to change its orthographic scale to 40 to match that ocean modifier special size setting then press 0 on the numpad and I have a top view of that water surface only thing to fix still is a rundown resolution 512 by 512 will do the trick cool knows that ocean modifier creates a tiling water surface which is awesome meaning the displacement on the left here perfectly matches the displacement on the right however the strong space is creeping in within the render origin to fix this I'm going to set repeat X and Y to 2 and then move the camera in X and Y by 20 meters to be dead center along those four water plates if you squint your eyes and pay attention to the edges you should be able to spot the same exact shapes going in and out of the rondier region right so it should perfectly tile to actually test this though I'm going to switch to the shading tab create material on either texture coordinate node drag the normal pin and plug that in the emission make sure that the base color is black speak to our zero and also make sure that the wall color is black and strength is zero press in the viewport and choose rendered okay that looks wrong there's two things happening here third that normal is actually a unit Vector in World space so it could be positive or negative X Y and Z negative values aren't necessarily an issue if you work with 16 or 32 bits textures but normal maps are usually stored in 8 bit textures so I'm going to remap that wall Vector with a multiply add node multiplier is going to be all five minus o5 to flip the Y Channel That's because blender on Unreal Engine use different coordinate systems don't worry about it and o5 add is going to be o5 as well Ah that's much better however if you're familiar with normal Maps you might have picked up on the washed out colors this doesn't look great that's due to the srgb color management so I need to turn that off here from there again then press n Go to The View tab turn on repeat image and it does look like you have a perfectly tiling normal map sweet now a quick word about that ocean modifier it does have options to break that Warrior simulation like displacement and normals and foam but I'm not going to use it honestly I failed to produce a big that I could work with the normality picks for instance looks super weird I'm really not sure what color space that's in and it sells those wave normals before this placement is applied and I really don't want that so displacement map it bakes is also non-normalized as far as I remember so it needs to be stored in at least 16 bits textures anyway it may sound confusing but don't worry about it we are going to do things Our Own Way moving on this no needs to be animated so let's go to the animation tab I'm going going to go to frame 1 set time to 0 press I to add a keyframe and do the same with scale then I'm going to go to frame 65 so the number of frames I want in my flipbook 64 plus 1 and set time to 2 scale to 1 and press I to keyframe once again I'm then going to duplicate that ocean modifier so press shift d while the mouse is over it change its geometry setting to displace then back at frame 0 it's going to pick up whereas at first ocean modifier stopped at so time was 2 on scale was 1 and then at frame 65 time is going to be 4 and scale 0. by the way feel free to tweak that time value to have a slower or faster animation right just make sure that if this modifier ends up at whatever time value say 3 this one needs to start at 3 and end at 6 okay there's one last step I'm going to press a to select all keyframes press V and change to Vector to have linear interpolation then change the timelines and setting to 64 press play and voila it perfectly loops okay so I'm going to run down that animation now if you follow along that tutorial pause now because I'm going to proceed forward but you shouldn't yet I actually need to do a critical tweak to this setup but I feel like it wouldn't make sense to explain why you right now so just hang on it will make sense in a minute alright so PNG RGB 8 Bits is fine select a folder on render the animation once that's done I still need to pack those 64 frames into a single free book texture right and that can be achieved in many ways slate is an option it's free and easy to use image magic with a simple command line is also an option or any decent 2D software really game Photoshop whatever open these images and create the flipbook texture manually or using scripts there are plenty of options here as long as the result is from left to right top to bottom you're good to go note that this flipbook is going to be quite a big texture all right baking 64 frames with a 512 resolution is going to result in a 4K flip book so that's definitely one of the downside of this technique it's quite memory heavy press the 512 texture resolution isn't going to look super sharp up close so that's something to keep in mind anyway time to test that normal map flipbook in engine I guess so let's switch to Unreal Engine foreign [Music] create a simple material gray color full roughness great apply it to that mesh then import that flipbook normal map make sure it's actually imported as a normal map good okay so playing a flipbook texture in engine is all about creating a counter to move your Visa around and Sample a given tile within the free book how can that be done let's start with time I'm going to use a debug scalar values material function plugged into the emissive color to see what's going on that the time the engine have been currently running so not super useful on its own a first multiply will let me control the flip books frame rate for now I'll keep it at 1 so the debug value doesn't go crazy and I can try and explain what's going on then use floor to round it to the smallest integer and divide it by the amount of frames the flipbook has in X so 8 I also need to use Frac to get rid of that integer part that way every second or whatever frame rate is set here this increases by an eighth which is exactly what I'm looking for right textual coordinates are always in a zero to one range so assuming the flipbook has 8 frames in X the first frame is at 0 in X the second at o125 the third at all 25 and so on okay so I essentially built a function to shift the coordinate at which that flipbook is sampled and jump from frame to frame at least in X funnel the Y chord in at then needs to shift by one frame vertically only once an entire line has been sampled right and that's once every how many frames there are per line eight so divide by 8 then flow again and divide by the number of frames in y Let's Pretend your flip book is eight by four frames or no to see what's going on I'm going to switch to a debug Floor 2 metal function make a vector 2 and that let me clean the graph and use a single frag here sweet so the third value shifts the texture coordinates in X by an eighth every second and the second value shifts the texture coordinates in y by a fourth every 8 seconds which is to be expected from 8x4 flipbook now that's understood hopefully I'm going to increase the frame rate to say 15 for now okay that's not yet actual texture coconuts in fact if I was to plug that in a texture sampler it wouldn't output anything really and it would flicker somehow because that's not a range of coordinates that's just good in it that form I'll jump around and point to a pixel or in between pixels somewhere in that texture what I need to do is to make use of an actual range of coordinates and use this to shift in that range of coordinates so you visual wall position are both an option but for now I'm going to use UVS because it will more easily let me illustrate a particular issue wait what issue well let's see the flip book does seem to play but obviously all frames are visible and it seems to scroll somehow that's actually to be expected because that plant mesh has the following UVS and it covers a whole 0 to 1 range both in X and Y so it's like the entire flip book is sampled and the entire flipbook is shifted from frame to frame so what I'm looking for is to sample a single frame like UVS are zoomed in on just one frame and dividing UVS by the amount of frames in X and Y will precisely do that moving on let's try and make that texture tile say four times because remember we were under the tiling water surface right so multiply those coordinates by say four well what's going on well the multiply here is obviously undoing the divide multiply anything by say 2 and then divide by two does nothing in the end to go around this I'm going to use fraction and make sure this loop on and on and remain in a 0 to 1 range which will still tile the texture but without working against that divide here but now there are visible seams we were sure that the water surface was styling what's going on well in order to render smooth textures regardless of the view distance xinjin applies a linear interopolation and you can actually see it in effect here if we zoom in enough right there's color from neighboring frames bleeding inside that frame and that obviously prevents it from tiling properly because this Edge now no longer perfectly matches the opposite Edge but it gets even worse when mid maps come into play if you don't know what that is well a game engine creates a chain of dawn scale version of every single texture so that based on distance it samples the right amount of pixels there's no need to sample a 4K texture if it's teeny tiny on screen right it would result in a noisy mess and that process is done offline so it's pre-computed and the result is stored in what we call Mid maps and so when done scaling a texture Say by a factor of 2 every grid of 2 by 2 pixels are going to be somewhat average to create one single Pixel and that kind of leads to even more color from neighboring frames bleeding in due to that linear interpolation it's a bit hard to explain but hopefully you get the ID however it's only one part of the issue because one Smit maps are built then Jin still has to figure out which mid map to sample and to do that the engine basically computes the rate of change in texture coordinates so say these are typical coordinates it's a smooth gradient great the engine can deal with that and say okay there's that amount of change in that gradient on the screen so I can deduce that it's best if I use this map now because fraction must be used here to make the texture coordinate style while zoomed in on a given free book frame the other sudden jump in that gradient and the engine have no longer any idea what to do with that which results in visual artifacts okay fortunately fixing that second issue is quite trivial first change the texture sampler setting to override house engine compute the MIP level and do that using custom derivatives instead then use the same gradient but with all the Troublesome jump in value so essentially the exact same thing but without the fraction and use DDX and dty to compare the value of that gradient at a given pixel and add the neighboring pixel in X and Y and derive a rate of change in string space and give the engine a way to know which mid map to sample note that keeping the same rate of change to compute the derivatives and to samples texture is critical to ensure the correct Meep level is sampled for instance say yes keeps that divide here those derivatives and output a rate of change that is 8 times greater compared to those UVS and thus the engine thinks it needs to sample a much higher mid map and it looks super blurry it's a bit technical but anyway that get rid of those early artifacts sweet but it's still not perfect because the Troublesome linear interpolation is still at Play all right then back in blender the goal here is to do what we call over sampling by the way shout out to Ryan Brooks for creating an excellent article on this subject I highly suggest you to take a look at it link will be in the video description below okay so over sampling it essentially means rendering a bit of what's outside this London region that way the actual tiling texture no longer takes up the whole frame but lies within the frame itself with some kind of inner safety margin so the linear interpolation that may happen on those edges May blend with pixels still belonging to that same frame and not neighboring frames now the amount of other sampling to add meaning the amount of extra pixels to include has to be carefully sort of and is directly linked to how many MIP levels you want to fix again I'll let you pick at Ryan Brock's article to have more details just know that if you include one extra pixel on each side the first MIP level is fixed and Sims will no longer be visible up to a certain distance even Century at an even greater distance Sinjin will sample the second Meep level and Sims will start to be visible again and fix the second Meep level you include two extra pixels on each side third level 4 pixels and 8 16 and so on so it's like you double the margin each time on thus you lose quite a bit of your actual texture resolution but then it becomes pretty much impossible to spot such subtle tiling seams past the distance so you can compromise usually fixing up to the cell map level is plenty enough okay so let's do that we Prevail 3 need 4 extra pixels on each side so 8 in total but how much do I need to zoom out to actually include 8 extra pixels that orthographic scale is certainly not in pixels right well my render resolution is 512 by 512 pixels on that render region is 40 by 40 units so a single Pixel corresponds to 40 divided by 512 so so to fix the third MIP level I need to zoom out by an extra 40 divided by 512 times 4 times 2 and that's why I said earlier to pause the tutorial if you were following along if you like that critical trick needed some context to be explained anyway now you can render that animation and create your final free book back in engine I now need to take that over something that extra margin within each frame into account so first I'm going to shift UVS by the amount of extra pixels added 4 divided by the texture resolution 512 that's four text cells this results in new V's quote-unquote starting where the tiling texture top left corner node lies within that flipbook frame but obviously now that it's been offset it kinda overflows on another frame so I need to scale UVS down by double that margin so it ends up where the tiling texture bottom right corner is and that little multiply setup will do just that sweet no it's perfect we are done with styling artifacts yay well almost perfect there's one last issue the flipbook animation isn't that smooth and that's obviously due to the frame rate 15 isn't much I may increase it to 30 and that's better but what if my game runs at 60 FPS and I want a better smooth free book animation well I may increase the frame rate to 60 but that would obviously increase the animation speed and I may want to actually keep some kind of artistic control over it here it's certainly playing too fast now because that flipbook has only 64 frames to play right that's when linear interpolation comes into play now I'm going to switch to a debug free book one that has each frame index written in plain text to better explain what I'm about to do so I'm going to duplicate that entire setup to sample that free book a second time but one frame ahead using a simple add one okay then I'm going to lap between the two textures but based on what Alpha value remember I'm rounding time to the lowest integer using flow so that if the frame rate is say 1 this increments by one integer every second and using fraction on that time will output the time in between that second meaning if I plug that in the alpha value the current frame blends with the next frame to come during that second and then this increments so the next frame becomes the current frame and it starts to blend again this results in a linearly interpolated flipbook which looks way smoother frame rate can be quite drastically lowered and you keep some artistic control over it also an even better solution would be to use a flow based interpolation but that's an entire other topic maybe in another video anyways the downside is obviously the slight increase in performance cost there are two texture Samplers just to play one flipbook and Text Your Samplers aren't free okay sweet we are done with the technical side of things we have flip book that tiles with no visible seams and that loops on and on yay let's get to actually trade the water material no so first of all I only have a normal map flip book to work with and I'd like at least a displacement map flipbook back in blender using a simple carefully scaled bunding box to generate object-based coordinates I can use the Z component to create a normalized linear gradient and that's their height map as simple as that I may also use pointiness to generate some kind of Crest map now that's a bit finicky to work with here it seems to not cooperate very well and adding a zip divide modifier usually does the trick then make sure to remap the value switch to cycles and you get that kind of texture which may have its purpose back in engine imposes two textures and make sure they are set to grayscale linear now I'd suggest you to collapse all that logic into a material function to ease this process but this video is already long enough so for this dirty water material example I'm just going to duplicate that setup to sample the hide my playbook and offset vertices in Z like so however the material doesn't compile anymore that's because using derivatives is only relevant and thus available in pixel shaders and wall position of that logic is executed in vertex Traders so I need to revert a normal texture sampler when that won't actually cause any turning issue here don't worry then plug the normal map into the actual normal and not emissive apply that material on a subdivided plane and here is a working example of an fft ocean that is quite cheap to render now again using high-res flip books come with a non-negligible memory cost and in the end it's just a tiring texture and the wave pattern will probably be noticeable at a distance nothing is perfect okay so next step would be to actually make use of the single layer water shading model and add the required output node and parameters like so then I built a metal function just like I suggested earlier to let me duplicate that flipbook sampling logic way more easily and keep the material clean so here I sample the web hide free book to displace vertices here I sample the wave normals and I use the knit trick which is to sample the three book once as a base normal and then use the sample normal as a direction to distort the coordinates at which the free book is sampled a second time at a much smaller scale to add secondary detail waves and have them displaced by the main waves then here I sample the wave height flip book again and with Crest field book and combine them both with some controls that follow from strength to drive the water opacity roughness and specular and honestly without doing much work thanks to that great water shading model you quite easily end up with that kind of result which looks sweet great one last final thing you might have picked up on that subtle particle effect quite cheaply done because it's really just to illustrate what's possible so it's worth knowing that most logic you can do in a material you can do in a Niagara GPU emitter and so if you can build a logic using math to sample a flipbook frame at a given pixel position you can do the exact same thing to sample that exact same flipbook at a given particle position build the flipbook coordinates build the flipbook frame counter offset coordinates with that counter and Sample the flipbook done and so I do that here once upon particle spawn to sample the wave height flip book then here I do a quick test and kill any spawn Pascal is a wave at that position isn't high enough I also some pressure the wave normal free book to add some initial velocity to the particles based on the wave slope and then use sampled wave height to vertically project particles onto the water surface and then particles are blown away with a wind Force voila is at the end of this long and quite technical video I hope it made sense and that you learn a single two along the way files are available as a chair one reward on my Patron so feel free to join in if you want to support me on my work and have access to all kind of cool educative projects feel free to also leave a like And subscribe to the channel if you like the content thanks for watching thanks for your support I'll see you in the next video take care of yourself bye bye

Channel: Ghislain Girardot

Views: 13,326

Keywords:

Id: rV6TJ7YDJY8

Length: 24min 6sec (1446 seconds)

Published: Mon Mar 20 2023