Deep Reinforcement Learning for Fluid Dynamics and Control

[Music] hi everyone so today i'm really excited to tell you about one of my favorite topics which is how we can use deep reinforcement learning in the field of fluid dynamics in the last couple of videos i've told you about reinforcement learning and reinforcement learning with deep neural networks or deep learning and so today we're going to talk about some of the many applications in the field of fluid mechanics so here i have one of my favorite videos of this eagle flying in an unsteady turbulent gust field and you can almost see the eagle feeling the wind and manipulating the vorticity in this case to gently glide onto a rock okay so this is an extremely hard control problem and it's the kind of thing that we as human engineers might want to be able to reproduce or mimic the capability of in in our systems so uh biological systems are extremely good at interacting with complex fluid environments from fish to birds and bats even flying insects and so how they do this uh has for a long time for thousands of years been a mystery to humans and has also motivated much of our own engineering design and thinking so it's really exciting that this biologically inspired idea of reinforcement learning is now starting to tell us uh kind of how we might interact with and control these complex systems so just a very very brief reminder of kind of the the setup for reinforcement learning so we have our agent in this case our eagle interacting with some environment our unsteady flow field it has a state that it measures it measures its state of the world through its uh through its eyes through its vision also through the sensing on its wings so maybe it can feel a bit of the flow field and it has some policy for how it decides its actions based on what it's sensing about the current state the past states and what it thinks the future states are going to be and it's trying to maximize some reward and this is a pretty abstract reward in the sense that it could be um you know catching a fish or not banging its talons on the rock when it lands or spending as little energy as possible while maintaining um you know kind of a patrol over an area and my wife is a neuroscientist and she pointed out actually that the reward in biological systems is not actually coming from the environment in biological learning systems the reward is actually internal to the organism so we release dopamine when something good happens when we're happy and that is the kind of internal reinforcement signal in biological systems so this is a little different than the biological uh reinforcement learning but this is kind of our playground uh when we want to simulate these things and design them ourselves and so one of the things we're going to do um so of course we have our policy there's some value function where the eagle you know estimates what its future benefit is going to be based on some control action and in deep reinforcement learning we're essentially replacing a lot of what's going on under the hood with the eagle with this deep neural network where we measure the state and we make some informed decision about a good policy that will get us kind of maximal rewards in the future okay so what i'm going to do is i'm going to go through some vignettes of examples of reinforcement learning in the field of fluid dynamics and this is just a small snapshot and i'm sure even when this video comes out there will be new examples to discuss so this is just kind of an instant in time now a lot of what i'm going to talk about we did discuss in our review paper in the annual review of fluid mechanics on machine learning for fluid mechanics and in particular we talked about kind of this deep reinforcement learning scheme and how this can be used in fluid dynamic applications and we tried to separate out those applications into applications where you are directly manipulating the fluid so flow control to reduce drag or increase lift other applications where you're trying to kind of mimic some biological activity like schooling or flocking and then another kind of paradigm of deep reinforcement learning is where you're learning flight dynamic control and so there is a fluid kind of that you're inside of and there's unsteady fluid dynamics for this you know for your glider or your quadrotor but you're kind of abstracting that out and you're trying to make a high level control scheme for a robot okay and so i i'm just going to highlight one of my favorite works on this uh which is this paper by verma novati and kumitsakis in pnas 2018 where essentially what they did was apply deep reinforcement learning on a very very sophisticated three-dimensional simulation of fish in different configurations to see if trailing fish can harness energy from uh from the leading fish can they uh kind of school or or tail fish efficiently can you um kind of build a control lot to do that let's see if i can play this again just so you can see now this highlights some of the challenges in applying reinforcement learning for fluid mechanics so first off this is an extremely high dimensional simulation the state space is you know i think in this case must be on the order of millions possibly even billions uh of discrete elements that you would need to simulate this fluid so very very high dimensional non-linear environment the individual agents probably only have very limited measurements of that environment so they are going to only measure you know a little bit so for example fish have a lateral line which is kind of a pressure sensitive line and that's what they get to measure about the surrounding fluid flow and so these simulations are extremely expensive to run and we know that reinforcement learning requires a lot of kind of trial and error so you need a lot of simulations and so this is extremely extremely expensive currently and there are lots of efforts to speed this up and to make this more efficient using reduced order modeling and things like that but i'm really really impressed by this work and there's a ton of great work coming out of petrus kumitsakis's group this is just one of many papers they were among the first to do this in the field of fluid mechanics good um so another kind of area i like to think about that is related this also comes out of petros's lab is using reinforcement learning for speeding up computational architectures so i told you that that big simulation of those fish was basically a super computer computation um like a real kind of feat of of computational strength so in this paper they actually use reinforcement learning in a clever way to try to speed up those supercomputer computations of turbulent systems and so in this case i don't have time to really dive into kind of the nitty-gritty here going to the hood but i encourage you to read this these blue cells which are pretty sparsely spaced out in the computational domain essentially use multi-agent reinforcement reinforcement learning to learn kind of how the flow interacts on these sparse grid points to fill in kind of the details and the statistics of this much higher resolution grid so the upshot is that you can get away with much much fewer computations on these kind of sparse elements uh but still have the ability to capture some of the high dimensional dynamics at a fraction of the cost so i think that's a very cool idea this fits into the closure modeling turbulence closure modeling field called sub grid scale modeling where essentially these blue elements are sampled at kind of coarser than the finest resolution you would need to resolve all of the turbulence and this multi-agent reinforcement learning is acting as a closure model in that scenario so again this is a lot of kind of deep stuff i'll hopefully talk more about large eddy simulations and subgrid scale modeling later but i encourage you to check this out okay so you can use reinforcement learning to mimic swimming and schooling you can use it to speed up computational architectures in fluids i'm going to point out another excellent review paper um this is one of my favorites and i highly recommend you check this out i've been following the work of of revolt for for a while and they do some really great reinforcement learning for fluids what i want to point out here is that they cover a lot of examples in the literature and they talk about what algorithms they use kind of how they actually enact these reinforcement learning algorithms in practice so for example we just pointed out the verma 2018 and so most of these algorithms are using some variation of q-learning ql is q-learning or a deep-q network a few different ones proximal policy optimization a3c i think is asynchronous advantageous actor critic so there's a few different ones but most of these are q learning okay and i actually learned of some new applications uh just in in reading this paper recently okay good um so a couple of other examples that i want to talk about are how you can use this reinforcement learning to control the fluid itself so instead of just being an agent in the fluid like my fish what if we actually want to do flow control and manipulate the fluid itself so often times in this flow past a cylinder or you know this is mimicking the fluid flow past a truck or an airplane or a ship where you might want to decrease the drag to increase efficiency so in this great paper they essentially do that with reinforcement learning and what they're able to do is get a substantially decreased drag coefficient so this is much much lower drag on those bodies just by applying reinforcement learning for control in a clever way and remember reinforcement learning is essentially the intersection of machine learning and control theory so very very useful for flow control good some other examples where people have used this for for controlling the fluid directly this is a very cool recent paper where they uh numerically and experimentally manipulate the drag on these bluff bodies on these kind of wake flows using active reinforcement learning control and i put the significance statement up here i think it's very nice uh kind of summary of the problem and you know they're able to reduce the drag by 30 percent very quickly in these simulations and experiments so i think i have a figure here good this is a snapshot um before training halfway through training and column c is after training and you can essentially see that the wake behind the cylinder that's the bottom row here is kind of evolving as the system is learning and what you can actually see is that it is learning to kind of trip it to turbulence and create a more uh streamlined wake so you know that's something that people who throw baseballs you know have kind of learned how to do in some ways very cool results uh and then actually in reading that review paper i i hadn't been aware of this this paper that was in siggraph in 2018 but i found this after reading their review paper and it's an absolutely beautiful demonstration of kind of learning different control strategies in a simulated environment to use fluids to manipulate objects i'm just going to let this play for a while because it's pretty awesome and i was you know unaware of this of this work really really like this and this is like a four minute video the link is down here so please you know check it out yourself there's way way more kind of cool stuff that they do here in this video i'm just going to see a little more the rigid body balancing i think is really nice so they're going to kind of balance this thing up and there's you know there's more examples okay good um so the last example that i i want to go into and i've mentioned this very briefly in my deep reinforcement learning video is this application of uh using reinforcement learning for flight in an unsteady fluid environment so we already saw the fish schooling that is you know using reinforcement learning to control an agent a fish in a fluid environment now we're going to look at how people have actually designed controllers for quad rotors helicopters gliders in real uh unsteady fluid environments and this is kind of where the the rubber hits the road you know connecting back to that eagle flying so some of the earliest work uh here was done uh in berkeley and stanford uh where they essentially reproduce these very very aggressive acrobatic maneuvers of these quadrotors so they essentially have a human expert training and flying these things in these crazy acrobatics and then with reinforcement learning they can essentially learn to mimic those acrobatic maneuvers and this is some of the earliest work i mean really ahead of its time 2004 neribs 2007 europe's and so on really great work uh a related work also uh pretty early on was uh by tedrick at all learning to fly like a bird and i really like this one because i think you can see it there's a an airplane here like a model airplane and they're learning to perch in an unstable an unsteady fluid environment you can see the fluid here and you can see this thing learning to stall its wings and perch uh on a ledge this is a very complicated maneuver this is something that we see birds do all the time when they perch uh on a telephone wire but this is extremely hard uh for you know for engineered systems to do so like i wouldn't recommend a human try to do this in an airplane this seems like very dangerous to stall your wings out create a huge separated vortex to simultaneously increase your lift as you decrease your dr you increase your drag and slow down for this nice gentle perch maneuver but with reinforcement learning they were able to learn uh this this very complex maneuver really cool paper another great one this is out of eth 2017 control of a quadrotor with reinforcement learning and there's a lot of groups that are doing really cool stuff here i know jonathan howe's group at mit is doing really nice work with uh with kind of advanced machine learning control for quadrotors um but i think this is just a really nice you know kind of learns these aggressive maneuvers and and disturbance rejection okay another great one that is pretty recent i think this is 2016. it looks like it came out by ready at all is essentially learning to soar in turbulent environments so what they do is they take a model of a bird and they put it in a chaotic rayleigh bernard convective flow so hot uh on the bottom cool on the top this is a model for you know when when it starts being daylight in the planes the ground heats up and it creates these large thermal convection cells and hunting birds or carrion birds will glide and rise on these thermals and then they will essentially patrol over an area looking for small prey and things like that and so panel a is before their reinforcement learning training and panel b is after their reinforcement learning you can see that they've essentially learned to use these thermals for very efficient climb in these unstable unsteady fluid environments really really cool work i mean i'm a huge fan of all of this and it kind of blows me away that in such a short time you know so much has been done okay another great one learning to fly i believe this one is also out of out of mit this is a really neat one where they you know they take off they hover or they glide they land you know it's really pretty sophisticated okay so what we have seen is that reinforcement learning especially deep reinforcement learning can be used to understand and control complex fluid environments and so there's a few perspectives so we can control an agent interacting in a fluid environment like a fish or like a glider we can directly manipulate the fluid environment itself for example in flow control applications and so i think that this is going to be an area of massive massive development in the next decades reinforcement learning certainly has been in robotics it is a very robust biologically inspired way to learn how to interact with a very complex system and so i think this is only going to kind of increase in how useful and how kind of central this is in fluid engineering in the future okay so with that i would like to conclude thank you very much

Channel: Steve Brunton

Views: 18,728

Rating: 4.9414349 out of 5

Keywords:

Id: NZNl-M040NY

Length: 17min 35sec (1055 seconds)

Published: Fri Mar 05 2021