Machine Learning Control: Overview

welcome back everyone okay so we're talking about data-driven modeling and control and in particular now we're gonna start talking about machine learning for control okay so I just want to recall this basic generalized feedback control diagram here you have some dynamical system maybe it's complex nonlinear multi scale high dimensional there are some disturbances to your system maybe a gust of wind or something unmeasurable parameters that change the dynamics there's some high-level objective function that you want to optimize either minimize or maximize and the idea is to design some control law based on sensor measurements of your system that can then drive the system through actuation through your control knobs so this is kind of the generic architecture that we've been working with from day one even back in the control bootcamp and what we've been talking about is how to use data-driven methods machine learning and other types of optimization both to design the dynamical system and the controller so we spend a lot of time talking about how you would use machine learning for characterizing your dynamical system this is essentially system identification but with modern methods in data-driven optimization and our machine learning so you can spend a lot of time and effort characterizing your system from data and then use that system model to design a control law but what we're going to talk about today is instead of painstakingly characterizing your system dynamics just to design a control law instead what we're going to do is use these statistical and machine learning optimizations directly to design effective control laws that you can then wrap around some unknown dynamical system so some of the motivation for this is the following so dynamics are hard to characterize and it turns out that in a lot of cases the model that you need to predict and understand a physical system is a lot more sophisticated than the model you need to control the system okay so we spend a lot of time understanding the dynamics getting a model that's you know really good description of the physical system that might be overkill for a control lock we have robust control can handle a lot of model uncertainty and missing dynamics so spending a lot of effort on dynamical system modeling might not be the best use of time in addition the whole point of this is to measure the system and then drive it to change the dynamics okay so if I have something like a rocket ship I'm thinking of like the Blue Origin or SpaceX rocket that they're gonna land back on the launch pad or on a moving platform at sea then I'm trying to change my natural system dynamics which are unstable and I'm trying to drive them to be stable dynamics so if I spend all this time characterizing my system it might be that right when I apply the control the dynamics change a lot and I observe new phenomenon that I never saw before this is especially true in very complex high dimensional nonlinear systems like you know political systems or financial markets or disease modeling where even if I have a great model of the natural unforced system when I start to apply control if I start to make a regulatory policy or a vaccination campaign I might dramatically change the dynamics and start to see phenomena that I never saw before and my dynamic model might very rapidly break down and start being useless for control so instead of putting all of that energy into the dynamics modeling which you know I actually do like modeling dynamics maybe we should be spending more effort directly trying to design effective control laws using data and so this is largely inspired by the fact that in modern systems of interest it might actually be possible to try lots of control laws to gather lots of data so the example that that we've been thinking about this is with collaborators like Baron Nowak and others are turbulence control problems so that's a very very challenging dynamical system a turbulent fluid high dimensional hard to model very hard to design reduced order models that are suitable for control people have spent decades trying to do this and it's very very challenging but in modern experiments you can try a control law you can try you could write down a controller that you think might work and you can try it in an experiment in a wind tunnel or a water channel in something like seconds or half a minute okay so in that short of an amount of time you might actually know if your controller was or was not effective at optimizing your objective function and so with that kind of a scale you might be able to apply hundreds or thousands of different control laws and use this data-driven optimization this machine learning architecture to go from kind of this broad sweep the shotgun approach of many candidate control laws and start refining them into these very very highly effective control laws through optimization so that's kind of the idea that lots of these systems of interest you can directly optimize your control law if you have a good parameterization of controller space if you have some good set of parameters that define what families of controllers you can try you can run data-driven machine learning optimizations directly on those controllers and then wrap that around your system now this doesn't necessarily work for lots of systems of interests so for example I don't want to do this in disease modeling because there are ethical concerns with trying a thousand candidate control laws knowing that a lot of them are not going to be very effective so that's actually a case where I might want a physical model so that I don't waste energy and time and hurt people trying you know to refine my control law through trial and error through kind of an optimized fancy trial and error another example would be I think like neural control like if you're trying to stop someone's seizures you're probably not going to do a trial and error based based control strategy so a nuclear reactor okay I'm trying to control some fusion reactors some tokamak probably I don't want to do trial and error you know try a thousand bad controllers where this thing might blow up just to find the one the one good one okay but what you might do is you might build a proxy model using kind of machine learning up here and then if you had that proxy model then in simulation you could try this machine learning control optimization approach without actually put in any humans at risk okay I hope that was clear so lots of options here what I would advocate where we're moving in the future in the next you know five 10 20 years is really putting this brain kind of in the middle and doing this hybrid approach where you your dynamics and you learn your control law in concert and that's much more like what humans do where you have some physical model of the world where you can run these simulations so for example if I'm trying to play tennis I have some basic physics simulator in my head that tells me what I think should happen depending on what my control input is and so in the background I can be running this kind of optimization to figure out better and better control laws and then periodically I go play tennis and I actually can refine my strategy with real data and I can refine my model and my control I in in real time okay so that's where things are going right now we're going to zoom in on on how to use machine learning for the control okay that's what we're going to talk about for the next the next few lectures in particular one of the things that I think is really interesting and exciting is the use of evolutionary algorithms and genetic algorithms for control so you have this is kind of a generic diagram like before I had my dynamical system I have my control loss on cost function J and what I'm doing is I'm running this meta learning algorithm this genetic programming algorithm where I try lots of different control laws but then I learn an effective control law after this kind of targeted optimized trial and error procedure over time and so I'm going to tell you a lot about genetic algorithms genetic programming and how you use those for control we're going to cook up a few simple examples in MATLAB for things like PID control I'm going to tell you how this has been used for turbulence control but in general you can also do you can do lots more than this I can have instead of this genetic programming optimization I could build a neural network okay so I could have this kind of neural network model that is that's driving my dynamical system or I could maybe in some cases I have a very complex cost landscape J the example I like to think of is if I'm playing chess how do you know what a good board position is that's kind of your cost function J how do I know it at an instant how good J is that's a very subjective and difficult to measure measure quantity so I might actually use neural networks to learn what my cost landscape is and then I can use something like reinforcement learning or model predictive control once I know something better about my system or my J okay and so that's what people are doing now with deep reinforcement learning and deep model predictive control is essentially they're using you know deep neural networks for the hard to model parts of those control laws which in reinforcement learning it's this kind of objective function this value function landscape and so that's another another thing you can do there's tons of options again I just want to harp on this because it's so important control is optimization that's constrained by the dynamics machine learning is optimization based on data and the really interesting control problems are nonlinear optimizations and fortunately machine learning is giving us better and better non-linear optimization techniques based on the wealth of data we can collect so that's what we're gonna be doing we're going to be talking about how to essentially use machine learning and data-driven optimization directly to find good control laws without needing models of the dynamics okay so that's all coming up next we're going to talk about genetic algorithms genetic programming and some applications okay thank you

Channel: Steve Brunton

Views: 33,663

Keywords: Control theory, Machine learning, Data science, Dynamical systems, Machine learning control, Matlab, Optimization, Control, Model reduction, System identification, Reduced order models, Regression, Feedback control, Applied mathematics, Linear algebra, Balanced model reduction, balanced truncation, singular value decomposition, dimensionality reduction, Genetic algorithms, genetic programming, artificial intelligence

Id: 3yU2k8R9JeU

Length: 10min 5sec (605 seconds)

Published: Sun Jun 10 2018