Extremum Seeking Control

welcome back we've been talking about data-driven control and in particular how we can use machine learning and related optimization techniques to either discover a model of a system or to directly discover effective control laws using using data and optimization what I'm going to tell you about right now is an adaptive control strategy called extremum seeking control so extremum seeking extreme seeking control is an equation free control law so this is equation free or adaptive meaning that it doesn't require you to have a model of the system and it can adapt to slowly changing parameters right so if I have a system that ages or it changes over the course of a day then this extreme seeking control is going to be able to compensate for those parameter changes with even without modeling or measuring them and extreme can seeking control is an extremely useful powerful and effective method of optimization for control that I use quite frequently so if there's ever a problem where I don't want to model the system and I have a control knob and there's some kind of overarching objective function or cost function that I want to maximize or minimize then I usually will try extreme seeking control at least in my first round of things that I try ok so extreme seeking control was largely developed by Miroslav Krista at UCSD and so I suggest that you go read some of those foundational papers and I'll give you some links links to those papers and books ok so I'm going to walk you through just the simplest simplest example of an extreme seeking controller making a bunch of kind of assumptions that make it easy to explain but then we'll break down those assumptions later so in general we're going to assume that we have some kind of system the system is going to have inputs u and outputs Y and those outputs are going to feed into cost function or objective function maybe let me call this an objective function because I'm gonna want to maximize the objective not minimize a cost okay so some objective function J and I can I'm basically assuming that if I move you I can get a measurement of J okay I can measure the the measurements that feed into my objective function and then extremely seeking control is going to feed back that measurement and do something to try to optimize optimize you to maximize maximize J okay so the general theory of extreme seeking control applies to dynamical systems systems with changing parameters lots and lots of cases what I'm going to do is I'm going to consider a massively oversimplified case to start with where my cost function or my objective function J let's call this my objective J as a function of U is just a static function it's a static paraboloid okay and so that means essentially that the system dynamics respond instantaneously to a new actuation signal you give me some measurements Y and some corresponding cost function J so I'm assuming the simplest case where there's a single global maximum to my objective function and then my dynamics respond instantaneously to actuation you now that doesn't have to be the case but this is just to illustrate how extremely seeking control works okay so in this diagram here what we notice first of all is that there is an optimal control value U star which corresponds to the peak in my objective function so what I want to do is I want to steer whatever my U is I want to find U star and track that U star so even if this J slowly varies over the day and this peak moves I want to track that optimal u that gives me the maximum and my objective function that's what extremists seeking does so let's do a thought experiment the the basic thought experiment here is that I'm going to have some estimate for what my best controller is and let's call that you hat so I have some estimate for my control law u hat and I'm just gonna draw u hat here so let's say I have I think that it's u equals two okay what I'm going to do or what extreme seeking control does is it adds to that best estimate it adds a sine wave perturbation okay so it adds some a sine omega-t to that and that's what my actual U is so I take my my estimate for the best control the optimizing control Lu hat let's draw a dashed line this is U hat and what I do is I add on to that a sinusoidal perturbation and what I what the sinusoidal perturbation does is it essentially probes what the slope of this cot this objective function is at that particular value of U hat so let's walk through this this thought experiment okay so so I'm at this U hat and if I held that u hat constant my J would be constant just have this value of J but if I swing my control if I swing my let's say my u hat when I swing it positive or negative when I swing u hat in the positive direction my objective goes up okay so I get a bump in my objective function so my objective function goes up when my u hat goes to the right and when my u hat goes to the left when it's more negative my objective function goes down so I measure this decrease in my objective function J given that u hat now this J has some some average value that's not zero because it's way up here but if I if I if I made it so that it was zero mean if I subtract it out the mean and then multiplied these two what I would end up getting is a signal that is merely purely positive okay because if I multiply two sine waves that are relatively in phase I'll get something that is mostly positive plus times plus as plus minus times minus is plus okay and so what this tells me is that if I am to the left of my optimum so that the slope of the cost function is positive giving this little perturbation signal and measuring the output perturbation if I multiply those two I would get a signal that's mostly positive and that mostly positive signal tells me hey I should be moving that way I should be moving to the right okay let's see how this works this works over here let's say now I'm to the right of my objective dot similarly I'm going to add a sine wave I'm going to go up here and now I'm going to do the exact same thing I have my J function and I have let's draw my you down here my you signal so now if my you swings to the right so if my you swings to the right here I measure a decrease in power okay so now these are out of phase and if I swing to the left I measure an increase in objective function I sometimes I say power because I use this to optimize power in systems and so now I get a J and a u that are out of phase okay and if I multiplied these let's do this in orange if I multiply these guys then I would get a signal that is mostly negative plus times minus is minus plus times minus is minus and so what that orange signal tells me is hey I'm in a place that has a negative slope I better go to the left okay so the basic idea here is that I inject some perturbation sinusoid on to my best estimate of my control U hat I measure what the sinusoidal perturbation on the output J is and it can either be in phase when I'm to the left of the optimum or out of phase when I'm to the right of the optimum and if I added this multiplied signal to my you had if I integrated that multiplied signal I would tend to move in the right direction so let's work that out in this control diagram here okay so remember this did not have a zero mean so I had to subtract out the mean that's pretty easy to do by just applying a high-pass filter okay so high-pass filter and in frequency domain in the Laplace transform domain that would look something like s over s plus some high frequency Omega H okay and then remember what I'm doing is I'm taking that high-pass filter so only the high frequency stuff not the DCE or steady-state component I'm high-pass filtering that let's call that high pass filtered signal row I'm taking Rho times my perturbation signal a sine Omega T okay and I'm multiplying the two together so I want to multiply this with my sinusoidal perturbation a sine Omega T and sometimes in some extreme of seeking controllers you phase shift this a little bit to get it to line up better okay and so this is kind of a tuning parameter that you can play with usually I start with this equal to zero and I just use my input sine and I directly multiply it by this high pass filtered output okay and this essentially gives me the orange signal so at this point I multiplied my sinusoid by my high pass filtered output and I have this kind of orange this is called a demodulated signal demodulated and it's generally mostly positive when i'm to the left and mostly negative when I'm to the right of the optimum and so what I actually want to do is I want to add I want to integrate this I want to add that to u hat so basically I'm just going to continually integrate let's write integrate this into my best estimate and an integrator in the frequency domain is just 1 over s the Laplace variable s maybe I multiply this by some gain K okay so this is actually completely the extreme seeking control diagram this is basic we all there is for this simple example you might ask what happens when I actually get to the top of this this objective function when I actually move this system to the right and actually get to U star let's look at what that that output signal looks like okay so if I had now a sinusoidal input about the optimal frequency then the output function J so J or kind of this demodulated row let's say I swing to the right of you well if I swing to swing to the right I go down but only a teeny tiny amount okay so I swing down a little bit and if I swing so that was if I had a positive swing in you now if I have a negative swing in you I also go down and this is kind of interesting so you basically either direction if I jiggle you I go down because I'm at the peak of J and if I multiply this with my my signal you basically I get something which is averages out to be neither positive nor negative so I get something that's a little bit positive here and a little bit negative there and basically this signal when I add it to this best estimate it doesn't move okay all right so I want to walk you through this one more time now that we have the diagram and we have all of these sine waves okay what is extra so you can control two extremes you can control is an optimization technique that jiggles your input signal you it jiggles that input signal and if it measures when it when it swings when you swings positive if J goes up and if you swing positive J's goes down then it adds a little positive number it integrates that little positive number into my best estimate you hat and I step to the right okay now if I was to the right of my optimum and I did the same thing I jiggle my you sinusoidal e now if I swing you positive and I measure our drop in J and if I swing you negative and I measure a bump in J that tells me this demodulated signal tells me that I need to move left okay and this demodulated signal sometimes we call this this is sigh sigh sigh okay so the basic idea is extremely seeking control just jiggles the input you and that probes the slope of this objective function if it's a positive slope you take a step in the positive direction if it's a negative slope you take a step in the negative direction the nice thing about this also is that if I'm in a region that has a higher slope this is a bigger sinusoidal swing and this C signal will be bigger so I will take larger steps when I'm in a region of really really big slope and then when I get to regions of really really minor slope this sign is so I get smaller and my steps get smaller so the farther away I am with a higher slope for my objective the bigger step I'll take the faster all approach and then as I get to the top kind of everything slows down and I just hang out around there okay so this does rely on you having access to your control signal and being able to put in this a little sinusoidal jiggle to probe the system this is really really closely related to another older method called perturb and observe which is basically the simplest thing you would think of to optimize to find this U star and what perturb and observe does is it basically takes a little discrete steps it says okay I'm going to take it discreet step in you did my power did my J go up or down if it went up take another step if it went down take a step in the opposite direction extreme seeking control is essentially a continuous time version of perturb and observe but instead of just taking fixed discrete Delta U steps it can take bigger or smaller steps depending on on the slope or the gradient of this cost function J so it's it's kind of a generalization which is faster tracking because it takes bigger steps when you have higher slope and there's also provable guarantees on when this will and will not converge for dynamical systems so very very useful this is just kind of the simplest illustration of what's happening here for a static objective function with instantaneous dynamics okay now there's a few assumptions I made here and these assumptions are true even when you're applying this to dynamical systems so extremists seeking control if all you were doing was just finding some optimum point in a static objective function this wouldn't be that impressive there's lots and lots of methods to do that but let's say over time this system is actually depends on some parameters so let's say that this is some curve that changes over the course of a day okay so over 24 hours this this curve might move to the left and move to the right slowly and so that you star my maximum my maximizing input U is also going to be moving to the left and to the right so as long as those disturbances or those parameter variations are slow compared to this the sinusoidal jiggle this extreme so you can control all will be able to track that optimizing controller over time and keep your system performing at peak performance that's really useful okay so you assume that the disturbances to your system the slow parameter variations are the slowest timescale the next faster timescale are these sinusoidal perturbations that you're adding to the system so these are faster than the parameters and the disturbances of your system and then the absolute fastest timescale you assume that the system dynamics respond faster than the sinusoidal input perturbation so you basically assume that as I'm jiggling the system left and right the dynamics of the system are very rapidly kind of equilibrating and you can measure some objective function for each point in this sine wave okay that's basically the assumption of extremists you can control now I will point out this is fundamentally a local optimizer okay so if my objective function instead of having just one peak if it had let's say two peaks like this if I started to the right I'm only gonna converge to this first peak because it won't allow me to go down to go to a more global optimum okay so I think it's very important also to recognize that this is a local local optimizer and so if I have a very complex cost landscape with you know global maximum and local Maxima I have to be careful where I initialize this extreme so you can control where I start my best estimate you hat so that I can climb that hill and get to that peak performance okay so this does generalize to dynamical systems we're going to try this out on some examples I'm going to show you lots of real world applications where this works and we're also going to code this up in MATLAB for this simple example and see how everything works okay thank you

Channel: Steve Brunton

Views: 13,926

Keywords: Control theory, Machine learning, Data science, Dynamical systems, Machine learning control, Matlab, Optimization, Control, Model reduction, System identification, Reduced order models, Regression, Feedback control, Applied mathematics, Linear algebra, Balanced model reduction, balanced truncation, singular value decomposition, dimensionality reduction, Extremum seeking, Extremum seeking control

Id: hxY-IWByn-Q

Length: 18min 27sec (1107 seconds)

Published: Mon Jun 11 2018

Reddit Comments

[u/TechRepSir]

Damn. Is he writing backwards? That's some fine control

[u/bug_eyed_earl]

This channel is my new favorite after Brian Douglas.

[u/TheJCBand]

Excellent, excellent video! Hope we keep seeing more and more content like this.