Running a Power Plant with Grafana

all right I'm Ryan McKinley I work at a hydroelectric turbine company but I'm actually a software developer my history is more working with pretty small early-stage startups and I'm working here as something new but my I'm really committed to the idea of working in open-source instead of doing things open-source by default when there's no reason for it to be closed source let's just push it out there people may use it they may not so that kind of explains a bit of why some of this stuff takes the path it does but to give you just an idea of what the company does is we build a turbine to generate electricity it's a different model for a turbine than you're used to seeing so typically you've got a wheel that spins this tries to convert that into being able to capture more flow out of lower head sites with the result of being able to have kind of equivalent amounts of power production with less environmental consequences so you know with five distributed sites could you capture a similar amount of power out of an environment without you know destroying it too much so so I started working there we're we're gonna plant this is in Oregon and show up here's this thing we've got a UI for it I don't know if you guys have seen industrial control you eyes they start to look like this now you want to see some new things we add it add another oh wait needs some buttons so let's add some buttons to it go oh here's some more you got to have our state machine oh man things kind of a disaster so I go - sure I can figure out what it is we're we're in this thing called rslogix which is it's it's a mess I don't know so you have to open up some some Visual Basic take Visual Basic write it out - in flux send it - in flux then you graph it in Griffin and like everyone's really impressed so now you've got some much much easier to deal with graphs so this is something showing you how servos are moving to keep things on target so now everyone the whole team's happy they're like wow that's amazing but then they want to start seeing some other things so here what I'm showing is we have a bunch of security cameras showing the operation of the plant so it's like let's graph those along with what our hydraulic systems are doing well that's going so I built that this is my very first introduction to graphing I was like how can I just have a simple thing so we made a very simple Ajax panel that was my introduction and published that so our team loves it they get really excited about this because now they can I think previously we had this very expensive platform from allen-bradley that would write to a sequel database and people had to run a script and download that run some processing to be able to see what was happening and we replaced that with very simple things next thing wants to see is NaN time series data so for us we care a lot about what's what's the efficiency of our machine at different stages so non time series so this is something where you put it in with normal sort of the standard influx queries and select I want to see markers say which fields you want on the x-axis versus the y-axis and again this was published these are sort of published to the degree of they work for me and then you know a few other people are using them but my goal has always been hey Griffin has built some great tools for us how can you know there's no reason for me to keep this there our business is not building a graph so let's get it out there so now we expose this to our team and they start going crazy so we get these graphs I don't know if you see this last one I couldn't even paste this in because it said it was too many pixels which there's this brief moment where Griffin are like only lazy loaded the panels and things are great things are things were going really well I think that one was like it's like a thousand parallel influx queries in that so taken a bunch of training alright so we did that first plant and now I've been there for some time we're starting a new plant this one's in Oregon orden where this one is in freedom Falls and in Maine so way up in the north is really cold it's kind of it's not off-grid but it's you know internet connections pretty poor it's a different thing but we have an opportunity to do some new design so with that we took a show you two things so there's that project and then meanwhile because this is a turbine under development there's a lot of things we're learning about it this is actually our office in Alameda so UC San Francisco in the background there so this is our office it's one of these hangars that was the old Naval Air Station there so we're building this a test facility that pumps water through through this thing and runs it through that like turbine area up there so these are two projects that are happening now and along with this here's a picture of the actual new turbine development and it close up so long with this we get a chance to update kind of the whole control system and platform and with the experience we had had with Griffin our early on I pushed a lot to to really use Griffin as our UI so we essentially switched everything out and we get this opportunity kind of I like this picture if this was kind of our old system and this is kind of our new veneer or so this is the so some of the key things we needed was how do we deal with this remote monitoring so if you've got a plant that's in a Maine with very limited internet access we need our engineers to understand what's happening it with it but we also need to be able to sort of you know how do you start it how do you stop it set power levels set parbat power targets other key thing is our needs keep changing all the time so the dashboard dashboard as a platform was super useful because you know whatever needs you have are really easy to just switch them around the key issues we found with just our off-the-shelf gravano and influx was zooming around you would build these dashboards that work great for a very short time scale and people would zoom out that dashboard with the you know a thousand queries they would zoom it out to see the last six months of activity and that was you know surprisingly didn't work there well so the other part well as a you know we have in the actual controllers there's units that are really important to them instead of keeping documentation in sync for like this is a unit that we're using in code how does that make its way all the way up through the chain without futzing around it so with these you know essentially through the plug-in infrastructure we were able to kind of tick off all the boxes so we have a custom data source that calls in flux we sort of we read all the variables out of the PLC publish those and work out from there and then we've published various we have a bunch of internal panels as long as along with some panels that we publish so you know this is not rocket science as far as our infrastructures we've got a plant it's uploading data to a - in flux storage the s3 is really it's saving off like the variables it knows about the units that go with them what roll ups it has a bunch of configuration stuff so normally sort of our analytics is just looking at that kind of detached from the online system and then obviously again not surprising sometimes you connect straight to it and you have a live sort of WebSocket stream to get the immediate variable so that's how we do direct controls so you'll probably care a bit more about so the the controller we're using is this thing it's Bekoff as a german company that make a it's actually a Windows IOT container that has a really good interface for talking to its real-time variables so it has it's a real-time clock but you can talk to it and get get variable variables out pretty easily so we have a c-sharp program that reads what's on that keeps track of what those things are publishes a list like this a custom this is what the it's a visual studio based thing so this is real-time clock running it each cycle is 10 milliseconds I think I reserved a yeah 10 milliseconds so we made a custom data source that reads that JSON configure and here here it is a just a panel that sort of lists off what each of our fields are tells us what sort of what period they change at so these are the things that we're reading at you know 10 Hertz 1 Hertz 50 Hertz is it something that's what the reel is what units it's in which you so the comments that came straight from the the real time code it's also kind of use this to then pop up information about kind of our internal documentation for it so every variable links to you know what part it is where it's wired to what we expect the voltages to be and then the influx side of this is those are really like variables that depending on how we're asking for it where you can either ask the PLC directly we could ask for a short-term query we can ask just a simple you know give me the mean of the last you know within a short time that's a fine query otherwise you want to pull from a different aggregate the other kind of key panel that we did was just being able to control individual fields so this is a panel that reads that data source I'll give you a quick now I don't complicate things with trying to demo you I'll just show you some screenshots so this was essentially you configure a list of fields that you have which features you want on them but we want a way to be able to say okay I want to change my request speed so you tell it which rpm you want to run it and that writes that out so this is a configuration lets you pick the various fields out of our system shows various information about them lets us work with that so the other kind of main thing that we need to show was the state machine transition so as the machine is running through different processes we need to know like does the you know is there electricity to the whole system is the loop running is the pump going what what processes do do these go through and how do they relate to each other so for this we built a panel that's essentially previously we could and graphed a whole bunch of fields with ones and zeros and tried to make an influx I dig a phantom panel but that didn't work very well to try and represent states so this is a pretty simple panel that shows that sort of discrete string values as a series of on-off graph and published that the next one is obviously documenting because people looking at this don't know what those states mean like if I mean an additional initializing what does that mean or does it go to so this is pretty simple walks through your states so you're in running you can tell as you're moving from state to state and these are all gone again all based on the same query is the same data source that we're all used to FFTs was another big one that we needed was understanding the health of the system is based a lot on vibration so understanding that the behavior in the frequency domain so this likewise is a modification of that plotly panel that I showed that this shows over time we can see where our key resonant frequencies are and can zoom in to specific times across across that query the next kind of custom panel that we have is about faults so this is in industrial control there's there's alarms that we have that are within the rogue or Fano so we can say you know this field has gotten close to this level send us an email we have another class of things that operate entirely in the real-time environment so you know if vibrations spike we want to turn off you know within 50 milliseconds so that's not Agri fauna thing this is a internal to that but we use Griffon a-- as our UI to manage what all of these things are so each of these as a configuration to tell you tell you this is a fault at what temperature if the temperature gets too big we shut it off so here's a screenshot of a few of these things together so we see this is someone hit the e stop so the whole thing's shut off this is our test facility you can see other things where we have faults that something happened we're in a condition that's okay but you have to say okay we understand that it was okay as you click it it goes bad so you see this a lot of this UI here is like I find stuff in Griffin I copy it and then styles change slightly so it's a little looks a little funky over time if I tell you you know we'll get there so we've kind of we have used Griffin I and like all over in the company I'm actually not going to show you everything we have but this was one that our engineering team liked even more I'm strangely was a lot of people use NX so you have these licenses for NX this is a solid modeling a software package but it's really expensive and so we're always passing back like who has which license when so we're graphing that over time but who has which license and showing you who's got licenses checked out at the time you can see where they are there's this button at the top where you can say I want that license and it sends a slack notice to everyone so people people are pretty excited about that the other thing that I'm starting to use this in is we recently acquired company that's we're working with a company called upstream that we do a machine machine learning on environment data to try and minimize water use overall so this does crop monitoring so there's things that we're looking at land from satellite imagery and group checking out in this case you know when is it irrigated when is it not irrigated so this has some simple graphs in it part of another part of what I've done is trying to dupe our estimation monitoring so if we put us plant in this site with these conditions and these expected flows how much power would we produce but the problem with that is you end up making these crazy graphs like okay we're gonna have six engines they're gonna be in parallel how much they make so I'm looking now how can I use how can I embed gravano graphs in this other product so sort of you know taking me the iframe element a little bit further there's some issues here but we'll get there let's see here's here's what my plug-in panel looks like so there's quite a few of them I think Daniels is probably worse but so another panel that we have published is the influx admin one so influx is great we use it all the time they recently remove their admin tool so I sort of added a very basic tool to it that lets you type in a command and see the results click on them also another one that will sort of automatically update and show you what the long-running queries are so this one's published as well again recently published one that this is the USGS so the United States government publishes a bunch of data water usage it's kind of in a funky format so made a plugin that you can go look at these this one's kind of interesting because unlike all the other data sources this one you can connect to and there's like no credentials there's nothing it's just you connect to it and it works so so there's that I was gonna give the end of trouble that it wasn't published for a long time but it got published yesterday so that's exciting so I went through this quick because we're at the end of the day here but I really want to thank gravano because you guys are awesome and yeah if you guys have questions I can show you I can show some demo whatever you want you

Channel: Grafana

Views: 24,166

Rating: 4.9313307 out of 5

Keywords: Grafana, Monitoring, GrafanaCon

Id: I3o3tnj5j6s

Length: 17min 7sec (1027 seconds)

Published: Fri Mar 23 2018