Full Momentum Episode 32: HDF Files

[Music] welcome to full momentum in HC Raz podcast I your host Ben Cary we are here on episode number 32 uh joining me as always is Chris gell we also have a very special guest joining us today before we introduce Isa Chris I do need to give you a hard time about your Oregon State Beavers coming up a little bit short from where I know you kind of had wanted them to be uh so both both I think in zaga basketball in Oregon State didn't uh didn't get to where we wanted them to be this year and just to be clear for everybody we're talking about Oregon State baseball um basketball is not very relevant these days at Oregon State but uh baseball usually puts together really good team and yes it was disappointing wish we could have gone to Omaha we got to the Sweet 16 the super regionals um and just ran out of gas uh and you're I mean you're a big baseball fan Ben so you tell me like when the hitting just goes away is it more the Batters just run out of gas or they just run into a great pitcher or a little bit of both or yeah it's just frustrating when you can't get bats on balls you know yeah I think you know typically what you see in play playoff Level Baseball at at in college or in Pros is that you know the good pitching really starts to come out and I think part of that is because you know it's kind of do or die situations a lot of times in those postseason games so you're putting your best pitchers out there and I think it's pretty well known that pitchers will ramp up uh their their level of effort in in the in the playoffs so usually you get a few miles an hour extra on those fast balls you get a little bit more break on breaking balls so this is not a baseball podcast but I agree with you Chris typically late in the season um sometimes your your pitching wins so yeah yeah I think that's probably what it was and just to wrap it up I'll just say hey this was the end of an era of the Pack 12 and Oregon State got to play the very last game of the Pack 12 being that Super Regional baseball game so in a in a sense we were the last team standing moving forward now everyone's in different conferences and things will be really weird from here on out but uh rest in peace Pack 12 yep exactly I got great news for you Chris if if you want to watch more baseball uh this summer there is a another really good Northwest baseball team room in the pickles oh oh you're talking Seattle I thought you were going pickles okay um the Portland pickles may also be playing well but the Seattle Mariners are in first place in the ls so feel free to hop on board yeah I may have to do that to do awesome well uh for everybody who's who's watching uh the episode episode here on YouTube you recognize that we have a guest with us today uh so Isa is a H&H engineer that works here at Klein Schmidt she's been with us for a while now um and uh does some really really cool stuff uh within our group specifically within the HC Raz kind of Realm of our group and so we thought we that she would be an awesome guest to have on so Isa for the first time welcome to full momentum thank you Ben well I just wanted to say that I know y'all's favorite College foot college sports team suck but my Texas Longhorns did go to the football playoffs so there's that absolutely yeah I mean it's hard to compete with your Texas Longhorns you know across all sports but uh they do they do a good job on color scheme um so it's not the perfect color orange but it's close enough that uh I can get on board with Texas now if we if we want to if we want to pit or uh Oregon State fan here against Texas what we would really say is that Texas kind of was the first stone to fall in the uh uh breaking AP part of the Pack 12 because Texas and Oklahoma going to the SEC was the first Big Move That kind of started this whole giant realignment thing so not saying it's T's fault but it's kind of Texas's fault it was yeah like a mouse whisker away from Texas and Oklahoma being in the Pack 12 Isa then we really could gone at it yeah buing conference rivalry but uh yep didn't come to didn't come to be and here we are yeah so Isa if you want to give a brief kind of introduction to yourself and how you ended up here at kmid and how you ended up here on the vcast so yeah so I joined kmid I think almost five years ago I was wrapping up grad school I went to the University of Texas of course for under and grad school for Water Resources engineering um and really wanted to work with Chris work in Portland um so I moved out here in 2019 and started working at clment since then I feel like I've been working on a lot of really cool Ras projects um with help from Chris and Ben and then learning a lot on the job um but yeah super happy to be here I think some of my favorite things working at punishment have been all the modeling but also getting to play around with a lot of coding stuff in my free time and on projects and so that's been something I really enjoy doing um certainly always have a lot to learn about it but um it's a really cool kind of way to Branch out so I still remember when we interviewed Isa and um you know her resum was was stellar and uh you know looked really good and that's you know the the primary reason we uh we brought her forward for an interview um but then when she got on and we got to have a conversation we were like wow she is uh really polished she's got great communication skills this is like a rarity in the world of engineering you know super smart good resume and can communicate so we we were like okay yeah we we have to hire we have to hire Isa and then we didn't hear from you from like for like I don't know a long time we're like oh no she went somewhere else um so I'm glad that wasn't the case although uh I think you were probably shopping around a little bit but hey that's okay as long as you ultimately came to Klein Schmid end up on the right place that's right that's right yeah yeah and Isa has brought a lot of really cool unique skills to our H&H group I know Chris before Isa you know there wasn't a lot of folks within our group that knew a lot about coding and how to potentially utilize that skill set within you know an H&H modeling realm but but has brought a lot of that to our team and that's going to be part of the top special topic that we get into today um when we get into the the primary topic uh but before we do that I do want to um start off with kind of a fun segment and uh this idea came from a listener uh to the podcast and somebody who apparently is pretty active within the uh the Raz community space Ryan Troy had the idea of what are the full momentum Ten Commandments of HC Raz modeling he threw out a couple good ideas there um for what he would kind of start his list with and so we thought hey that's a that's a cool topic Let's uh let's bring Isa on to contribute and kind of develop our our Ten Commandments of HC Raz modeling um so we each came up with uh three Chris you have that bonus fourth to close us out to 10 uh so why don't we go through one at a time and just share what our kind of uh core foundational beliefs about HC Raz are all right so this so mine are fairly General uh non-specific uh kind of high arching Commandments of hecr modeling can really be applied to just about any software but super important in HEC RADS modeling the first commandment is know your objectives so know why you're building your model what's the purpose of it what are you using it for what are the answers you're sorry the question questions you're trying to answer and if you keep that in mind throughout the entire process you're going to end up with a much much better model and probably um not have it crash on you as much yeah very good that that object or that uh commandment goes really well with with isa's first one so Isa what what's what's yours my first one similar to Chris kind of defining your objectives I think is checking your time step um I feel like that is such a crucial part of the computations um it makes a big difference sometimes if you don't pay attention to what your time step is um and it kind of got to ensure that your model is doing what you think it's doing big part of that that that's a good one I have a funny story about that so in earlier versions of hecz um maybe before 2D came out but certainly you know people were do an unsteady 1D modeling with it the default time step was 1 hour for models and so people would come in and they didn't know that the time step was something they had to change and so I can't tell you how many times somebody would send me their model and it just crashed right away and it was simply because the time step was left at the default of one hour and then if I just changed it to whatever it needed to be 1 minute you know 30 seconds whatever it ran perfectly and uh sometimes these poor people spent hours or even days trying to figure figure out why crashing it was just a Time step thing yeah which is why it's such a good commandment when it comes to using HC R my first one is uh you should always use distinct naming convention for plans geometries and flow files this is another one that uh we see all the time when we get models sent to us uh and to be fair I put my hand up I do this at times too if you're just trying to bang out a quick model get some stuff running it's a lot easier to name it plan one geometry one flow file one then to give it distinct naming conventions but man alive Isa you can attest to this if you have a a big model with a lot of different runs having very distinct clearly named plans geometries and flow files will make your life so much easier as a Raz user so I think that's a good uh commandment for everybody to abide by as they're developing their Ras models yeah really go ahead yeah go ahe go ahead just that it's really been something I've like tried to start nailing down because there's no worse feeling than being like uh what does this plan do exactly yeah yeah can can I add a kind of um an add-on to yours too and that is keep all those files in the same folder um I don't know why it seems like I've been seeing this a lot lately you're breaking into my second commandment Chris oh I'm sorry you didn't review you didn't review our own man y I did I just saw terrain I'm sorry okay I'm getting ahead all right I'll shut up okay all right Chris what is your second what is your second commandment oh man okay um this is another kind of General one but keep it simple so keep your model simple Ben and I talk about this all the time in the class that we teach that you keep your model simple build it in stages don't add things in in that don't need to be there keep it nice and concise that's going to make your model run faster it's going to be easier to troubleshoot and you're just going to have less problems overall keep it simple basically what I like to tell people is build your model just to the point where it can answer the questions you want to have answered and nothing more than that good keep it simple and I left the uh the other s off you know the kiss principal because I you know I don't want to call anybody names not the Commandments what's that yeah that's right we have to be have a friendly Ten Commandments here all right Isa what's your second uh my second one is to not rely on the green finished computations message or on the full blue bars um always kind of look closely at your results look at those stage and flow hydrographs um look at RAS mapper look at your crant numbers profile plots all that good stuff um to make sure the results are really doing what you want them to and think they're doing yeah this one's really important particularly for 2D modeling right um ond modeling tends to give you better output messaging in terms of if there's errors or issues but you can have a pretty poor performing 2D model and not realize it if you're just relying on those two things so that's a that's a really good one Isa yep yeah yeah really good my second one is to Chris you were you were uh you were Prophet here you were a prophet you know predicting what was coming here but uh my second one is to include all of your model source files inside of your model file structure um particularly if you when you zip up and send that model to somebody else there are again so many instances uh in my experience where I'm reviewing a model that came from somebody else and it doesn't have really important basic pieces of information included in that model when I open it up and that's because of the fact that they didn't have the terrain file or the projection file included in that same kind of model structure um and so when you go to zip everything up and send it out those links get broken and you're missing really really critical information so always include everything that's in your model should be kind of within that modeling uh structure now you can have individual subfolders for things like terrain data or GIS data but it needs to kind of all be within that same um model folder structure so yeah here's a little tip for everybody too if you use the um the zip or archive projects um option under the file menu item the main Ras window it's a good way to send projects to somebody but if you do that open it up unzip it before you send it just to see what it looks like if it has a folder in there that says external dependencies that means you've got files outside of your folder structure and uh that's going to really screw up the person who receiv use your models so um yeah that's a great one Ben sorry I stepped on your toes there and uh gave everyone a little preview but um it is a really good one and I I see people breaking this commandment a lot lately I don't know what it is but you know that there's this desire to kind of compartmentalize different files and different folders and and it's just don't do that just put it all in the one folder and like Ben said you can have subfolders too if you want yep yeah what I always like to do is zip up the folder bring it onto my desktop open it up and then open up the model there um and if it everything's looking good in terms of the file structure then I know that if somebody else does the same thing they'll be able to see it so it's a good little kind of self qaqc all right Chris what's your third commandment our seventh as a group read the manuals um It's seems pretty obvious and I'll be the first to admit uh I'm not a manual reading guy whenever I get um you know a a desk to build at home or you know a bookshelf or something I don't read the manual and um but in this case with hecz modeling the manuals are so good and there's so much information in there read the manual if you're running into trouble you're not sure how to do something get in there and do a contrl f to search on whatever you're looking for um but there's a lot of stuff in there that can be really helpful and I recommend too if if you can if you can figure this out get a full PDF of the manuals to bring onto your computer because it's at least in my opinion it's a lot easier to search around when you have the full PDF versus the online version It's kind of segmented into little just mini chapters here and there so it's a little harder to search through the whole manual that way but y that's just me good good well I might have read the manuals to pull out a trivia question today uh for the end of this podcast so all right we'll see who actually reads the manuals then exactly exactly uh all right Isa number eight number eight um this is one that I feel like I have made a mess on multiple times so my number eight is double and triple check your datums when you are pulling data into to Raz um especially your vertical datums horizontal datums are usually kind of make sure you have the right one but once you set it it should be pretty straightforward put the vertical datums you're pulling data in from a bunch of different sources document it all um write it in the descriptions it makes your life way easier yeah yeah especially I think like you noted there each if you're using multiple data sources uh this commandment becomes exponentially more important um you know if you using one data source it's still important to check that data but worst case is your you know results are off by you know a set number of feet for instance if you're using a wrong datam or a different datam than what you thought you were using but when you're combining different data sets that's when things can potentially get pretty hairy so really really important um good one number nine uh and my last one before we turn over to Chris for our 10th commandment and that is conduct sensitivity analysis on uncalibrated model parameters so um as you're building out your models um in many cases hopefully many cases you have calibration data that you can use to refine and um you know build some confidence in your uncertain model parameters this could be values like Manning's end values turbulence coefficients um other kinds of you know energy loss or or um important parameters that affect the results within the program but if you don't have calibration data um or if there's particular model parameters that you weren't able to calibrate with the calibration data that you have it's really important run sensitivity analysis on those to really understand what your what the results in your model mean what the uncertainty of those results are and that's going to make your presentation of those results to your customer so much better um and you going make you a better better modeler overall that's a good one yeah close this out so last one is uh report bugs okay so we are a community of hecr users and there exponentially more users out there than there are developers on the hecr team so the developers can you know only catch so many bugs but it's inevitable that we are going to find bugs when we're using the software so please please please if you do run into what you think is a bug send a bug report to the developers if you don't know how to do that go to their website ATC website and you'll see a place for a bug report um if they don't get right back to you don't don't think they're ignoring it um it's likely they're busy and they probably have a you know a list of them they're working on um and if you're still not sure get a hold of me send it to me I'll forward it on to them um because I think that just makes the whole software uh experience better for everybody and you know hopefully uh can avoid some serious errors and inaccuracies in models so and there have been some big ones in the past and um I just ran across one recently I won't mention right now but uh I have already shared it with HC so um anyway yeah yeah good so to review the full momentum Ten Commandments of HC Raz uh we have Thou shalt know your objectives keep your model simple read the manuals report bugs check your time step don't rely on the green finished messages and blue bars look at your results in more detail double check your datums use distinct naming conventions include all model source files when you zip and send a model out and conduct sensitivity analysis on uncalibrated model parameters I think that's a pretty good list if you do all those things you're going to be in good shape I think that's a really good list and I'm sure uh you know our listeners out there uh if anybody's still tuning in is uh probably has some other uh ideas for what could be on this list so yeah let us know um and I think Ben we ought to make some official stone tablets to put hang on the wall behind you for the next episode so that we can continue to remind everybody what the Ten Commandments of hecz modeling are there you go there you go good good well thank you Ryan Troy for that idea that was a fun little segment and thanks Isa and Chris for your uh participation there so all right let's uh jump into the primary topic for today before I before we do that I do want to just give a quick shout out to uh the sponsor for this episode that's our firm Klein Associates who is known throughout the industry as a firm that provides practical solutions to complex problems affecting energy water and the environment you can learn more at kinmit group.com um so thank you Klein Schmid for for sponsoring our uh discussion today and allowing us to to share silly things like the Ten Commandments of HC Raz but also cool more in-depth technical topics like our discussion today on hdf files so we wanted to talk about hdf files this is an episode that we've been wanting to do for a little while um Isa has done some really cool work around this topic within our group so we thought she would be a perfect guest um but Chris why don't you go ahead and kick off the discussion by giving everybody kind of a high level overview on what is hdf what are hdf files yeah so hecz historically um before version five came out was uh made use of two different output files they're called the for lack of a better name the output file that's the O file and then the DSs file the O file contained a lot of the post-processing data from uh for 1D models and the DSs file kept all the unsteady time series data and so other stuff in there too but if you look at a project folder like like I'm doing right here uh this is one I'm working on right now but um you can see that we have um no o files and why do you think that is Ben oh you're trivia me before the trivia uh yeah is it because Isa is it because there's no 1D element I take it back there are o files yeah sorry I messed you up I was because I was like I'm prettyy because output no output files would mean no 1D results right that's right yeah and this is a this is a 1D 2D model so this this is a 1D 2D model so yes you're right um sorry I I kind of flubbed that but yeah there are output files but that is because we do have 1D elements for our four different plans here and so this again will only contain 1D output and it's all the post-processing stuff there's also DSS file this is another output uh file that we have and that keeps mostly time series data so if you have an unsteady 1D model or 2D model you're going to see data sent to the DSs file uh what's interesting is if we look at the unsteady flow analysis window the detailed output interval that we see every time we hit the compute button that sends your output your detailed output post-processed 1D output to the O file so this is where the o file is being built and then the hydrograph output interval is where the DSs file is being built and then the mapping output is the third and newest output file called the hdf file and so if you look kind of scan through here you see a bunch of hdfs in here and Raz started using this for I think in version five and it was specifically because models are getting much larger now being 2D and uh taking longer to run and the hdf file is built for that it's built for fast data processing and data exchange and so that's why we have the hdf file and you can see that there's an hdf associated with each of the plans so here we have plan one and then we have our plan one hdf file this is our output hdf file this is when you go into Ras mapper and you look at your velocity map or your depth map those are all built off of data in the hdf the plan hdf file there's also a geometry hdf file and this is strictly your geometry input data stored as an HTF file so we have our regular geometry file and we have our geometry HTF file the geometry window reads this file the go1 and Ras mapper reads this one go1 hdf and so when you're in Ras mapper and you save your geometry it actually saves a new go1 file and likewise when you're in the geometry window and you save your geometry it saves a new geometry hdf file so there's some other hdf files associated with the unsteady flow and and even your terrain has an HTF file as part of it but um I just wanted to point out they're all over the place in your file directory and those are output files and you can look at h HTF files but you need a specific kind of software called hdf view there's probably other ways out there other software that'll look at hdf files but the hdf view is the one you can get right from um the web it's free it's called hdf view if you just Google that you'll find it and this will allow you to look at hdf files hecr hdf files in this tree format and you can kind of drill down to find out exactly what you're looking for and if you're looking for results you usually have to go results on un steady output output blocks base output unsteady flow time series 2D flow areas and then finally your plan name and then you'll see base velocity and water surface which by the way are the only two things computed in a 2d model did you guys know that by default well these are the only two things that are actually computed in the unsteady computations everything else is postprocessed off of that so you get you get your face normal velocities and your water surface um and so with these two things you can create just about anything you can think of and this is where this is why we brought Isa in here because she has an expertise in programming using Python and interacting with these hdf files and um which I really love because this makes life a lot easier in some cases and quicker and faster and more precise so if you go in and look at um it's kind of interesting if you look at one of these records we'll look at Water surface it pulls up a you got to double click it and where is it it's hiding somewhere on my computer I'm sure uh Chris real quick the folder under 2D flow areas it's not your plan name it's your uh 2D area name y you're right yep yeah because it does say 2D flow areas yep that's right that's your two yeah I just happened to name that my plan as well that's the confusion so um BR one of our Commandments not have playing out in real time oh gosh I couldn't have scripted that any better do as we say not as we do okay um all right so yeah you double click on any of these these records and you get a table of numbers and a lot of numbers a lot of big numbers and you can scroll over and see everything and unfortunately the hdf viewer and I don't know if this is on the Raz side or it's just the software the hdf viewer software itself but it doesn't tell you what the the numbers on the you know the y axis versus x-axis are in this table so you kind of have to just either remember or refigure it out every time but I know this is time step over here and and more precisely it's your mapping output interval um because that's how this data is stored by your mapping output interval so these are all your mapping output intervals and then across here is your cell number from zero all the way to 3361 so I know I have 3,361 cells in this project and I can get for every mapping output I can tell you what the water surface elevation that was computed for each of those is now you can imagine this is trying to find you know exactly the water surface at a specific location you have to figure out okay what cell is that or what group of cells and you have to come in here and kind of scroll around or you can do what I do and call Isa or your favorite python developer and have them write a script that'll do it for you very quickly now before we before we get into scripting just a couple questions for those of us who are novices in the scripting World um Chris the RAS mapper when it's basically plotting depth velocity Dynamic Maps static Maps is it accessing the hdf files to do that yes it is and um that I'm glad you brought that up because when you go to animate your model and you either you know press the green play button or you just manually an animate every refresh of the screen whether it's a velocity map or water surface or depth map it doesn't matter every refresh of that screen Raz is accessing the HTF file this thing that you see right here it's pulling the data out it's Computing whatever it is for every single cell or cell face then it's interpolating between them and then it's putting that on a map that's all done in a fraction of a fraction of a second and part of the reason that can be done so quickly is because Raz uses these HTF files which are data storage files that are very very quick to work work with so yeah and you'll also notice that as your models get larger and larger and longer and longer your hdf files the size of those gets more more significant so yeah that's a really good point in fact yeah if you ever have a really large model it's almost a guarantee the biggest files are going to be your hdf files now these because this is a small model these are pretty manageable uh although even 10 years ago I would have thought oh that's that's a big file but nowadays in Raz World these are these are small manageable files um but these can quickly get up into the gigabyte size so that's a good plug for keeping your uh output interval smaller uh or sorry not smaller bigger so you have less output intervals so if I you know open this up and and we go back to this window here this output interval the smaller you make this number the bigger your hdf file is going to be and of course the longer the simulation time and the more cells you have the bigger your HTF file is going to be also yep yeah yeah so I was working on a Model sorry go ahead yeah no go ahead Isa I was working on a Model recently where every time you open the HTF file in HTF view once you opened a table it would crash so you can only look at your table but if you want to look at a different table you have to open it all back up again just because the hdf file was super super big oh really oh was that a um do you think that was just a a computer limitation or there's something with the hdf viewer that wasn't letting you I would guess it's a computer limitation but it does kind of push that so to keep in mind definitely yeah I mean you can with these large H or yeah large hdf output files you can quickly overwhelm your own computer and I get this a lot from people who say ah Raz is just not running it's it's crashing on me or it's it's just um it it's halting mid-run or something like that and then you see something like um something to do with file size or access issues a lot of times that's cuz you just basically filled up your hard drive and there's no more space left so so be careful about that yeah a question I have for you Isa and this might be a good opportunity for you to share your screen is just around some of those additional options that you can get uh in your output um that can be stored in the hdf files um I know Chris talked about the the default ones that are computed but there's a lot of other information that you can have post-processed into your hdf files I believe do you want to tell folks about kind of how to access that and and when you might want to do so yep I can share my screen if you don't mind CH yep so there's a lot of cool information it's been said that can be uh written to the hdf file that isn't normally written so Chris reference usually it's just the uh face velocity and the water surface elevation um but there's a lot of output that you can maybe want out of your model um and you want to read using the HTF file so you can that if you go to your unud flow analysis window options output options then might have written some restart files and such um this detailed log output will write to the O file if I'm not mistaken but when we start getting into here we start writing to the to the HTF file you can I know here you can definitely write the HTF file but uh computational level output um if you want info at every time step that's something you can do and I know um Chris and I have done this on some projects where we weren't able to capture like a peak number in the regular output so kind of bracketing where we think that's going to happen um and writing computational level output and then we have this entire list of hdf5 right parameter so HTF 5 name of the file um a lot of different Duty variables to be written here so these are all things you don't want to write unless you need them um because they do make your files pretty big as you start clicking more and more of them um but there's a lot of really interesting information here so I know I've used the hydraulic depth invert depth sometimes um cell velocity is a big one so this will give you the cell X and Y velocities in case you need some directional information cell volume that one in particular cell velocity is an interesting one um because Isa correct me if I'm wrong we've used that XY velocity in um other model types such as an agent-based model um that needs that sort of XY component to our our output velocity um so it's it's a way sometimes there's other programs that require more detailed information than what we can provide with from Raz output by default but we have a little bit more flexibility here yeah absolutely um so these are all just really really interesting things that Raz doesn't need to finish the computations but it is something that it can compute um as it goes along and you can get some more information out of the model um I certainly haven't used a lot of these um but if you have a use for them or if you have you know Downstream use for them um definitely worth checking out what is that um exclamation like warning message button up there what is that I've never clicked on that let's find out together learning things I've looked at this before but it's been a while for sure this okay this is making really good pod us all reading this so um it's okay I think it just has to do with the written velocity at individual nodes um because that velocity is not actually used right it's postprocessed so I think it's just a kind of a warning I don't know why they give it for that one and not some of the other components but yeah maybe just because it's common it's like up here it's not even well maybe that's be maybe that's because um Chris correct I'm wrong there's other 2D software out there that actually computes velocity at nodes and not at cell faes is that right um yes I believe FL 2D does that um yeah I'm not sure anyway we can yeah we can cut that out but I wait I I I think there it may be because of the fact that there are velocities computed at nodes for other softwares and so if there's somebody who's used to using that comes in here and sees that you can get face Point velocity they just want to be really really clear that that's not something that's computed yeah yeah yeah I mean if you if if you uh you know forget anything everything else remember that Raz in a 2d area only computes water surface elevation for each cell and face normal velocity for each cell face everything else that you see is post-processed off of those two variables yep uh one other thing I want to point out here is the very top option under optional hdf output parameters which is the right to warm right the warm-up time steps to Output um this can be helpful particularly if you have a model that's crashing immediately when the model starts to run but it's getting through your warm-up time um that sometimes allows you to view those results during the warm-up time and sometimes you can tell based on that uh maybe what's potentially causing your model to crash before you get any you know realtime results so that's a a helpful step there too yep to consider so Isa I'll maybe pass it over to you now to kind of change the direction of the conversation from the tangible understanding of what hdf files are how we can modify what dat actually goes into them within the program and shift the conversation towards how we can potentially utilize coding software to help us even take this to the next level so this is kind of where my expertise really falls off and I'm going to leave it to you and Chris to to kind of pingpong this conversation a little bit well then I think you can certainly ask good questions and provide input um but yeah I I was kind of thinking about when I've used hdf files um instead of just using the default Raz output and do want to clarify that there's a lot of scenarios in which you don't need to go to to the HTF file level um Raz does have a really good way to look at look at outputs for ond models for um a lot of the use cases for 2D models but when you do want to use HTF files it's oftentimes just a u volume thing you have a lot of data you need to look at um maybe you want to look at the same data over and over um that makes it kind of hard to keep clicking on points and maybe plotting water surface elevation um for multiple multiple uh plans multiple locations um or maybe you want to keep looking at the same type type of information across multiple projects um another times hdf is really nice is when you want to kind of have that unique output that we talked about um and if you maybe want to just go ahead and have that plotted or formatted in some specific way um I think going straight to the HTF file is really valuable um yeah I don't know do y'all have any kind of Standards when you go to use HTF for well yeah I mean nowadays it's so easy to get specific output right in Ras mapper um that I don't use the HTF file or viewer to get to get that data as much anymore because it's so easy it didn't used to be so easy to get that data out but I would say for me it's it's if you want something that is like you list here is not standard hecr output maybe you have a special way you want to compute shear stress or uh you have a a certain parameter that um is desired for Habitat evaluation you can go in and you can pull your water surface elevation and face velocity or some of these other parameters that aren't there by default but you can add them in as options if if those work for you and then manipulate the data yourself using python or you know at a really basic level you can copy and paste out of the hdf file into Excel and do manipulation right in in a spreadsheet but uh python as I'm sure you'll agree Isa is a much more powerful way of doing that yeah I think it keeps it really clean um kind of reduces the human error involved of course there's room for other human error but yeah and I'll give I'll give everybody a hint too that Isa taught me about but python makes way better plots just cleaner looking and just more professional looking than you can get in Excel sorry Microsoft but Python's got you beat there it's so true I every time I make an Excel plot I cry a little inside [Laughter] but you almost made me spit my coffee out when you said that that was [Laughter] funny um but before we get into that I kind of just wanted to do a brief like chat about python um I don't know I'm certainly no python expert so I want to make that clear I've just kind of learned python by trial and error a lot of winging it um probably a lot of messing up but I think I I certainly don't know a lot of the intense python stuff I do a lot of the data processing data analysis GIS kind of stuff um which python is great for so I think that's still valid I think that's a really good thing to mention too because I think there are a lot of people who are a little intimidated by python myself included because they have this impression well you have to be a computer science expert or you got to know you know programming code but you're saying hey I taught myself this basically yep I taught it to myself in grad school and you know only messed up like 15 times and like broke my computer a couple times but not to scare you um but yeah I think there's a lot of free resources especially for python it is open source um it's totally free to use like basically every library on python is free to use um lots of great resources i' start with YouTube um like infinite numbers of intro to python tutorials um but I know I really like if you do have some money you want to spend or even going their free version I love data Camp um especially for those data science kind of stuff I think it's a really great kind of way to sit down be able to try things out and kind of learn um while you're like basically you get your own little coding environment on browser you get to It'll like check your work be like no you need to do it this way um I think it's really great way to learn all these data science things I think python are um all great resources I kind of know R I'm not that good at it so I won't speak to that much um yeah I think the other big thing is with I think with a lot of coding stuff is you don't have to know how to do it you have to know what to Google so what do you think about what do you think about using large length anguage models like chat GPT to help you code have you used that much Isa and how has it worked out I've done it a little bit I think it works great you definitely want to make sure you understand what it's doing um you don't want to just take some code that you can't kind of process in your head and maybe you get an example you go back you kind of do some research on what some of the libraries and stuff are doing um I love it I will also say I love figuring it out um I'm kind of like dang it I don't want to have to use chat TPT because it's faster because sometimes I love to getting in there and having things fail on me 50 times then you get it on try 51 but yeah it to me it seems like it's a lot like technical writing where if you can write a a report or a memo on a particular technical topic then I think using you you know um AI like chat gbt to help you write it faster is is a reason reable approach but where you get into trouble is if you are using AI to do something like write a report or write some code that you yourself don't have the ability to do and then that means you can't review it for its accuracy for its applicability for other things so I think that's the big differential to me um Chris I don't know if you agree with that yeah I think that's a really good point I mean everything you produce you want to make sure you're you're producing something with um you know the highest degree of quality and how do you know if you don't know anything about the code that you're using so yeah it's a great way I I like using it to get me over a hump you know like I just can't figure out how like what there's got to be a way to do this you know besides me writing you know 20 pages of code there's got to be somebody who's put together something that you know a library that I can steal some code from or whatever and that's a great way to um or chat GPT or or other large language models great way to to help you get over those hurdles it I I I feel like it's a very similar thing to when I started coding way back in the day um way before you guys's time um and I was using VBA in Excel and I used macros to learn how to write code in VBA and macros you basically you'll just perform a bunch of tasks in Excel while recording them quote unquote recording them and it would write the code for you and then you can go look at the code and go oh that's how I tell Raz to move from this cell to that cell or to you know put a you know change the setting here or whatever you know you can get that all just by recording a macro so it's kind of like a new modern version of that so but I agree with you Ben 100% that um you know definitely know what it's spitting out you know if it helps you get over a hurdle great but understand what it's telling you or what it what the code is that was written for you yeah now I know we're kind of getting knee deep into some of the the coding conversation but I'm sure there's users out there thinking you know what what's the real reason that you would want to use Python for evaluating or or processing hdf output so Isa is there some some examples or some particular situations in which maybe this is helpful to be able to do if you are a Raz user yeah so I kind of came up with a list of a few different examples that I've used P python um some of them were way more intense than others so the first one was pretty simple um Chris and I were working on a breach model and we are unable to get the peak breach flow out of the Raz outputs um so no matter how small we got the interval we realized that the peak flow wasn't being captured um and so what we did is we wrote computational level output um and then just had a really simple script that pulled the computation level breach flow um time series and it found the maximum and just spit it out um so that was a really interesting like tidbit of like hey sometimes Raz isn't giving us a full picture of the answers um but there's a little more information if you dig a Little Deeper so this this was to look to get Max flow everywhere um regardless of time but just you know what was the maximum flow that happened at every single cell and uh at the computation level yeah you can imagine at the computation level how many records you have in that hdf file trying to find those manually would be a um well a chore to say the least and that was maximum flow at each cell or each cell face uh it was actually the maximum breach flow okay okay so oh sorry okay yeah I was thinking of something else but yeah Max breach flow okay so not at the cell but going through the the breach opening going through the breach opening it just wasn't being written um in like Ras mapper or um in the hydrograph Stow hydrograph windows so we just needed to be able to replicate what MCB was telling us um what about what the max breach flow was um so that was kind of interesting pretty simple um I think the next one I think was actually really cool and I use some of the examples um in the code I have to share today but we had a model that we had to calibrate at multiple different flows um using some shape files of level loggers um so what we were able to do is have python go in um read the shape file and figure out what cell was attached to each location of level loggers um and then go into the hdf file and read the water surface elevation and velocity data for each of those cells plot it and Export it to an Excel file and compare with the um calibration data that we had so that was a really efficient way to do it because we kept changing geometries um mult geometry so each of the cell numbers kept changing um kept making tweaks to the model and so you know every time you recompute that 2D area you're changing your cell numbers um so to keep going in each time and pulling all that data for I think like it was upwards of 20 Level loggers was kind of a pain so we got python to do it really fast um I think it was a great great use of it and definitely kind of fun to have that pulled together very cool was that a was that a difficult uh code to write I think it was difficult in that I had to it was one of the first times I used hdf files so it's was kind of difficult to kind of piece together all of the information of like hey where should I get this from and like um how do I read through this and figure out what information it is that I want but I think the actual coding itself is that hard um that was that was pretty interesting um fun fact I think the hardest part of it was figuring out how to get python to read the time step time series The Format of it took me like hours I think but I know it now so we're all set well one of the one of the trickiest things I think with um reading HTF files through a program environment is knowing the of the HTF file I mean if you remember the the data tree that I was um showing before oops sorry um my Wi-Fi just cut out for a second yeah if you remember the data tree I shown before there's a lot of different paths you can go down to get to data and so you kind of have to decipher that you have to figure that out to be able to code it into your python script right for sure um I think having the HTF open on the side is really helpful when you're kind of piecing things together yeah um other examples I think we kind of talked about this one but using the X and Y cell velocities for fish passage analyses um and agent based models that we're doing on one project um I think that's a really unique way to look at the information so yeah are there any examples Isa that you can show us in terms of like how the code actually works and how you can go from you know Ras results to code to product yeah well I I kind of pulled together an example that I can go through okay that okay um so we see this right yeah okay I was looking at the Mony uh kind of example data set from HC Raz um and this is I think the just a 2d uh breach plan um through every Mony Indiana I kind of just pulled this one as an example I don't know a lot about it but can pull some data from it so here I was trying to figure out if you were in the town of Muny during a Crazy Event um and you wanted to figure out how to spend your Friday night um you need to know where what the flood flooding was going to look like um assuming you can predict the breach um you can kind of piece together where where it's safe to be um and what you can walk through so quickly your I do this every every time I go out to dinner I always check the flood maps there you go so you have some options here um you could go catch a high school football game um you can go to Elm Street Brewing you can go to Madison Park or maybe you're very uh cultured and you can get watch a play at the theater um so just for reference if we just like play this where would where would you likely to be be on a Friday night Ben of those options oh probably the brewery which means I'd be wet I know Isa would be at the theater I'm not that culture you're not oh okay um so say you don't want to run this entire simulation you just want to look at the results in your little python hdf Raider um we can pull them in so how we would do this um set up all your well set up your python so that you import all the necessary packages I think a lot of these are pretty straightforward they're pretty common data processing tools um pandas numpy map plot lib um um datetime functions but the important one here is this H5 Pi um this is what's going to help you read your hdf file um and these are these are all just available like you can find them online or or where do you get all these libraries great question um so I like to use Anaconda Navigator so if you you can just download this it's open source well it's not open source it's free um and you can kind of set up your virtu environments in here and then you can open can manage your environments um and add all of your different packages um and go in and use all of these nice platforms I love Jupiter notebooks I love using spider for other types of scripting um yeah and it'll all be kind of set up in there for you and it's a much better way to manage your packages um rather than downloading them yourself yeah and for for those of you who are not very familiar with coding or python coding each one of those libraries has its own like library of code that you're borrowing basically so you know you got two options you can write all that figure it all out yourself or hey use what somebody else has already put together and um yeah and I will say these are all uh very standard very common um using python without some of these would be a totally different game so um yeah we could just run that and see it's run a little number in there um so now we have all of the libraries loaded into our code and then we pull in the project and the input variables we have so I have this hdf path variable over here that just links straight to the plan hdf that's where I'm going to read the results from um and then I also specified the name of the 2D mesh um just because that's one of the inputs into the hdf file similarly what I did for this project just to keep it simple is I created a dictionary with the cell numbers for each of the locations we want to look at um just so we can kind of point the HTF like Point python to read what we want out of it really clever what did you call that a dictionary yep so a dictionary is kind of a standard variable type in Python where you have like a key and you have a value um so the key here is a cell number and the value I put as the name just so we can reference it later okay yep and then all I did here is I just kind of translated this dictionary into a data frame um so kind of like a spreadsheet table um just so we can kind of help start populating some information so I can go ahead and run this nothing crazy is going to happen it basically just defined these variables um with the paths and the names and um the cells that we want now this is kind of where we start getting into the meat of the code um so I have a couple different functions here that I've written um this first one is called fill cell minimum elevation so what this one does is it goes into the HTF file and it pulls out the minimum elevation for each of the cells and this is helpful um because we're pulling water surface information um HF file if your cell is empty or guess if it's dry it just writes your uh minimum elevation instead of the water surface so we want to make sure that we can compare them and make sure that there is actually water there and it's not just dry um so what this is going to do is just pull it in um this is the main function for reading the HTF pile and then um goes through and reads the geometry 2D 2D FL areas and then we're filling in the mesh name reading the cell cell minimum elevation and then populating it in our data frame um pretty pretty straightforward um similarly we're going to go in to the water surface elevation read the same thing the difference here is that the water surface elevation is a Time series cell minimum elevation doesn't change over time water surface elevation has a Time step associated with it um so go in here put put in our results pth so this is kind of something that's helpful to figure out ahead of time um python will probably give you errors if you don't have this figured out correctly it'll be like you can't read this hdf path um so that's where it's really helpful to go into HTF you figure out what it is you want to look at um similarly filling and our mesh name um and then we get results I like the comments you have in there because I think that's super important I mean not just for presenting it on a podcast like ours but you know for you going back you know years later and wanting to use this again and going well what the heck does this code do here but having lots of comments in there can really help refresh your memory or if you got to give it to somebody else they can see what you were doing too absolutely I was honestly pretty proud of myself that I went back to the original code that I pulled this from and I was like oh I already wrote comments in there nice yay yeah it's not uh it doesn't always happen that way for me so this is kind of a stellar example on my end um so once you execute a script like this what is what do it what does the results look like compared to you know what you have in in Ras mapper okay so the results can kind of look like whatever you want them to look like right um so if I just run everything here I end up this is just kind of a clip of the table that I have but I have a Time step as my index and then I have columns with each of my cells and it has the water surface elevation I told the script that if the cell minimum elevation is the same as the water surface elevation I want it to write in a nan um just so we don't get kind of confused with our results and then if you take this information and you want to plot it so you can see it really easily come up with a really nice plot of what the water surface looks like over time so I know Chris you kind of mentioned the plots in Python look a lot better than Excel this is a great example of that even though pretty straightforward yeah um so we can kind of figure out like hey Madison Park and Elm Street Brewing and the high school football field are going to start flooding um but if we went to the theater or the cupcake place we're gonna be in the clear um very cool you can also see it looks like a Time timing a little bit on that too Madison Park starts flooding a little bit first and then Elm Street and then the high school football field yep exactly um so obviously there's like infinite ways to look at this data um and this is just like one kind of far stretched example of a Friday night plan but you can you can kind of look at things in as many ways as you want so you could be looking at velocities you could be looking at depths you could be doing some type of postprocessing on this data so for instance maybe you had depth data but you only cared about where the depth was greater than two feet because that's what you can walk across you could filter it um you could potentially even map it yourself if you wanted to create some different types of mapping products um there's a lot of good GIS libraries in python as well that can help you do that um yeah I think kind of just a cool way to re-envision the kind of outputs you can get from res and is it fair to say that you know this example like you said it wouldn't be time prohibitive to do this just yourself within r mapper I'm assuming it probably took more time to write the script than what it would do in that particular case but where you would potentially utilize something like this is if you found yourself hand having to do this for multiple plans multiple models and you're adding up the time in your head about how long this is going to take you and you're like well maybe this is the opportunity to write a script to have this kind of automated or maybe this is a process that you implement for many of your project types and so it's something you could use over and over again in the future those are the type of situations where maybe you would consider jumping into this absolutely but I also just want to say that when I pulled this together from a previous script it took me all of like an hour including writing a bun think of the rest of this so once you have it written it's really easy to just go in here and adjust this results path to whatever it is you want to look at okay um and then it'll just kind of plot whatever you want um so there's a lot of ways a lot of ways to do it I think some of the other things that python can do that Raz can't do is just iterate so say you want to look at three different models that are located in three different folders and you want to plot everything on one graph that's super easy to do here um whatever reason that would be that you would want to do that um um so you want to compare some models you want to say hey what what why do I have multiple versions um of the same model what what are the differences in the results this would be a really easy way to see that um so you can kind of build on the basics here super easily um I also want to just point out because I referenced this the date time stamp um in the HTF file you have to use this very special shape called s22 and that makes reading it very easy um I don't know why is it because it's stored in like Julian format or something is it's yeah I think it I don't remember the specifics but I know it took me like all of this code is just to get time in a usable format in a usable format for python so just something to keep in mind yeah I think that happens in a lot of other areas too in the software that just in the base code of hecz they they like to store date times in a kind of a number which I think is called Julian or something like that yep um and yeah it's unless you have that that conversion it's really tough to work with yep but I mean python does make it pretty easy this is I mean yeah is probably a way to write this a little faster but very cool well we only have a few minutes left and I do want to leave some time for some trivia here at the end but before we transition there um Chris or Isa is there anything else that you wanted to kind of add to this topic again you could probably have a standalone class on using python to process Raz results but anything important um to add well I I just want to say I I think anytime you have an idea automating a process in Raz whether it's you have you know 90 plans that you want to run all all of the plans and grab one bit of output from each one and store it in a spreadsheet you know you wouldn't ever want to do that by hand it would just be uh a nightmare and so writing that that to me correct me if I'm wrong Isa but that sounds like a very very short script to do something like that right absolutely very easy might take you you know obviously if you already know python yeah it's going to take you a short amount of time if you need to Learn Python then then you got to learn it but if you already know python I mean you're going to write that script in 5 minutes and and then you can press the Run button and go home for the day come back the next day it's all done um might not even take that long might not even take that long that's right depends on how fast your Ras models run um and yeah so if you if you think oh I mean if you have this thought in your mind oh my gosh this is going to take me forever to do this thing in Raz and there's a lot of repetitive stuff involved this is a great tool to solve that problem for you y and I think what I will add to that is if you have time or um want to play around with this even if you don't have an application for it right now I would encourage you to kind of have some building blocks set up um so for instance Chris your suggestion about pulling the same data for 90 different plans that would be like adding two lines here so once you have these Basics kind of save somewhere you can go in and make a lot of changes really fast and um pull a lot of stuff together so you're you start building your own library right Y and then you could call it like isaie or something right and share it with everybody yeah and there's a really good um Community out there of people who share their their p on code I mean that's part of python coding is sharing and borrowing from other people and improving on other things and and if you look hard enough you'll find there's a lot of Raz related um open source python code out there that you can get from GitHub or something good stuff well great conversation uh thank you again Isa for taking the time to to share some of that with us if you have questions for Isa or if you want some of this code that Isa developed feel free to reach out to her directly or um leave a comment in the YouTube channnel and we'll do our best to to respond to you um Isha I also don't know if you knew but you are the first woman that we've had as a guest on full momentum so uh congratulations on on breaking that long overdue barrier that we've uh uh accidentally employed on the podcast so we have a couple other gals that are lined up to to be on the program this year but you are the first so congratulations and well congratulations y I think yeah exactly and coincidentally I think uh we were told before this that it's International women's in engineering week is that correct inter International yeah International women engineering week so uh coincidentally uh but definitely worth noting out there um awesome yeah so what do we uh what do we got for trivia here yeah so I think to end the segment here just a a fun little trivia opportunity so we can each go through and ask our ourselves a question here uh no pressure to to stump everybody but uh I I'll go ahead and start um and then Chris we can go over to you in the Nisha if you wanna you want to close it out so um my question and no cheating here can't jump into the manual uh my question ready Hydra hydraulic computations through gated spillway that's the topic of this particular question so frame your mind with that okay okay um all right so um as you all probably know when the Upstream water surface Upstream of a gate is equal to or less than one times the gate opening the flow through the gate is calculated as wear flow okay so if it's equal to or less than 1.0 times the gate opening then it's it's got good as gay flow there is a transition zone between one and this upper limit uh where it's actually computed as wear flow and gate flow and the program kind of balances out and actually iterates and comes to a solution and then Above This certain depth uh it's uh the gate flow equations are applied and so again I I'll just read it directly from the manual and then your guys' job is to fill in the number so when the Upstream water surface is greater than or equal to x times the height of the gate opening the flow the uh gate flow equations are applied what is that number might be a little bit of a guessing game it's gonna be a it's GNA be a guess for me um because I don't know off the top of my head but Isa Do you know would be impressive so I am also guessing Okay well Isa why don't you start us off I'm going to guess 1.1 okay1 times the gate opens I I have a feeling feeling so I I have a feeling you're closer than my gas Isa and I don't know why I just have a feel should go closer to her guest then I know but I'm you can't caveat that because you're taking try to take credit for her guas and your guas I'm gonna play this trivia game honorably and go with my initial number even though now after Isa said it I want to change my answer okay but I was going to say 1.5 okay but I think that's too high I think that's way too high so the the answer is almost dab in the middle so it's 1.25 times the height of the gate opening is where outside the transition zone so by prices right rules I guess you win that one Isa because I went over so yeah and then the dam failed and the gate broke and everything yeah when it's in between one and 1.2 times the size of the gain the flow in that zone of transition the program comp computes the Upstream head with both equations and then calculates a linear weighted average of the two values um and that's an an iterative process so I thought that was an interesting one to sh that is an interesting question I want to ask you guys have either of you seen this happen on a real project this transition from free flow to gate flow I would assume it happened on the uh Carpenter project that we were working on right Isa yep yeah yep well I also feel like sometimes it happens I feel like there have been times where for like a lower flow I've gotten one number for the up water surface elevation and then for a higher flow it's been like really close or like you know it hasn't quite aligned as you think it would and a lot of times because it's transitioning between wear and gate flow um yeah but what I mean have you been out and actually seen this in person oh I don't think I have no so it's actually quite interesting it's a very it can be a very violent process switching from free flow to gate flow there's a lot of fluctuation and so it's good they use that transition because it's not just an immediate okay boom it's wear flow now it's gate flow it'll bounce back and forth you get a lot of splashing a lot of what they call burping sometimes where air that gets trapped gets kind of pushed out and um that that's usually when you have more of a tunnel Downstream of it but um but yeah it can be really kind of very um non-steady very um what's the word I'm thinking of um ununiform yeah non-uniform for sure but uh more time based just changing um transient that's what I was thinking of it's very transient a lot of bouncing around and then it'll all of a sudden get to gate flow and it'll just settle out and be nice and calm again so interesting yeah yeah all right Chris what's your trivia question for us all right I hope you guys get this right um because we deal with this all the time oh boy he always does but I want to know what the E and the L and the m stand for in Elm like as in shallow water equations DM for the equation set I know this but Ben you're you it's always hard for me to pronounce the e it'san legrangian um method yeah that's it is that what you were gonna say Isa yep she I was gonna pronounce it correctly yeah I think I don't know I've heard I'm actually glad I went first so I've heard ulian and I've heard oian I I've always said oian but I honestly don't know which is correct oian told me oil Arian oian Oiler yeah it's by Mr Oiler Dr Oiler probably and Dr lran or Dr lrange yeah I remember that name coming up a lot in like physics and class lrange yep well and Oiler too for that matter but yeah all right cool Isa close us out what's yours right Chris this was specially made for you because we spent a lot of time talking about this earlier this year um when Raz computes turbulence um it models it as a gradient diffusion and the diffusion rate of turbulence is the Edie viscosity uh there are two terms in the Edie viscosity equation what do each of them represent two terms in the Eddie viscosity equation um so there is the I wouldn't know this one there's the um oh gosh lateral and longitudinal turbulence right and then so that would be your that would be your diffusion coefficient both lateral and longitudinal are in that part of the equation and then the other one is the smarinsky right which is like the The Edge effects the turbulent Edge effects those are the RAS inputs that that's what sorry those are the inputs into the RAS model yeah but there's two that wasn't your question maybe that should have been my question well I guess I made this hard because Chris you said you were gonna try to trick us so I wanted to I didn't say that Ben said that well there's two terms I won't read them out loud because they make no sense but the first term is the vertical Shear component and the second term is a horizontal Shear component they sum up to be the Eddie okay okay so Ed viscosity the equation for Edie viscosity is made up those vertical and and horizontal Shear you say Okay components oh so water spinning this way and this way right yeah yep okay cool those were all all great great questions um that finishes out our our trivia segment for today uh so we are just about at time uh want to again thank Isa for joining us today to talk through our Ten Commandments of HC Raz modeling talking through hdf files and and using python to kind of maximize the utility of of those files and the utility of extracting results from them and then for putting up with our uh trivia segment which is always a little bit of an adventure so thank you Isa again for for joining us yeah thanks for having me thanks for co-hosting as always um this has been episode 32 of the full momentum podcast and we uh will look forward to talking to you all next month on episode 33 so thank you all for joining us today till next time bye everyone see you later

Channel: The RAS Solution

Views: 454

Keywords:

Id: 5-zNOo1g21g

Length: 83min 36sec (5016 seconds)

Published: Thu Jun 20 2024