NetBox Live Coding 2021-10-08

all right everyone looks like we're going live now there's always a bit of a delay so i get confused easily all right so my name is jeremy stretch i'm the lead maintainer and the founder of the netbox open source project um which you can see up here on my screen uh let me start off by apologizing uh before ahead of time because i'm still getting over a head cold and you can probably tell i'm a little congested and uh you might hear me mute occasionally just to uh cough uh as this eventually makes its way out of my system uh so today we're gonna be looking through some issues uh seeing what we can do to develop that box and i thought it would be cool for a slight change of pace we've done uh this is the fifth one of these i think we've done now uh or are doing uh for a change of pace i'd like to run through releasing a net box release actually just to so folks can get a sense for that and see what that looks like um hopefully that'll be of some benefit i'm not sure uh but you know it's it's i think it's always kind of cool to see how people you know open source maintainers actually manage their projects how they actually go about pushing something out to the public so hopefully that'll be insightful all right so let's get started here let's go through the issues list now i don't have a whole lot of issues actually on my plate today uh this week i've actually gone through and managed to reduce the backlog of open issues down from i think we were around 160 open issues around the time of the version 3.0 release all the way down to as of last night it was 120 and of course we got about another you know five percent bump overnight um but that's fine uh so we can look through some of these maybe we'll just uh talk through a couple of these and see what makes sense so uh here's a feature request which is open a few hours ago and this is to add uh vlan and range data uh when looking at an ipam roll that's interesting so basically the idea is that you've got [Music] vlans and and ranges and so forth so assigned to prefixes and prefixes are assigned to roles or vlans go to or assign directly to roles rather so that's one thing we can do i this probably needs a little bit more thought or specifically around how to do that in a way that's um performant and that's not gonna um just like absolutely take forever to to calculate and we actually have some other stuff that's that's come up recently that's um around uh like calculating prefix uh and range utilization uh because right now in that box does all that on the fly so if you are you could come up with an example here so let's pull up the prefix let me log into my my demo here first let's go to prefixes so you'll notice that this takes a couple seconds now this is going to be a little bit slow because i'm writing it locally it's on you know it's running with debug enabled and all that but when you see the utilization column here this is actually being calculated on the fly uh so that's being done at the database level um which you know of course uh imposes some overhead if we get rid of that utilization column let's see if we just remove the you'll notice it loads a bit faster that's because the performance overhead so one of the things that's been proposed recently is to start caching that that utilization data on the objects directly in the database so to pre-calculate it so we're not doing it at runtime all right query time rather which definitely sounds like it'll work and would certainly be beneficial because um then you could even like order you know order columns by utilization and things like that so that'd be really cool but uh a lot a bit more thought needs to go into that and to ensure that those cached values are in fact accurate right that's always the problem with with caching that's a big part of the problem or part of the reason why we ripped out caching automated query set caching in version 3.0 is because we were running any constant issues with it just being inaccurate uh incorrect and not being invalidated automatically as it should be so that's something to think about so i'm not going to move anywhere with this just yet let's see here's a bug it's uh all creating devices does not check rackspace within its own context so that's interesting uh i took a peek at this earlier it's basically saying if you create a you know 42 42 unit rack and create a two unit device type and then you're posting some devices in there and it looks like you've got one in position one and one position two that uh you're you should expect a validation problem right because if you have two two unit high devices you obviously can't put one in unit one one unit two that's overlapping in u2 of the of the device of the rack rather excuse me the observed behavior here is that we get a key error all right well that's not great uh and of course it breaks svg rendering as as you might expect all right let's play around with that a little bit let's see what we can do um cool so first off let's try replicating oops let's try replicating a an api request to do that so i'm going in here and this is my local development instance and i think i've got oh that's not what i wanted i think i've got an api token set up already so let me copy that and let me i got a scratch pad here that has some api calls on it let me just grab that so we can get our api token ready so remember what we're doing with this bug report is uh first we are seeking to reproduce the bug to verify the behavior that's been reported now this is this is a great bug report uh it's it's very clean saying hey this is what you need to do and here's a sample api request with the data that i'm passing it makes sense uh don't by the way the reason this is red is just github's formatting for uh json obviously it's not valid json data but that's fine um that's more that's exactly what we need to know so let's get our dsim devices so let me uh here let me bring this in the window so you guys can see i'm sorry uh let's see it's just something i grabbed from earlier so we are going to send it some data to the decent devices api endpoint that's good um oh you know what so first let's uh let's go find a 2u high device actually let's let's make a rack um yeah when i do this i'd like to start with something clean so we're just going to call this uh it doesn't really matter where it goes we're going to call it my rack one and 42 ui everything else should be fine we'll create that let's go see if we have a device type in here that happens to be to you high here we look at the height column um here we go 48 pair fiber panel sure that sounds good right let's get that as uh its id is 11. uh so we need to create two devices in the rack of this this device type so let's go find our scratch pad actually let's get let's copy this i'm gonna put that in here and hopefully that'll be okay uh device roll we don't care device type um the id of this is 11 so it's device type 11 so we're going to use that when formatting the api request so i'm going to say device type 11 site uh where do we create this that rack let's go back to racks and we'll find um the add a bit of delay here by the way is also from uh utilization uh just as we talked about with uh prefixes same thing over here for space and power if i get rid of those columns it goes way faster um we might not make those default anymore not thinking about it anyway i had to gross um so let's find this rack all right so the rack is 43. uh it's site is akron which is id2 uh okay so let's find our so let's put those values in here so the site is two the rack is 43. uh okay that's fine let's um do the second one as well site two rack 43 okay so we are going to be creating two instances of the same device type of the same role in the same rack the same cell same face two different positions position one and position two this shouldn't happen this should or rather no buck shouldn't allow us to do this because they are both to you high devices and will overlap so let's grab that and go to a console and paste oh would i mess up line one column two oh um i'm actually wondering if i it's always fun troubleshooting these things live on camera let me see if i can just maybe just all need to be together oh wait is it because i have a extra i think that's it all right let me see hopefully i fixed it no extra data sorry folks i probably should have had oh here we go i'm missing missing an opening something happened okay let's see that worked um okay so we've got two devices created we've got rack uh let's see rack 43 is my rack one yeah we got that and he is in position one and the second device here is in uh position two okay that seems to work let's go check out the rack well it broke something because we can tell that the front elevation does not render the rear one does but the front does not and i bet if i go to my ide where i have the surrounding yep here it is um great so interestingly the api has no problem with it um i think i i misread his uh or her bug report um initially right so it does actually okay yeah so the api returns a 201 meaning the devices were created as exactly as we told in that box to create them however obviously that should not have been possible because the devices are overlapping um great and when we and the this is the key error here so this is what we're seeing when we are actually attempting to render the rack elevation uh the way this the way this works here is when you're looking at the uh netbox ui it's rendering this this whole page and then through two separate api requests it is pulling down these svg images uh those can be pulled directly if you click the button there so this is this is all it's doing he's taking that it's embedding it in the page but obviously doing it for the front uh does not work as we can see here this will yeah there we go that pops up the debug message for us um as you'd expect so we've we've got kind of two bugs this one this key error actually isn't a bug this is symptomatic of invalid data right the problem is that we were allowed invalid data to enter the database to begin with so that's what we'll have to figure out now the way um netbox does this normally when you add a device through the user interface you're adding one device at a time um although it occurs to me that this may or not we may have the same problem with importing devices as well but that's something for me to check out later um but when you're importing them one at a time it'll import the device to save it and then when you go for the second device it will check for any existing devices uh for any conflicts right so so physical conflicts meaning um you know something different rack units or overlapping rack units rather that's not happening here because we're importing both devices at the same time so both devices are being checked against what's currently in the rack which is nothing it's an empty rack so there are no validation problems that's actually a tricky one i'm not sure off the top of my head how we can how we can really solve that um if we look at gosh let's look at the model uh i think it's under it's on the device model where we're doing that because i know we have a lot of validation logic in here all right so here's what we're cleaning that's not really gonna work that was it that's actually a tricky one that that we're going to have to figure out um gosh this i picked a bad one to do on a live stream because that's going to be a little involved we're going to have to dig really far into the logic that that we're using to validate these things but here's where you can see some of that right so if you're doing one one device at a time this is how that's happening yeah here's where we're validating rackspace right so it's it is going in there um yeah the problem is just that you're we're validating the way the api works is we're validating each of those devices those json objects really uh independently uh of of either getting created so trying to do it at the same time doesn't doesn't quite work out we might need to come up with something kind of clever for that um i don't i don't know if the answer there is to uh if we can cash like cash a list of uh you know 2b occupied units in the rack um all right well at the risk of the just because i don't want this to get terribly uh boring um let's let's punt on that one for now i apologize for that i was really hoping i'd be able to come up with something for that um but you know what we were able to replicate the bug and that's half the battle sometimes so we're gonna go ahead and mark this as uh needs owner so what that means is basically hey someone someone took a look at this issue everything looks good there is work to be done um and the work can be done meaning you know we don't have to wait for a new release or anything obviously it's a bug should be fixed um and it's it's free for someone to pick up so maybe if you're out there watching and you came up with a maybe you're screaming to yourself hey jeremy just do this you know by all means feel free to volunteer if you if you have a good idea for that um if not i'll go back and check that out later um let's see we've got a couple other issues here i apologize because i haven't had a whole lot of time to look through them these just just came in overnight um some of them are responded to uh trying to find something that would be good to do on the on the call here i know one that we can do let's see let's go find this is actually a really old one um somewhat sometimes what happens is you know i'll assign something to myself with fully intending to work on it and uh it gets it gets you know just just um de-prioritize or you know keeps getting pushed to the back burner in the back burner the burner behind that um and then unfortunately i'm trying to get better about this but you know sometimes because it's been assigned to someone and has been marked as accepted we don't see it as an outstanding issue so we kind of work against ourselves a little bit in that regard as you can see this one's been around for a while this is a housekeeping item so it's neither a bug report nor a feature request it's actually something that has just pertains to the code base itself you could argue this as a feature request it's basically to add a general purpose search filter so like a queue query parameter for all objects in that box uh there's a question uh do you see a future where nutbox can be used as a documentation management tool for firewall rule sets good question um definitely i mean that's that's definitely something that we can do and i think would make sense to do the biggest impediment to that i think is figuring out exactly what that looks like so when we talk about firewall rule sets there's it would be very easy well fairly easy to come up with a model that would uh basically just accommodate like the typical five tuple rules for firewalls right so that's um protocol um address import or you know source address or support destination address destination imports like five five values that you really have to track um we can do that we can like that's that's all pretty straightforward we can come up with models for that we can introduce maybe that would be a separate app or i don't know what um and that's definitely feasible uh it's just the the um the whole process of getting there and and first figuring out what that looks like um that's the kind of thing like you know do we is that is that enough did what what kind of relationships do we have to track are those rules you know should it be free form or should they be relate like when you specify a prefix for example should that be just a prefix value or should that actually point to something else in that box right because there's there's huge trade-offs there depending on which which approach you want to take so that's just you know an example that's the kind of thing where i think it would be best initially conceived as a plug-in so where you can extend that box leverage netbox where you think it makes sense and kind of have a bit more freedom around fine-tuning it and figuring out okay what are my workflows what is my data model what does all this look like um and it's a lot easier to manage something like that as part of a plug-in because it becomes decoupled from the uh the core netbox release cycle meaning if you want to change an api in your plug-in you can do that with complete freedom depending on you know your your own uh plug-ins release cycle so for example if we put something in the box core and we want to add a field to a model right now that has to that change needs to be deferred till we release version 3.1 right and then we change it and then we decide oh we want to change something else oh that has to wait for 3.2 right which means you're talking about the span of months typically three four five months you know between these major releases whereas if you're doing a plug-in you have much much more control over that becomes a much more closely iterative process you can decide you know what what your release cycle is for the for that plug-in and then if we uh if we find a plug-in that has become mature and robust and tested and and if the author is interested we can look at bringing that into that box core so that would be my my suggestion there um sorry if i went on too long ended up a tangent there but then i mean it's a great question it comes up all the time right at what point do we at what point does it make sense to extend the scope of that box beyond what it already is um we're certainly in a better place now than we were even six months ago and just in the amount of features we've added uh you know with a ui overhaul out of the way gosh that was a huge endeavor i mean big shout out to uh to matt love um who led the the the charge on that um now with the refreshed ui we've got some other you know major features out there and uh we're reevaluating you know stuff for for three one three two three three at some point we'll we'll get those milestones together so um we're getting closer to the to the time where we have those discussions and i think we'll see a lot more of that next year so anyway getting into uh this issue uh so this is a housekeeping issue it's basically um the goal here is just to add a general purpose search filter so if you look at like the sites list for example uh and i just want to look for i've got like gosh you know i know there's a site in camden somewhere i'm gonna look for camden um you notice the url up here it's it's it's appending for the query parameter it's got a question mark and it's appending the the search parameters so q equals camden that's just the general query right i don't know what the uh fields are for this model offhand but let's go find out so like when we search for sites like i can search for um uh maybe it's in joyzi all right let's see if there's a that doesn't work um i'm trying to think of examples to search for right now but basically let's go pull up the filter so when you look at the filter set for the device model it is under here and go back up here and find it uh device filter set so one of the the uh methods on here is queue or or i guess object or properties rather is q so this is a character filter um so this is being read from you you can filter by you know manufacturer slug id device type role platform all this stuff but q is a special kind of filter it's called a it's a method filter meaning that it's going to actually take the input value and execute a method on the filter set class and we find that down here this is a very big filter so down here so basically it's saying i'm going to take this value and i'm going to construct a django query set with that value filtering for many different things so i'm going to check against these are separate queue or query objects for django underlying so it's basically going to check for the presence of that value upper or lower case anywhere in any of these fields so name serial inventory items if i have an inventory item with that name uh asset tag or the comments so for example you know if we go to uh gosh where's um i just completely lost track of what i was doing so let's go up to sites and let's say that we know oh i just realized i'm sorry i'm looking at the device filter i don't i don't know why i was looking at that one it's the site filter oh gosh uh so the site filter same thing though right so the query um query here is calling the search method so this has a different one right here we're looking for name facility description somewhere in the address the contact information right anything in there it's going to look for um so if i you know if i put myself as the contact for this site just put my name in there i can type there we go i'll save that and now if i go to look for sites all right it'll find it'll find that because it's looking it's hitting on my name in the contact field so the idea behind this issue uh number was it 2101 it's a fairly old issue is to ensure that that ability exists for all objects in that box i was looking through this earlier today and the good news is that it does for for most objects it's already there but there's a few places where we still need to add it so one example is these device type component filter sets so when you're looking for so a device type component is basically a component so an interface or console board or what have you on a device type so if we go to look at device types um this one should be good should be a good example i mean uh so here's obviously that image is not correct for this that was me playing around with image attachments um but here we've got all the different components right so it's got 51 interfaces defined it doesn't have anything else um we should add like console ports and power ports to it but basically when you're going to look for these through the api uh there's no dedicated ui view for these just yet although this is something i'm looking at adding pretty soon but you can add you can check these through the api uh but there's no way to filter them right now other than by the exact values of their fields right so if i was looking for uh let's see let's go api esim uh and let's see i wanted to just find interface templates all right so these are all that i have defined for all the different device types but i want to find see if there's anything specifically for the matches like net one i don't know uh oops so it would be like q equals net one uh nothing happens all right so i still have a count of 200 it's returning all of them it's not even looking at this q field whereas if i wanted to see uh give me everything with this name with the exact match i can do that at least i should be able to match on the exact name and i do right here's my interface template that has that exact name now if i do something like zero zero zero i think there's a few copies of that oh i guess not um but that requires me to have the exact name right if i wanted to omit one of these it doesn't work because there's no exact matches for that i need that q filter to look for that that to provide that sort of flexible you know lower case any you know partial match um so let's let's look at doing that let's go let's see before okay so for this we need to find first find our filter set let's start with interfaces because that's what we were looking at so interface filter set okay uh so here it is this i'm sorry not interface but rather interface template filter set right because we're dealing with device types uh you first you know the general flow is you create a device type within that device type you create component templates so interfaces console boards and so forth that are assigned to that device type then when you go to create a device those templates get instantiated to actual device components so an interface template becomes one or more interfaces or becomes one interface once you go to actually create a device so let's look at interface template filter set um so we need to add a cue method to this basically which we can go let's go grab some of that boilerplate this is all we really need to add q is a power filter we should probably move this actually onto like one of the base filter set classes uh because it's repeated a lot and that's just part of you know general cleanup that has to happen at some point when someone's thinking about it and they're not doing a live stream so you'll notice the interface template filter set here inherits from change logged model filter set as well as this device type component filter set which from which all these other template filter sets inherit as well from ports rear ports console ports power reports so what we can do is define this actually on the base class and uh it will just be inherited by all the individual filter sets for the different components so two parts to this one is we just need the to set up the character filter so that it can it'll start looking for that query parameter and the url the you know the question mark queue equals uh and then we need the method to actually tell it what to do so i'm just going to grab this i'll start with that put that down here so it's basically saying okay when you see q in a url as a query parameter run uh take that value and feed it to this method which is down here that's that search so the method is going to take three values query the current query set the name of the filter which we don't care about and the value then it's going to say okay if the value is empty then ignore it and if it's not we're going to apply it as a filter so what fields do we want to filter on for device type components so device type components are pretty simple actually they don't have a whole lot there there is a name and a type pretty much for most of these and you can see these here in the filter sets so name type uh feed leg um interface template filter so i don't think it makes a whole lot of sense to search on the type because typically if you're looking at type you want to find a specific type because it's a choice field it's one of those things where you you know have a limited set of choices that are valid values for that field so for now let's do name let's just limit it to name because we know that name is at least present on all of these you remember we're inheriting from all these so unless we want to override that on a child class we want to make sure that the parent class is the fields that are defined are supported on all the inheriting children so we're just going to actually make this pretty simple instead of using cue objects we're just going to type name under i contains equals value all right uh this is a bit of a django orm filter vernacular that basically says okay we're gonna look on the we're matching the name field and then uh the dunder here the double underscore is saying we're going to apply a filter to it which is i contains that's going to be case insensitive contains so we're going to apply a filter to our query set that matches on any name containing any case combination of this value fairly simple right should be good um cool so this is this is added to the um interface templates to all these component filter uh all these filter sets now um and that's actually all we needed to do if we let's see if i lose focus here yep it'll um once i um once my ide loses focus it detects the change in the file will automatically restart the uh the run the development server that's running behind the scenes here so if we go back here to our example i bet not name but if we do q now it works now it's only finding two it's finding uh this one zero zero zero and it's finding this one zero zero one that was it that's all we had to do to to make that work and uh if we go to just to make sure it's working everywhere else let's go to console port templates and let's say i want to find uh q equals something with the word cons in it there you go there's nine that have cons in the name right if we if we if it's unfiltered you have we see we have total um we have 11 of these total if we filter it by cons we only have nine if we filter it to like well obviously it's gonna work um con zero maybe nope nothing that but if we do a space we'll get that one that has the con zero and the name or if we just look for um what if we just want to find anything that has a zero in the name right that's how we do that uh so hopefully i mean that's i don't want to belabor the point but you guys i'm sure you understand how the filter sets work um that's really all we need to do for this part although it occurs to me looking through this that each of these children these child classes is inheriting from both change log model multiple filter set as well as device type component filter set that's unnecessary because this one device type component filters that should be able to inherit directly from change log model filter set i believe the reason it's not is because we when it added this at a some point you know the more recent release and it was added in bulk uh so we just didn't detect the opportunity here to uh to clean this up a little bit but let's go ahead and do that since we're in here so i'm going to replace the stock filter set that comes with the django filters model or excuse me module here students to extend that from change log model filter set and then update these to both uh to all just inherit from device type component filter set and that just cleans things up a little bit um yeah from port airport there's like eight or nine of these device bay template yep that should be it okay um just to double check that's not being used anywhere else device type component filter so let's find usages of that uh yeah we just see the ones here in this file so that's good all right um that that actually works i think that's pretty good uh what we should do now is under tests go down here and find [Music] uh test filter sets so this is a set of test cases where we are actually testing the filtering for all these models so maybe a few of these where we need to do it let's see the interface template test case you know i don't know if we have a general i don't know if we have a general pattern yet for testing the q filter i don't think we do actually that might be something that we need to add um in a broader sense so you'll see like here's an example for uh gosh what is this the rack test case so for the rack filter set for every field other than the queue uh the queue parameter apparently uh we're testing so basically okay let's test the name filter the facility id filter the asset tag filter each one of those gets tested against known values just to make sure it's being applied correctly we don't actually have that for the q values which is a little strange we probably should um you know for the sake of time though we're already more than halfway through this i'm going to admit that for now um which is okay because i'm not actually closing out this issue just yet anyway i'm not going to go through let's pull it up so you'll notice my notes here said basically this is just one of the several that i identified that still need to be worked on so right now i'm just gonna push this change for the device type components and then at a later date i'll go through and uh of course that was my intent back in february of 2020 right uh i'll go back and add the remaining ones hopefully i don't it doesn't get lost this time um so that's good enough for now what we can do is let's see let's add our actually let's add our changes okay not much changed obviously um cool so you see the changes that we made we've uh are now inheriting uh from device type component filter set from change log model filter set we have our q parameter up there with our search method and we've just updated the inheriting classes of our child classes so that's it i'm going to go ahead and commit this this is a you know reasonable partial implementation of this normally i would try to get everything done but it's actually going to be somewhat substantial amount of work so it makes sense to break you know across a couple commits and nothing's breaking because all we're doing is now adding this additional functionality that frankly should have been there in the first place um so it's a pretty low touch change so let me find uh what's the name of the the number of the issue 2101 i laugh because we're in the mid 7000s now let me go ahead and change and uh uh let me apply that real quick hey come here uh number 2101 i'm not going to prepend this with closes or fixes because it doesn't it doesn't close the issue it's just related to that issue but we're not done with that you just yet so i'm going to say implement cue search filter for device type components that make sense right cool i didn't run tests on this normally i'd run tests although i don't expect anything will break because this was a very very minor change um so in this case we'll just let's see yeah oh of course this had to say something right so let's see we'd let ci handle that for us what happened there oh you know what am i on the wrong branch oh gosh yep whoops oh gosh all right i was doing some work on the feature branch earlier and hopefully there we go all right i ran into a bug there that was uh has been fixed in the develop branch but not in the future branch that's why i saw the error message okay let's give that a second shot here double check everything live live coding it's always fun all right diff still looks the same right yeah that looks good okay like paranoid now that i changed something i shouldn't have okay there you go all right i'll try that again i love that as i was talking about how i don't need to run tests a big error popped up i love it all right uh so that'll be fine tesla will run obviously once again are you kidding me what oh man maybe that's not the bug i thought it was well let's see get fuel parts model made a num type hmm all right let's see let's try this um what are we running that's actually having a problem with that that's the management command right or the uh yeah let me try something the command doesn't actually it's not gonna do anything i'm just gonna see if i can replicate that bug okay what is the problem then just checking for missing migrations right that should be okay well it's clearly a problem with something that i had here oh you know what that's probably okay well that was to leave me then i think um there was actually a reason we needed to have that done oh right um that's why it's not redoing the error already that's my my changes there okay i see what the problem is now i think um we're we're having an issue with the um method resolution order or something to that effect where because the way the filter set class works it does it can't actually um this can't inherit directly from change log model filter set that's fine here's what we're going to do [Music] put our changes back there and we're just going to back these out nope i think i lost my uh i kind of um shot myself in the foot when i changed the uh the branch there that's all right here we're gonna do this just because we're doing it live and i need to move on i'm gonna i'm gonna copy these for now folks i'm sorry um i'm just gonna reset these start over and then unfortunately there's a pretty minor change so i can do this on the fly we're gonna put the q filter in there we're gonna put our search method in there we're gonna leave these alone because obviously that's that way for a reason i don't know maybe i should put a note in there but uh that's okay we'll leave that as is let's go double check real quick make sure our filters and everything still work can i still filter by that and get oh um didn't give it enough time to reload can i still filter and get through no what is going on i was probably hanging on that error i would like to say it's not normally like this but it is that's this is you're you're see what you're seeing is is representative of how it normally worked oh gosh all right there we go all right cool so our filter's working we're good uh the only thing different now is that we are not condensing the uh class inheritance cool all we've done is add this it's getting um it's working for all the inherited models and everything so that's good oh gosh all right let me go ahead and commit those if we look at the error up here we can actually see you notice that when it's when it's loading this class device type component filter set the next problem down here is when it's actually calling the new method on the class which we don't normally deal with um because of the way the filter set class is being created it's doing some kind of i'll call them magic or call it magic you know behind the scenes that actually um take those declared attributes on on the class and turn them into [Music] what they need to be it's the same way like django models work how when you put you know like a name on a site model for example it becomes a field in the database sort of i don't want to get too far into it but um basically that has to happen in a certain order and by trying to condense that or to inherit from our own custom changelogged filter set uh class that introduced obviously some kind of problem that um we can dig into but it's not you know not a big deal okay that's done we have that change um cool don't have a whole lot of time left but i think i would like to run through actually uh releasing a net box release so if you go over here and we go to [Music] docs release notes we see that we are at current releases uh 306 and we've got a number of things here that have been fixed recently for 307 and there's one bug in here actually it's uh um this one 7474 this is actually kind of a big problem right now um there was a regression in the prior release that causes an exception to be raised now when trying to open a custom script or a report obviously that's that's um pretty breaking for a lot of people so we want to go ahead and push this out even though we had a release uh was it wednesday i'm going to go ahead and push out this this next release just to get that taken care of so that people don't have to uh deal with um patching it or anything so what does the release look like in that box uh here we go let's we actually have it documented so check that out um if we go under development release checklist i've gotten all the steps written down here mostly to remind myself what i have to do every time i push out a new release so the first half of the document is from um technically it's a minor version bumps so going from like 3 0 to 3 1. uh we can ignore that because this is just the patch release so we'll go down here to the all releases uh first thing we want to do is update our base requirements so let's see what that looks like uh that's going to run uh this is down here basically you know create a virtual environment that can actually be updated since we're always working in a virtual environment now anyway but we define so every release defines a set of requirements that's here in requirements.txt these are the dependencies that we rely on to run netbox basically what we're going to do is update these by referencing this base requirements which is basically everything without a pinned version in here and explaining why it's in there and you know where to find out more information about that package so first what we're going to do is upgrade all the pin dependencies right now we're going to do that by running pip install with the upgrade flag and reading from the base requirements file everything's pretty much up to date oh it did find a new package for um make docs material so we can bump that from 731 to 732. make docs material is a supplementary package that we use to render the documentation that we actually happen to see right here this very lovely documentation it's all made possible by make docs materials so quick plug for them and a shout out because they uh they do some really good work the maintainer of that cool so we can upgrade that uh ignore this down here this is actually a workaround for a different blog i need to look into at some point cool so there's the material package and then next up on our checklist so we've got that upgraded uh verify the ci build the status so let's go back on github granted we did make one change i don't think that's going to break anything but let's just make sure yes the develop branch is currently building correctly and if you're curious you can look at that and it's all managed through github actions um cool so that's all good uh we need to update the version and the change log because right now everything says 306 so we have to do this in a couple places first off we have to put today's date october 8th uh here in the changelog for 307 and then we have to go to under settings and we no longer are on dev we are on 307 and up here in the github issue templates this is kind of a pain but when you go to open an issue on github it's actually reading from a template here that's managed as part of the repository and we just want to upgrade update the placeholders for the version to the current version uh the reason for that is we want people to know when they go to open a bug report or a feature request uh if they are running 3.0.4 for example and they get to open an issue and it says that it's 3.0.7 hopefully they'll think oh wait maybe i should upgrade first and see if something's already been addressed that's the hope anyway okay so that should be everything right we got uh the version and settings.pi we got the issue templates and we've got the uh date and the release notes uh so i can go ahead and commit those to the develop branch there's all my changes nothing super interesting normally if i was updating something like uh you know an actual python package i'd i'd be more thorough with running tests and everything but i know that makeox material is for the documentation build so i know exactly what to expect from that um famous last words right all right uh and then i'm going to give this a release seven oh geez three to oda seven all right excuse me uh so this will run through uh the same checks that the pre-commit checklist that we uh had in the prior commit all that's doing is looking for you know doing some pep 8 linting looking for any uh sku data migrations and uh checking the um compile javascript and all that good stuff so that's good to go and uh just to triple check we are on the develop branch so we are going to now push the develop branch into github and i never have never seem to have my github ready to go all right so we're going to push that and now if we go to oh i'm always blown away by how fast github is with this given given the you know the size of their user base and the complexity of the application that they run but if we go over here and now we can see that uh release 307 has been pushed to the develop branch right so we've got a few branches here in that box um here branches so there's typically going to be three branches that kind of hang around all the time there is master which has the current stable release of that box so if you are downloading netbox 3.0.6 right now you're getting a pristine copy of the what's in the master branch there is the develop branch so that's always the next patch release so for example we just we're pushing out 307 everything has been happening in the develop branch so everything you see in the change log basically right here this this is the difference right now you know in summary between the master branch and the develop branch the develop branch has all this stuff done whereas the master branch doesn't and then there is the feature branch so the feature branch isn't always there but it represents the work being done on the next minor release so anything that's going into netbox version 3.1 happens in the feature branch um the reason for that is to keep it kind of you know separate to make sure that we're we have a base off which to work with that and we can still release things under the the patch you know we still have three ox releases while we're working on 3.1 um what you know what causes uh a change to go in 3.1 versus you know 3-0 patch uh any kind of api changes uh model changes any obviously anything that's breaking has to go in there um and you'll see those tag as well right so if we look at what's happening for you know like what's coming up in 3-1 now well we just look at the 3.1 milestone um so here's you know a couple things haven't done already here's everything else we have planned some really cool stuff in there i'm pretty excited about uh like what's some stuff we've done already oh we oh okay so we're bumping like the minimum required version of postgresql uh from i think it was 9.6 to 10 right version 10. um the reason for that is actually one of these other features which is adding a um a wwe n fields like a fiber channel worldwide name field to the interface model uh because that requires an uh it was a 64-bit like eui field uh which is different from a 48-bit mac address and to support that the 64-bit field we actually needed postgres 10 because you know other than otherwise we'd have to create it ourselves and it's easier just to go to 10. um 9.6 is getting pretty close to end of life now anyway i think uh so anyway this is the reason i'm talking about this is because right now ci is running on the put the uh excuse me the change that we pushed right now it takes about i could think about eight or nine minutes it kind of depends so you know depending on the time of day and how busy github is and all that um so let's go back to our documentation here and it says okay we've we've pushed the changes ci is running now let's go make a pull request so let's go on here and we go to pull requests new pull request and we're going to merge the develop branch into master there we go so now it helps gonna pull down those changes so this is a summary of all the commits that have happened between um basically you know the 306 release and now uh so not a whole lot again there hasn't been a whole lot in this uh release specifically but a couple issues you can see have been fixed uh this all looks good so let's hit pull excuse me create pull request and then so we're going to title the pull request release v307 and for the summary you notice that there's a there's a template in here right so typically a pull request is going to require you to reference a specific issue and then describe what you did in it because this is a release and we're not actually fixing a specific issue we're just going to include a list of the um or the most recent snippet of the changelog so we're gonna put that in there let's change these to the level two headers there we go because that's what it'll look like so i'm gonna go ahead and create this pull request now i can create this pull request while ci is still running those are two separate processes uh i don't have to wait for i mean ideally you wait for ci before doing the pull request but it's okay uh it's not gonna interrupt anything if i do the pull request first uh ci is gonna fail it's just gonna fail doesn't really matter if there's a pull request created or not hopefully this won't take too much longer because i'm definitely not going to merge this until ci is completed that's it so basically right now we're just waiting on ci and i'm going to merge the pull request once that happens as soon as that pull request has been merged the new code will take effect in the master branch the master branch becomes 3.0.7 and it shouldn't change again until we release 3.0.8 via another pull request through the same process but once it's done to make it accessible uh we need to tag a release off of the master branch so that people can reference it so if you look at code over here you can see that we've had a total of well we've got 203 tags something happened with github recently because i think we should actually be at like 203 or 204 releases something like that i'm not sure why some of those have been emitted it might have something to do to do with the tags and i do know that they changed recently the way tags are assigned to releases but anyway um each one of these releases here is as you know exists as a tag so once that happens i'm going to create a new release off of the master branch and i'm going to call it 3.0.7 with today's date and then that will be accessible as a release just like these are so here's like 306. here's three or so or you know three or five three or four all the other it's predecessors uh have i looked at python 310 already i've not i've been like i i kind of try to keep up with it a little bit but honestly we just moved off of three six we were just now 37 has been out for i don't even know how long number of years two or three years now and um with netbox 3.0 we just dropped support for python 3.6 which is actually end of life in december it's got like two months left on it um which reminds me that you got to get christmas shopping huh uh so you know the the newest the newer stuff in python while well it does look cool um i know a lot of people are excited about like the much more human-friendly error messages than 310. um it's i try not to get too excited about it because i know it's going to be years before i can even really use it realistically um that said i do know that django 4-0 which currently i think they just came out with an alpha release for it um that drops python three seven i think that requires three eight actually or no maybe i just dropped three six i don't know um so my point is obviously there's there's you know a delay between when the shiny new python comes out and then when you can realistically use it in in the real world now if you are if you're a smaller shop or if you're doing you know um completely closed source proprietary um tooling you might have more flexibility around that but you know where it's open source we try to support you know order python releases as much as we can um which isn't to say that you know three like 310 obviously would still work barring any breaking changes in that release um we we're just prohibited from using any of the new features in it like um was it three eight or three nine introduce the walrus operator for um you know assignment on comparison uh which which is really cool and i i know that we'd be able to use that right now but we just can't adopt it yet because we still have to support python37 until such time where we decide not to so um kind of a long-winded answer but yeah i've looked at it but try not to get too excited about it myself oh ci's still running all right let's see if we can find out what's going on with ci because we're almost at time for the stream oh why is it running twice oh it did it did create a second uh i think i think i told you guys wrong earlier it did it did create a second ci process just for the pull request i didn't think it did that okay so we're probably running you know two jobs half as fast you do notice actually on the topic of python versions we do run separate ci builds for three seven three eight and three nine so those are the three python trains that the box you know officially supports my guess is that if you were to run it under 310 it would probably work fine we don't do anything crazy with python we're primarily you know depending on django for all that and my stream just cut out didn't it oh oh man did my stream cut out i should still be working i'm not seeing it anymore okay sorry about that um my stream died there for a second all right still waiting on these tests to finish running let's go back to pull requests here yeah i'm still running okay all right so we've seen some of these okay this is this is what we're looking at all right so we've got different different tests for both uh you know the push process and the pull request process these are these are going to be fine um so the push is already completed that's the one that was running you know when from when we we pushed the um the commit earlier and then this is just the pull request running so i think we're going to be good with that i'm going to go like i don't want to start the release yet until i merge it all right but that's okay so anyway uh we'll talk through the rest of this real quick once we go to actually create the release that's just the process on github you know it's basically tagging a um a commit with a specific commit within a branch as a release and we're just specifying a tag targeting the master branch again that being the the stable um the stable branch and giving it a title and then once that's done it becomes accessible and you can download it as a release and you can check it out as uh even using git so you can do like uh you know get check out um if i do like v two or what three to o.4 if i wanted to check that out i could right to check out a specific release from a previous point and get that's an option typically though we just work you yeah for development purposes we work in branches but what's nice about that is if you're in production and you're using git to update you can very very easily transition back to a release of your choice without having to mess around with moving files and so forth and that took about eight minutes so shouldn't shouldn't take too much longer here tests are running i have a couple minutes left i should have gotten through this a little bit earlier i think i got derailed a little bit by my uh my attempt to prematurely uh optimize class uh class inheritance earlier but yeah so here's what my my inbox looks like if you're curious we uh we've seen a tremendous number of issues new issues come out lately with with that box and um that's not a bad thing you know so issues are they check bugs they check feature requests um a lot of times people will open issues just to start a discussion um we do ask that if you are just if it's more something open-ended or you need assistance with something github discussions are actually the best place to go for that um and that's where you know you'll get help for uh different things or if you just have an idea for a feature that you know isn't quite fully baked you just kind of want to get the community's take on it or you know some input from a maintainer by all means open a discussion and uh just throw it out there so i saw this like this is another one probably you know static media failure i think it's probably not a bug right this is probably an installation issue because this the steps to reproduce are just installing that box and something something went wrong that's not reproducible right so i'll deal with that later uh let's see should be another couple minutes i'm sorry over time i want to just finish this so that if you have to drop by all means i won't be offended um but these sessions do get recorded uh so i want to be able to point someone here later uh to see the whole process if they uh if they need to uh cool so yeah let's take a look at what else we have coming up for three one um you see some of these are fairly old right it's like the concept of contact objects when you like i said we're in the with the mid mid mid to high seven thousands now in our issue counts uh a lot of these are you know the three thousands the four thousand so these are these are pretty old right like this one was open july 2017 uh netbox was released as open source in june of 2016. so to give you a sense for how uh how far this how long this has been around um and this is basically just uh creating a modeling contacts like for sites and so forth as separate objects it's one of those things we you know it's a always a discussion around priority and and ease of implementation and value to the user base and all that but we're finally getting around to it our oldest issue is actually number 51 and that is uh half height rack units which i today still have not figured out how to do correctly so that might hang around a bit longer all right ci is finally done on our pull request let's go back there check that out everything's green uh so we can get we are free to merge that to master on merge it and now master is version 3.0.7 that's live so the last thing we have to do is create that release so i'm gonna go to code uh releases draft a new release so we have to create a new tag called 3.0.7 and we're going to of course release that from master and we're going to call it 3.0.7 and we're going to put the date on there i like to include the date so just so that people have you know a frame of reference for how older their release is and again we're just going to copy and paste the release notes in here to show okay what's been fixed in this release what's different between this and 306 so we see two things and two enhancements and then some bug fixes uh not a pre-release and we don't need a discussion for a pre-release uh we would use this if it was like 3.1 beta we would really tag it as a pre-release just so it gets specifically tagging github's api as not ready for production use and then we can publish there we go that's it so 307 is now available for use hopefully that was um entertaining if not or you know informational if not entertaining um appreciate you guys showing up on the on the chat we'll continue to do these live coding sessions until people get bored of them every two or three weeks probably so um we might squeeze out another one in in october but you know follow me on on twitter or uh uh stay tuned or the net box channel on the netdev community slack it's a great place to uh do uh you know just be aware of what's going on with the project or if you have any questions by all means you know dump them there on github um cool so until next time everyone thank you very much for uh for your attending and uh hope to see you all next time take care

Channel: Jeremy Stretch

Views: 275

Keywords:

Id: Al63TfNoWFE

Length: 66min 8sec (3968 seconds)

Published: Fri Oct 08 2021