👿 Go DEPENDENCY HELL: save yourself with go mod vendor

vendoring go mod vendor specifically why would you do it it doesn't seem to make sense and somebody might suggest it to you you're like why on Earth would I do this um I just completed another video uh about dealing with building private repos for go projects and you can go watch that video ahead of this time ahead of this video but let me talk to you about some of the problems that you might have so here as as I did before we have we have a a sort of mock uh program that has a dependency on a private go package again another video completely about that you can go watch that and you can go see that but this actually will work after we dealt with all the errors and set go private and and did all the things necessary to get this to build so in fact I have a build command still here um go build- o hello and this will pull it down and get it and and you can see that in the go. some file uh we were able to get this private repo this this is a private package but we're still able to get it because we said go private and because this account is a member of the organization that shares ownership uh a read only access to this other package which I demonstrated here so here are the different access this and stuff like this I don't want to redo the other video too much that I went through in the det that and pretty substantial details so go back and watch that video so so the next question is what happens if that package goes away right so there there's actually two reasons to do go vendoring and I'm going to get into what go vendoring fixes before we get into it okay because I have not been a fan of vendoring my whole life I've been like what what is vendoring let's talk about what vendoring is in the first place and I'm just going to run it and you're going to see what it does and then we're going to talk about how that matches the definition of vendoring so very simply rather than having a g sum file here actually I think let's try that let's just remove the go. Su file and see what happens we remove the G Su file let's let's actually modify our go file so now we don't have anything now if can I build I probably can't right if I do a gold build it's not going to work anymore because it needs the module right well let's actually vendor the dependencies um so let's do go mod vendor uh we're going to do go mod vendor and it's like oh okay this Imports this no module require provides it try it by adding go get so the first thing you can you can't vendor a thing until you've tried to get it the first time when you do a goget this is important that you understand this when you do a goget it updates the module cache which is this mysterious thing that lives in a place you can never remember on your system and that contains not only the source code for the module that you have gotten but also all of the you know it's compiled form and everything so it makes it much easier to use when you're doing your building because go as you know uses static linking which just stuffs everything all in a big binary so you still have to do the go getet and if you have the errors that I encountered uh any I made a whole video about the possible errors you can get from goget there are many of them and they are very weird you don't even know what's causing them a lot of times so please make sure you watch that video uh but once you get past those those problems you can go ahead and you get this successful usage of uh of the of the thing now if you're not using Vim you might be using a tool that automatically attempts to download these things um even when you're editing and that's all well and good but chances are going to have to do go getet at some point and then and Gog get- you which automatically updates the latest version which is not done unless you explicitly do it for obvious security reasons right but there's actually something better than Gog get- you and what that is is vendoring so now that now that our our I can actually do a build here I can prove it uh you can see there's my go build it built really fast because the module cache has already been downloaded and updated but what happens if the the project on which this project depends goes away what would happen you know that does make you wonder right well one of the solutions to get around that is something called vendoring so let's do that now so go mod vendor very simple command that was added some time ago that I I totally missed and didn't really care about because I never really understood it I remember reading about special directories and how the vendor directory had special meaning but I never really connect how valuable the thing was until I started trying to build from private repos and this is one of the major advantages of go VOD Vendor by using go mod vendor as a user loged I'm going to say this like I'm going to zoom in here so by using go mod vendor as a user who has access to all the repos in the organization I can add all I can pull down all of the dependencies both on the private repos as well as all of the external stuff that I might have repes on you know like stuff in the standard library and thing and so when you do go mod vendor what it does is it creates a vendor directory that has everything you have dependency on it also adds something to note is it does not download the test code it only downloads the actual code and it downloads the readme uh from the vend from the dependency dependent modules okay now let's let's talk about the implications of this okay so there the first of all let's look at the vendor directory here so the vendor modules directory uh it tells which one it is what package it's on everything it's basically the same information that is in the god. suum file and by the way you still have to have a go. suum file even if you have vendored the stuff uh in fact I think let's try this I don't think I can build it without that with vender I'm trying to see if go will still do the build without the GM file this will be interesting oh my Lord it did it oh that is so that is absolute gold information to know right there so so look at that I have a require line in here can I get rid of the require line I wonder if I can get rid of the require line oh my goodness gracious go build nope can't do it uh oh wow this is a new era I haven't had it before this is interesting inconsistent vendoring um this is interesting it's marked as explicit in Vendor models but not explicit required in go mod aha okay so the answer is uh Thou shalt add it to go mod interesting ignore the vendor directory use mod equals read only or mod equals mod to sync the vendor directory go go mod vendor go mod vendor what did it do did it remove it oh I have to do another goget is this going to update my check sum oh this is this is interesting this is this is another example of how this stuff was tacked on uh to go after it was already doing stuff right um whoopsie all right so I'm guessing it's going to add a somefile yeah but I don't want to go somefile the somefile this that's very interesting go mod tidy I wonder if it's going to make a su file again even though it's been vendored it does oh how stupid is that um that's very very interesting okay anyway so I I mean I don't like the the the that we have a gome file there I I guess I don't know anyway so that we apparently the go mod we don't have to have the the Su file there um I personally I would like this G sum file to not be populated with stuff that's in the vendor directory but maybe I'm missing something there anyway I mean that still builds just fine so I got it look at how fast that it built it right now did it build it from the stuff in Vendor or did it build it from the stuff in the module cache that's a real question isn't it right um so uh that's basically that's because you theoretically can have multiple versions on a package vendor yes you could that's right that's a good point so I guess you do need to check some good Point thank you for that thank you for that um yeah no but I could see I could see that technically you could vendor multiple versions of and uh of something you can have like I could see that with yaml or something so so so you you probably should just do as as you know they intend you can't even vendor a thing until you can go get it for the first time and that has been pretty easy for us at this time because we are uh a user that's logged into the uh organization and and we're not doing cicd where we' have to create a pseudo generic user or a pad or Pat personal access token or something like that um and so let's go ahead and do let's go ahead and do this so so you see what it's done in the vendor directory um and we can do our go building it works so that is pretty much it now let's let's talk about let's talk about let's not let's talk let's stay on topic everybody because we're doing a video so let's let's talk about the considerations of when you would use vendoring and some of the reasons that I didn't like vendoring before the reason that I have never liked vendoring is because I always felt like vendoring meant that I had to copy and paste you know the code out of some you know dependent module and put it into a place that I decided uh that would be used and that is no longer true go mod vendor which is just one command that you can rent anywhere will automatically pull everything down and you can see all the stuff about uh go help uh mod vendor I I think it is uh and it'll tell you all vendor resets the modules main vendor directory uh causes the vendor to print the names of the vendor modules e caus the flag to attempt to proceed despite errors o caus the vendor to create a vendor directory at the given the stad if you can say if you don't other than vendor uh vendor is a special name though so I would never change that um again the idea of vendoring is pretty simple vendoring means taking something from a dependency that would normally be external to the project and bringing it into the project that is what vendoring means these days and why would you do vendoring so the first reason you would do vendoring is you wanted to simplify your repo you wanted to take you only really want I mean the the downside of go mod vendor is it pulls in the whole thing um and quite frankly that was new information to me because before when you did vendoring another form of vendoring is to cut and paste out one fun function from some other module and just put that in your project right and you probably should make attribution and Licensing and all that same stuff that you need to do but that occasionally you only need one function from another project and it's actually better for you to copy that function down especially if it's a pretty simple one and put it in your own code and it's not illegal to do that it's it keeps your project simpler you want to reduce the number of dependencies and if you've been doing any amount of serious go development you know exactly what I'm talking about when you get into the business of doing tons and tons of go you know dependencies you you're waiting around forever um and so yeah you have to be able to get a module to vendor it because it was forced to move some stuff to to internal instead of vendor oh really you put some stuff yeah internal is another special package name uh module uh subdirector package name which we'll talk about uh later um but yeah so you have to be able to access the thing that you're going to put in um interesting that that it doesn't allow you to to move the internal stuff interesting yeah anyway so so yeah your package even if your package repo your module is private and that can include get a Enterprise Cloud on inner source which makes everything private um you know you you can still get at it right um and as in this case my arix Rob get J's test account has read only access to those to any of those repos in the under the organization and therefore I can read it therefore I can do go get to do the update and then I Can Vendor it so so the other position of the other point of vendoring it is let's take a hypothetical situation so let's say let's say that you want you have a project and you know I mean this is an extremely hypothetical but it's a very important one you have a very important project that's got a dependency on uh uh a private repo or another repo uh actually vendoring works with public and private obviously uh we'll get we'll get to vendoring for private repos as as a reason all by itself in a second so let's just talk about the other major reason for doing vendoring what happens if the dependency goes away right and you might think well I've already got it in my stuff and everything if you don't vendor it you don't have a copy of the code and unless doing some crazy proxying of the git repos or something like that let's let's take a very real scenario okay let's say that you have a dependency on and believe me mean this has happened this has happened let's say you have a dependency on a an external library that has that is discovered to have a really glaring horrible security bug right and let's say it's so embarrassing for the organization that you are dependent on that they drop it they get rid of it they go private it goes dark and you don't have access to the thing anymore and your project now no longer has access to the source code that was used to build it think about I I really want to pause for that to sink in for a second if you are on a huge project and you have a dependency on a public repo or private repo that is completely organized and maintained by somebody else and for whatever reason they stop publishing that package you're screwed I mean you're really screwed because even though you may have built with the last cash version when you go to try to build again you're out of luck and that that that module cach that we talked about that only lives on your specific machine for the time that you were building it and if that were to happen I can almost guarantee you there's going to be a lot of a lot of really panicked developers going diving into their cash looking for the source file so they can create their own vendored solution or replacements for the for the package that's now unsupported and unavailable that is a very real risk to a to a project and it is one of the main reasons that you should be very very very very careful about adding dependencies to any other project ever um uh experience with C dependency management to implement it by yourself yeah because that kind of thing you think it's unlikely to happen but it's a very real thing another another likely scenario let's say somebody has um uh let's say somebody has a very you know horrible security flaw right and they're so embarrassed by the security plaw that they take their whole thing down and you now have the security vulnerability and maybe for security reasons or maybe just shame reasons they don't want to say how bad it was and then your project pulls up in a security audit that's having a vulnerability for whatever reason right and then you're trying to trace where the vulnerability came from and you're you're looking through your own dependency tree right you're looking through your own dependency tree you're looking through your own PR history and your commits and everything and you can see that that perhaps the audit ran against a version before this and and the only the only evidence that you have of the problem at all is in the checkm folder right so if I do I'm going to go get again so I can go update it um let me see sorry get uh go get so let's let's do this again so just so I can update this now this has only got one but if you've seen bigger projects the only evidence that you would have of a change that in that brought about the security vulnerability in your project is going to be a line item a singular line item in your go require or in your go some file that's going to show that something changed so when you're doing root cause Anis and you're trying to find out what changed and you're looking in your project the only thing that you can look for is what is in the goome file and what is in the go required file and at best at best since they pulled the project at best the only thing that you can actually further research as to why this problem even happened is you can just say well it looks like it might have been one of these possible dependencies that is now gone now I in a perfect world that project would still be available and you could go deep dive into their PR get history and source code history tree like that but if but in an imperfect World they've already pulled it and you don't have access to it anymore enter go vendoring if you have been using go vendoring then For Better or Worse all of that code is now saved with your project for better because you can now see instead of having to see the line item change and go sum which could maybe trigger you but then as soon as you see that there's a change in your G suum file you could then you could then Deep dive into your vendor directory and see the exact changes in fact when you update your go mod vendor you should commit a PR you should commit one PR for your go mod vendor updates because that PR will contain a very exhaustive dip of every single dependency that has changed and all and that can be rather large and that's the downside of this the downside of this is that when you add all these this you can add some dependencies it could be several thousand lines of code and now that code is all permanently stored in your repo bloating your repo Beyond you know the where it might need to be and I don't even know when go does caching this is something I'd probably have to research but I don't know how many um uh yeah I I don't know how many of these um like when you're I I don't know what happens with G module caching of a thing that has vendored stuff in it I don't know I don't know so like for example if you were to do a go this is something I need to research but if you do like it's like so somebody does a go get against my stuff right here right is if if they if they have a dependency on my thing are they also going to pull in my vendor tree and I think the answer is yes I think that the vendor tree would travel with it so you can see that this this also has a downside because you're now this is the reason I don't like I haven't liked you know cart blanch vendoring of everything the way go mod vendor has worked and it might be that this is is the thing that ultimately makes me decide to use a more precise version of vendoring or go back to using you know generic accounts for dynamic cicd builds without having vendoring at all which is we're going to get to there that's another major reason to use vendoring um but but this meth this this part of vendoring uh comes with bloat it does it comes with bloat it's a minimal amount of bloat but it and it's but it's it's bloat right on the one hand it's it's good bloat because it gives you exactly the code that was in your project at a given point in time exactly the exact code is saved with your project so you can go look at it at any moment you can do audits against your own code base face it would be really interested to see if hey how's it going tiny fall it would be really interesting to see if um Auditors security Auditors who are who are passing you know who are sending who are who are examining Go stuff uh bypass the vendor directory because I imagine a lot of security audits would probably look into the vendor directory if they're if they're security auditing a go module because they would want to ensure that the vendor dependencies uh which are also in the check some files so I imagine you know go auditing would probably just be good to look at the checks some file and see if you're at the right version um assuming that the check on file is in sync with anything that's in your vendor directory which according to everything I've I've seen so far it is um so again like you get you get all the code for better or worse it's in there right um so so that is the primary idea behind vendoring the primary idea behind vendoring is to save the code on which I have a dependent with the project for better or worse for all the reasons that I talked about so you always have a copy of the code no matter what happens if your dependencies go away if you're the people that own your packages your package dependencies die if their if their GitHub repos blow up you still have the code and if if the code was released under open source and that were to happen you would you would then therefore have the option to take that vendor directory and and do whatever you wanted to with it because open source and that that is you know all by itself is is a reason to use go vendoring but there is another reason to use go vendoring related to GitHub Enterprise Cloud migrations that have dependencies and this is huge this is so huge this is going to be our new Direction uh within the company for our Enterprise migrations to GitHub Enterprise cloud and and and I'm going to take a moment to to really talk talk about this okay so um when you the the the most important take and I've done a lot of videos on this recently because this is a very important thing so in my rather large multinational corporation we are moving from GitHub Enterprise server which is effectively public when with regard to go because everything's public right it's all on a service just behind a VPN but as far as go is concerned everything's public and go the go module system with was designed without question to use public modules everything else has been added on over time to make it work and you're going to see evidence of that all over go watch the videos I I already said this but there's a video I did about the five things to consider to get around dependencies on private modules and go because it's not you're like swimming Upstream there's like so many comments on everywhere on the internet about swimming Upstream when you're doing when you're trying to use private modules you can do it but it's always Against the Grain it's always feeling like it's a hack it's always tacked on it's always some extra thing enter go modules go vendors sorry go mod vendor go mod vendor so so let's let's let's do the math here okay so what does go mod vendor fundamentally do it gets rid of the necessity of having access to those external repos so so let's let's paint the picture here right you've got you've got your to you got your internal tool and you have another Library you want to break out some of your code into a centralized Library um uh so so so then you have a dependency on you have a dependency on this internal library right well when you're want to go Enterprise server all of that stuff is public it's all public and you can break it out you don't have a problem right if you have a single project like I do K login I have a single project K login it has no dependencies on anything that's a standard go thing and it's fine right it's totally fine the the go releaser just works because there's no dependencies on any private modules whatsoever right it just works and the go releaser works and it's able to use the GitHub uncore token which lives for the duration of the GitHub action when it's running and it only has access to that repo it's really great for making release artifacts and all kinds of stuff like that it's super easy to use it's already automatically there you don't have to create a personal access token and add it it's there already so if you are building a go project that has no external dependencies on private repos then you can get away with just using that you don't have to do the SSH setup and all the other stuff that I've done in other videos you don't have to do any of it you just just builds it just works with go releaser now if you as soon as you have a dependency on even one private repo you now have to make all of that stuff work right but go vendoring makes it so that you don't have any ex external dependencies on anything so I let that snc in for a second so so if you if you have um and the reason it works is because the bringing in the code into the vendor thing as a part of the pr process that bringing in of the code that is done by a user who already has access to all of those other external private things and it's actually pretty easy to build you know locally and just by setting go private like I did in another video and and you can play with with go go you know no proxy and go go no some as well and those are all things you can go read about elsewhere the point is that because the vendoring action is a develop it's in the developer space and not the cicd process because it's something that a a a a developer does like you have to do go vendor it's equivalent to copying all of those Tech those source files into your project and then submitting a PR for that to be an actual part of your project you didn't write the code but you brought it in as if you were writing the code because all of that happens by the developer that happens way before cicd ever gets a piece of it and so when cicd is invoked as far as the GitHub action is concerned every external dependency is now available within a single repo under the vendor directory and that means that a GitHub token Works a GitHub token works and you the same exact action that you reuse all the time for all your go projects now works for every single project every single go project you have in which you use go vendor so so by doing that you radically radically simplify the steps necessary to make this happen and if you have got if you don't know what I'm talking about it's very basically instead of creating convoluted SSH and Pat and and proxies and all these other things that are documented in the 45 to 60 Minute video elsewhere instead of doing any of that and instead of wrangling with your GitHub team to add a a different account or to add a justifying the addition of a readon pat on on an organization or or you know fighting with them or getting deployment Keys God help you if you do that instead of any of that go mod Vendor by doing go mod vendor you bypass all of that you don't have to talk to any of the teams responsible for setting up your organizations you don't have to justify a pseudo account generic ID you don't have to deal with 180 day expiration day pad tokens you don't have any of that that all goes away there's no secondary you know Vault Storage for the primary secret key for the SSH that goes with the generic account so that you can just change that when it finally expires so you can force it or re reissue it and then have you know the secret Keys managed on on the generic the the level of complexity or and God help you if you feel like you have to have a GitHub application ID with an installation token just to do something that you could have done with go mod vendor so so by accepting the fact that you are bringing in all of the source code dependencies they're no longer external dependencies because now they are effective internal dependencies that have been organized under the vendor directory they are external dependencies but they are explicitly managed external depend dependencies that are present and accountable under the vendor directory and and I'm trying to be as clear as I can about this but the takeaway of this video is that go mod vendor bypasses any problem you could ever have by having your project depend on one or more external private repos which GitHub Enterprise cloud forces every single git repo to become and and I that is a problem that's been presented by the by by Go's assumption on public repos and if you ask me I mean you know I've watched other people who have said that they didn't want any solution that couldn't also be done by the user from the command line go mod vendor is the easiest of those Solutions if if your primary Criterion if your primary requirement for this solution is that it should be usable by interactively by a developer go mod vendor is the definition of easy to Ed by a developer so get over your version for go mod vendor in my opinion and try it out and see if it saves you a lot of time if it helps you bypass all the hassle by having you know private repo dependencies and please please please stop making monor repos that's a horrible idea

Channel: rwxrob

Views: 521

Keywords:

Id: nInBqtpA0-A

Length: 31min 29sec (1889 seconds)

Published: Tue May 07 2024