Nomad Pack Walkthrough

hey everybody my name is mike noetch and i'm the product manager for nomad at hashicorp today i want to talk to you about a new tool that we're really excited about which is called nomad pack it's a new package manager and templating tool for nomad in this talk i want to give you an overview of what nomad pack is i want to go over the various ways that you could use nomad pack within your organization and then i want to dive into the structure of what is in each nomad pack and then we'll go over the process of taking a nomad job file and turning it into a pack that could be consumed by anybody so by the end of the talk you should have a good sense of what the tool is how you can start using it and how you can contribute to the pac ecosystem all right so let's jump into it so what is nomad pack nomad pack is a cli tool that can be used either in your terminal or in a continuous integration pipeline and it sits between registries and a nomad cluster in these registries we have nomad packs which are groups of templatized job files and variable definitions nomad pack will pull down a pack from one of these repositories allow a user to input variables and then send these job specs to the nomad cluster to run them and manage them over time so in this example we have two github repos one could be public the other could be private and to get lab repo nomad pack will pull these packs down and then send jobs to nomad in this case running traffic jenkins and postgres so let's go quickly over what it looks like using nomad pack so right now i have a nomad cluster and i have nomad pack locally so if i run the cli command i can see the various commands that i can run and right now i'm going to list out the various packages that i could deploy so i use the nomad pack registry list command and so this shows me all the packages that i can deploy this right now is pulling from what's called the nomad pack community registry so this is a public registry that anybody can contribute to and we really encourage people to contribute to it so you can open a pull request and then your pack will be added to this list and then when people use nomad pack in the future by default they will have access to the pack that you've written so this is what we have right now as options to deploy to nomad so i look at this list and i say okay i can deploy some cool things i could deploy traffic i could deploy nginx i can deploy prometheus but for the purposes of this demo let's just deploy hello world so i'm going to copy that and the first thing i want to do is see okay what are the options that i could pass to the hello world pack so i will run the info command nomad pack info hello world i run this it gives me a brief description of what this pack does deploys a simple application with an associated console service and it gives me various variables that i can pass into the uh package so that when it's deployed to nomad i can customize it for my own needs in this case we'll just focus on the message variable so now the next thing i want to do is see okay what are the jobs that this would be deploying to nomad each pack can have one or many jobs that it deploys to nomad and so let's see what that looks like for that i'm going to use the render command so i'm going to say nomad pack render hello world and now this puts out what the nomad job spec that this pack defines looks like so i get a sense of what it's doing i say okay this is a server a service task it's a docker driver and it's using this m-nomic hello world server with this environment variable hello world now the important thing about pax is that everybody can use them how they want by passing in variables so i saw that there was a message variable so now if i pass in the var message equals something let's say hola mundo now when i render this file oops there we go now when i render this file this message says ola mundo and now that's a very simple variable that we can pass in but as we go we'll see that we can get really complex with the logic that we insert into packs so now i have an idea of what this is doing i can now test what this looks like against a nomad cluster so i'm going to run this nomad pack plan command and this says that this will deploy a single job to nomad and it would be successfully allocated all right so it's time to actually deploy this so now i use the nomad pack run command i give it my hola mundo message i guess it doesn't like exclamation points all right now it says the pack is successfully deployed and now what i could do is i could check the status of this pack i can make sure it's up there this now lists all the packs that have been deployed to nomad the status command and if i want the status of a specific one i can check hello world and this says i'm running the latest deployment of hello world it has a single job called hello world and it is running so let's check that out in nomad all right so now if we go to nomad we see that the hello world job is running if we click into it we see that it's a pack and we see some pack details along with the standard job details so we see that this packs name is hello world it's coming from the default registry and it's the latest version of the pack and then if we jump into the actual uh server that was deployed at the port it's deployed on here we see that it's saying hola mundo so we properly took a variable from the cli we passed it to nomad pack that put it into the nomad job for us and then it deployed properly so that's a really simple uh version of running nomad pack with the simplest possible pack now you could do this with a ton of different jobs and really quickly with you know a few key strokes be deploying common applications like wordpress like nginx like aj proxy to nomad now let's jump into the details of what is actually in a pack and how do you make one okay so right now i have opened the hello world pack uh directory so this would be the directory that gets pushed up to uh the community registry so that other people can use this hello world pack so let's just jump into the contents one by one and we could explain what's in there as a brief overview we'll have a readme a metadata file a variables file then an optional outputs template and then also we'll have the actual template that makes up the job spec and potential helpers so first let's check out the readme so in the readme when you're writing a pack you're going to want to write a very simple description of what your pack is and then how one would use it and then also call out any integrations or dependencies so here i say that there's an integration with console and so that someone who uses this will know oh you know i will need console up and running to potentially use parts of this and then i call it other integrations that this has so that people could really use it successfully i'd say in your readme just try to set up whoever the end user is for success and then also we have a list of variables that can be used and this gives people a general idea of how they could um customize this application for their needs so that's the readme file now let's jump into the metadata so this metadata defines uh some general information about the app you know what the url of the app itself is so if this were let's say uh you know wordpress this would point me to the wordpress homepage and the wordpress author and then this pack stanza says this is the name of the pack gives a general description and then a version and so if i were to um you know run nomad pack info it would be pulling this data from the metadata.hcl file also in the metadata file you can call out pac dependencies so we won't go into this in detail in this talk but packs can depend on other packs so you could have let's say reusable code to keep your packs dry in some sort of helper pack and then inherit from that in a variety of other packs so that's the metadata file now let's jump into the variables file so if you're used to terraform this will look really familiar we tried to model it based on the variables that you see in terraform and this is running with hcl as well so here we have there's a job name variable a region variable data centers variable account message whether or not to register a console service etc etc and now whenever i run this pack i could actually pass in different values uh for each of these variables so we saw earlier i passed in um the d the message of ola mundo it overrode the default hello world uh and then that rendered that to the screen in this pack i see by looking at the variables file that i could pass in a different account so i could you know potentially run 10 of this pack and then i could also say whether or not to register a console service and what that console services name is so it's really important when you write a pack to figure out what you want to create variables for how you want to parameterize the pack i would say that it's important to strike a good balance between parameterizing too much or too little if you parameterize too little it's really not that useful to pack you might as well be sending someone drop somebody a job file and having them figure it out from there because they can't customize it to their own needs if you parameterize too much it's also not that helpful because they have like you know 50 variables to read and try to figure out how to actually use the job so i try to strike a good balance there so that's the variables file and then lastly we have the actual templates themselves that define the nomad job specs so these are created in the templates directory and any job that you want to deploy to nomad as part of a pack gets a dot nomad.tpl extension at the end of the file name so if i had multiple jobs that this pack wanted to deploy i would say nameofjob.nomad.tpl let's jump into the hello world job so this job you can see looks roughly like a normal job spec except you'll see some delimiters the open brackets and the closed brackets we'll get into that in a second but you see that normally it looks like hcl except for that now go template is what's being used with these delimiters so whenever i open a double bracket it says that i'm using the go template templating language here this allows me to do things like inject variables in this case i'm injecting the message variable it allows me to do things like call helper functions so here i'm using the quote helper function that says i'm going to take this message and pass it into quote and then it's going to render that into the job spec so in this case it's going to take whatever my messages wrap it in quotes and we'll have message equals quote so that's a really simple use of go template but we have actually more complex uses so here we have some conditional logic so this is saying if this variable exists then render anything in this block to the hcl so now i can get a little more complex i don't just pass val variables in but i could actually you know completely omit or skip certain stanzas there's also the ability to use helper templates so here in go template we're using this region template and we'll see what that looks like here all right so this is the helpers file and what this does is it defines templates that can be used elsewhere in your code so for instance if you have let's say a console health check that you know knew that every job was going to be using you could define a console health check template and then insert it into various jobs in this case we have two very simple templates one that defines the job logic for a job name if the job name is equal to nothing give it the name of the pack and otherwise pass in the job name variable as the name of the job and here we have another one the region template that says if the region is not equal to empty string insert the region value here and so this logic gets inserted right here into the uh nomad job spec all right so really quickly let's see this in action let's use the register console service variable so if i am the author of this job i can use the render command to see what this job spec looks like so i'm going to give it the render command with the var register console service true and we can test out the go template logic so i do this and i see that hey yes this service is being rendered so everything in here was properly rendered now if i give it a false value instead we see that the console service was not rendered in the job and when this gets pushed up to nomad no console service will be registered so that's a really simple use of go template logic to parameterize values and pass in variables into uh packs so that you could customize them for various use cases all right so now that we know you know what the contents of a pack are how to use nomad pack generally let's go to go through the process of actually writing a nomad pack ourselves so the application that we're going to packify is something called next cloud which is a productivity suite can kind of be thought of as an open source self-hosted google g suite or google drive we're going to take that in nomad job spec form and we're going to convert that to a nomad pack so at the end of all this what we want to do is push something up to the community such that anybody now can go install nomad pack and say nomad pack run next cloud and get this uh no you know productivity suite up and running on their uh self-hosted cluster um this is something that's been really popular for people who you know run home labs or really into self-hosting but it's also something that's also often used by big enterprises if they want an alternative to g suite okay so let's get into it all right so here are the basic steps to writing a nomad pack first you want to start with a nomad job spec and we're going to assume that you know how to do that if you don't know how to do that there's some resources online you want to start with something that you can push up to nomad with a nomad run command yourself and then we'll work from there then you want to get some basic scaffolding we'll show you how to do that then you want to input the metadata and the readme information um for future users and then you want to go through this process of identifying parameters so these are all the things that could be variables in your nomad pack and you want to figure out okay what are the things that i actually care about allowing people to customize in this pack then we are going to go through each of those parameters and parameterize the template we're going to make a variable and we're going to pass that variable through to the templates and conditionalize what we need to conditionalize input you know the values where we need to input them so we do that for each of these parameters then we're going to test our pack locally and then lastly we're going to contribute by pushing up to the nomad pack community registry all right so let's start with our nomad job and i'm coming into this with this job already lit written and let's go through it really quickly just to see what it contains so it's job next cloud has some basic information on region data centers name space etc it has a constraint that says that it will only run on linux and then it has a task group and this task group has a couple of tasks within it they are connected via a bridge network so that means they share a network namespace we have a couple ports that are static and this is because i know that the virtual machine i'm running it on has these exposed so i want my http port to be exposed on on 4001 and my db to go to the postgres port that's conventionally 5432 then i have my application this is running next cloud's latest image uh it mounts a uh volume from this directory on my host to this directory within the task this is conventionally where next cloud expects this information to be has a resource stanza and then it has some common environment variables now nextcloud can be run with a postgres database or a mysql database in this case we're running a postgres database so we have these environment variables exposed as you can see it's very secure and then we have the database tasks this so this is running postgres again i have kind of an older image being run because i know that that works someone else might want to use a different image we'll get into that in the future and then i'm mounting another volume here for the postgres data again on my machine i know this volume exists here but on someone else's machine it might exist somewhere else and then more environment variables and more resources so it's a relatively simple job and i know that it works i've tested this before i'm able to run this job so let's just start from that assumption so we've written a nomad job step one is done let's get to step two and let's get some basic scaffolding so i'm going to pull down something uh here this example nomad pack registry so it's a revo we e-host within hashicorp and it could give you just a very basic registry to get started if you wanted to write your own packs this is where you could start you just clone this template and right now we see that the hello world pack that we're familiar with from earlier is in here so i'm going to clone this and we'll start using this as the basic scaffolding for our pack so we'll take this and we'll actually convert this from a hello world pack into a next cloud pack okay so i'm back here and i've now taken the working nomad job i've plopped it into this pack that we got from that scaffolding and we see that it's a hello world pack let's see what we now need to change so first we need to go and change this hello world name to nextcloud we also have some variables that are referencing hello world variables and then also we have this output that is specific to hello world we have this readme that's specific to hello world and we have variables so i don't want to you know make you sit through all this so what i'm going to do next is i'm going to change the names of all these references to nextcloud i'll change the name of this file i'll delete any of the variables that we don't need and then in the readme i will get rid of everything that we don't need so with the power of video editing i'm going to do that right now all right so i removed all the information uh that was specific to the hello world job i've replaced next cloud i've renamed it next cloud where it needs to be renamed i've gotten rid of all the variables except a few and i've cleared out the readme so i think step two is all good all right now it's time to input metadata and the readme this should be pretty quick all right once again through the power of video editing this is done already so i just dropped in a brief disclos description of what next cloud is uh some caveats about how you would uh run it successfully i called out dependencies saying that it requires linux and it may or may not need console more on that later and then i left the variables blank because we haven't yet identified exactly what the variables will be variables will be i have some ideas that i spelled out here but we'll go into that soon so that's the readme and the metadata similarly quick drop a brief description a name and some links to uh attribute the author all right so that is step three we're done with that now it's time to move on to step four identifying parameters all right so i've taken the working job spec and i've dropped it into the template file but not much is parameterized right now so i think what's important is to go through this and see what assumptions are we making about our environment that might not be true for other peoples and then we'll make a list of those and those will be our parameters that will pass in his variables into this pack so the first thing that jumps out to me is this constraint this linux constraint so uh i might have only this constraint but other people might have more constraints you know they only want to place this job on you know certain nodes in their system so we should allow people to add their own constraints that's one the next thing that jumps out to me is the network stanza so i've assumed that i want to network stanza and i want these static ports these ports are static on my machine because they're opened up to the outside world but on other people's machines they might not want static ports and actually we advise against static ports on nomad so this is again something that we'll want people to be able to customize with the variable so now we have uh the network and the constraints as our two parameters let's keep going so now the next thing that jumps out to me is the image i'm using the latest image but a lot of people like pinning images so we might want to pass this in as variable after that i realized that this mount is making some assumptions about the system too i have my data at this directory other people might have their data at some other directory and so we should make this mount uh customizable same thing for the postgres task that has its own mount and also its own path for the source data so there's that and then next up i realize that these resources are custom for my system so i'm planning to run this in a home lab i can run this with you know relatively low memory and relatively low cpu other people might be running this for you know an entire enterprise and they might want much higher memory or cpu or they might want lower memory or cpu so that should be customizable as well and then lastly these environment variables i'm you know making some assumptions about the admin and the password um and also i'm hard coding in the postgres password which isn't good a lot of people want to use vault they'll want to pass in a stanza that says or a call to vault saying that they want to grab that password from vault so that's something that should be parameterized as well and then lastly there's this question about whether this database task should exist in this pack or not now i want this database task because i'm running this on a single node i want this database running in the same name space but in the future uh you know i might be running a database let's say on rds or on on some managed database service and i won't want to you know have to deploy this database on my node myself i'd just rather point to it in the future so we can probably identify that some people will not want to run this task at all and they'll just want to run this application task so that's another parameter all right so now we've identified our parameters in step four let's take a look at the list okay so i've added these two variables next cloud image tag and postgres image tag i give people a description and a way to find all the options a link and i inject my defaults as what i was using previously now that these variables are here i can inject them using go template into the template so here for instance our old code looked like this it was using a specific image and our new code looks like this it's using the variable let's test this out really quickly so i go to the terminal and now i can render and pass in this image tag as latest when i do that we see it's using postgres latest when i don't use the variable it uses the default great now let's move on to resources and for this one we'll use a helper template all right and now we have two more variables we have this app resources variable and db resources uh this is a little bit more complex than our other one it's an object variable so it takes a cpu and memory value for each and you can see that these are the values that we were passing in before before as the default now when i use these i'm going to use them with this template and so our old code looked like this and our new code is calling this template and it's the same thing above in the application task so this template gets an argument which is the database resources variable and then you can see how it uses it it takes the argument it assigns it to this value resources and then it prints out the resource stanza and this is a nice dry way to uh write our code now next up volume bounce okay so this is our most complex variable so far so this uh postgres balance variable defines a list of objects each of those objects have a lot of keys type source target etc and then bind options has a list of objects itself and our default is right here and this is what we were using previously in our template so let's look at how this is used so in the template we're passing this variable into the mounts template the mounts helper template so let's look at that this mounts helper template defined here takes the argument it iterates over it and for each mount it creates a mount stanza and then it injects the values from the variable into each of the keys in the stanza and iterates over each of the bind options to create a bind option stanza so that was volume outs and i'm actually just going to skip past the others as it's a very similar process you add variables you add logic you templatize et cetera et cetera so let's just go to the end now and the last step we have is to test this and then contribute this is the fun part now we have a fully parameterized job spec we have a constraints parameter passing into a constraints template we have uh the network being parameterized we have environment variables being parameterized as well optional console services being passed in and then also this conditional saying that if you include the database task you'll add a database task but if not don't render a database task and then lastly i've added this pre-start task that creates data directories on the host using raw exec this is optional again this is off by default but um if you run this it'll make it easier for people to use the pack without doing pre-existing setup and that's really important so let's test this out let's go to our terminal let's render it once just to make sure it works okay that looks good we'll run a plan [Applause] okay that looks good too and then we'll do a run and this is now pushed up to nomad excellent we'll give it a second for the images to download and then we'll check back in all right the images have downloaded so we'll pop in here we'll check out this task group we have one allocation we have our pre-start task and then we have our application and database task this application if you remember was running on port 4001 so now if we go to port 4001 we see that we have this login and we gave it admin and password as the admin password and now when we log in it works and now we have an instance of next cloud up and running and now we just have one step left pushing this up to the nomad pack community registry once we push this up and it gets merged anybody will be able to run next cloud on their nomad cluster with just one command nomad pack run next cloud so here's my pull request i give a brief description of what next cloud is and i create the request and there we go so that's it we've gone from a nomad job to a nomad pack and i hope throughout all this you now understand what nomad pack is how it works internally and more importantly how you can contribute going forward i really cannot encourage you enough to contribute to the nomad pac community registry it's something we're really excited about we've already had a great response so far it's something we're going to continue investing in in the coming months and years so please stay tuned for more around nomad pack and its integrations with nomad have it going

Channel: HashiCorp

Views: 117

Keywords: HashiCorp

Id: Y87nXbQTw7w

Length: 29min 11sec (1751 seconds)

Published: Wed Dec 08 2021