Getting into HashiCorp Consul, Part 5: All About Access Control Lists

we've been really focusing on console cluster construction and operations uh we haven't really gotten too far into the the capabilities or the things that you would use console for we are getting to that that will be in a couple weeks uh we are still on the operational aspects of it including but not limited to limited to um initial setup configuration scaling and now we've been working on security so if you want to look at any of the other setups or any of the other things that we've learned in part of the previous streams we have a youtube playlist and you can also find them on this twitch channel as well now in terms of what we did last time shall we do a dramatic recap i always say we are going to do a dramatic recap of what we did the last episode but it's never as dramatic as i think it is last time on getting into gonzal uh last time we encrypted gossip between the agents and we also encrypted grpc between the agents the secures agent communication we also took the brave leap into turning on access control lists aka acls aka ackles much to kohl's uh you know and all of these things will help secure not only the traffic between agents within the cluster it also secures and ensures that agents are getting the minimal access so that would be least privileged access within the cluster um so we didn't think about adding in acls last time it was actually pretty impromptu yeah why not we'll just add asls in there uh acls are the ones that often cause quite a bit of heartache um you know in in the console space uh they it's not that they're necessarily complicated but it is sort of tricky right because when you're trying to implement least privilege you end up breaking functionality and you go through this iterative process of trying to make it work so today we are going to finish off access control lists and we got to a point last time where we implemented access control and effectively leased privilege for client sort for node communication as well as for service communication but we broke dns so we originally set up dns for service discovery and we broke that so now we have to go back and fix it that is our hope that we fix that and the other thing that we're going to do today is hopefully talk about access control lists and the policy policy and rules uh syntax i know cole you were like i know we do stuff with acls but i kind of want more detail on how this works and so as a result we're going to demonstrate the access control list rules engine so to speak with console key value store so we're going to do that and if we have time we may not have time or we may not have the inclination because we had some fiddly bits that we didn't we didn't make we didn't quite get organized and that's securing the console cluster with an external ca we probably won't really do that right now but anyway that's kind of a recap of what we're planning to do um what changed cole i know that last time we did some manual setup with ucls as well as the security stuff so what did you end up going back to automate more of yeah an interesting part of this series for me at least is after the fact because we do this all manually uh and then i need have to to go back and say okay let's figure how we can automate this and um with this particular part of the thing there's an interesting almost chicken and egg problem you encounter in the sense that you need console up before you do things but to do things console needs to be up so for example uh in our last episode when we went through it we used the built-in console certificate authority to generate those certificates right and then we passed those off to we generated ones for the clients we put those on there as well well automating that in a boot script is yeah difficult in the sense that if you have three servers go up how do you keep all three from doing the same thing if you want them on not a scaling group which has led me to an interesting thing where i would love for a way within auto scaling groups to differentiate between the nodes in them even though i know that kind of defeats the purpose of an auto scaling group it'd still be nice but i digress um the same thing has sort of taken place with the acl system so anyone who's following along the github repo as of right now you will have to do those steps manually but i'll make sure to have those all laid out but to talk about the automating of the gossip key and the pls part which is automated let me uh share my screen here get it up and alright can we see my screen we can see your screen yay so let's talk about the gossip encryption key all i did to do this was just pulled in the random provider made a 32 byte length random id and passed that in as the gossip key since that was what is being done under the hood uh from my understanding the console when it generates those when you generate it through that so we're just passing that into the user data script and for all of our different nodes here that's just something that we set uh right in our hcl file so that was super easy because when we did it live we had to do the gossip inc the gossip encryption key but we also had to toggle those encrypt verify incoming outgoing in that rolling update to get that to work so automating that was very simple the next part was automating the making it so there's tls encryption between the console agents that was a little bit trickier for the reasons that i just mentioned earlier about needing to have the servers live and differentiate between them and past things so what i did here is i pulled in the tls provider so the tls provider lets you make these different certificates and made the root certificate right here just right up in this tls file followed by the server certificates followed by the client certificates the different one for the different services and then ultimately passing those in through our user data and putting them into the user data script and booting them up now with this just i want to give you some some heads up this goes up this does work but there are some problems with it and the problems are the first one is that uh four the probably the biggest one is if we have three servers the problem you're going to encounter with this particular shut up is that all three servers will be using the same uh certificate and private key not the ca the ca is is fine that's all good but they'll be using the same certificate all the clients that are for the api will be using a similar certificate for themselves and the web will all be using one for themselves as well as opposed to having a unique certificate for every single one of them i do know that that would be ideal but in terms of making it easier for someone coming to this project i went this route so that you could just go ahead and launch it and not have to worry anything about it so that was kind of the hack there if that makes sense do you have any comment on that rosemary no i mean like i said for those who are tuning in and looking at this repository that we have a working repository i'll put it into the sort into the description and and everything else um but keep in mind that a lot of these practices while we're we're trying to demonstrate as much as we can in a you know in sort of the minimal production configuration a lot of these practices may or may not as cole pointed out may or may not be 100 perfect so um the things that we are doing here are for your benefit so that you can get started with this repository and it's self-contained yeah yeah in a perfect world you would have a unique certificate signed by the ca for every single node that you have live doing so is a little bit tricky with auto scaling groups to the point to where i think the approach that i almost landed on was having a unique auto scaling group for every single node and using something like count or forage to make those happen but it seemed a little heavy-handed for what we're trying to learn and do here so i stayed away from that and then beyond that um the acls and automating the bootstrapping of acls was not something that i had a satisfactory solution for so as of right now that's all been reset up manually but like i've stated in the previous episodes we do have the manual steps that manual step by steps uh in the repo so like here's part four where we just go through and you can just do the exact steps to get things up and running and that is all much feel there on what has changed excellent so we've got a lot that changed uh between now between both the the last stream and now uh what didn't change was that dns is still broken right cole so like if you go into uh was it web yeah web hit the load balancer here we can get web but suddenly it cannot talk to our api can you increase the font on the browser yes thank you yeah so it can't communicate because when we enabled access control lists access control list in console is like api authorization as a quick summary it is controlling service as and node access to the console interface right so some api endpoints certain nodes or services have access to while others do not uh and what happened is that when we configured access control lists our dns interface has broken because it likely doesn't have access to console anymore to make those resolutions uh so what we're going to do is we're going to fix that because this does not help us if we were running this in production and we were supporting some services we effectively have broken our services so now we have to go back and fix it so what i'm going to do is there's a learn tutorial um for fixing dns uh and acls um we are going to go through that so that we can once again resolve our services uh web communicates with api yes cool ah so this is stuff we're going to do in the policies themselves yes we're going to play with policies okay um so the first thing that we're going to need to do uh is that we need to establish a policy so for those who didn't tune in the last time console has policies that are linked to tokens so the way that the access control list system in console works is that you create a policy and then you can create a token that's attached to that policy so what we are going to do is go and create a console acl policy that allows the nodes services and prepared queries to communicate with uh console for dns now console prepared queries this is the first time i think we're coming to that right yeah okay so i'll paste a note about the console prepared queries but prepared queries are uh service queries basically so you can use it to to get information about a service that's kind of how uh the underlying dns sort of dns itself right if you combine it with console dns it gives you more complex information so if you wanted to get more selective about the kind of service information that you wanted to retrieve you can add complex queries we have not or sorry we you can add prepared queries i don't believe we are using prepared queries so we don't need that policy um we have not configured prepared queries as part of this series at all um we don't have a need for it like we don't have any real complex service searches that we need to make so um in terms of yes there we are in terms of the policy uh cole i don't think we are going to need the prepared queries section we just need the uh top two policies gotcha so i have a question for for that because i just want to understand reading it um [Music] and just could we before i even like put this in here just go through and just in plain english say like what each of these little bits is doing yeah so in the acl system um they have these uh abstractions right so you kind of have this so you could say right um let allow me to read the service catalog of nodes of console nodes for example right you can do that by api path so it would be like what v1 console v1 uh service v1 catalog nodes i think that's what it is okay that's interesting yeah yeah so you could have that um and that would you could specify that kind of that kind of access control if you wanted to right yeah um however the acl system is a layer on top of that that just maps makes it easier to map that kind of policy control uh to the api so you would want to use these policies in some ways because node prefix right won't change for you as someone who's configuring console um but the api the underlying api might change and in which case you don't have to worry about it so yeah all of this policy these policy rules map to some corresponding api endpoint in console gotcha and so this node prefix is this saying then that any endpoint that has to do with nodes that you can read is that without saying yeah and then this one's saying any service with regardless of how it is you can read yeah right and then query prefix we can come back to that one but let's say that i wanted to limit our service prefix let's say i wanted to say all right uh i only want you to be able to read from the api like how would we put that there well part of this comes down to like what part of the api um i don't believe there's any kind of escape patch if you know what i mean where you can just say like oh you must have this part you can just read generically from any part of the api as i recall it must be one of these very specific rules so it's either like session prefix it's always something like something prefix right which is uh more generic and then you got service app which is more specific right so you will notice that the acl system the acl rules always have the object structure and you can either be really specific about the object that would i guess be the equivalent of like a get slash service slash app in this example um or you can say get slash service which is service prefix right so that's give me all the services related to this you know this thing so to speak um so it's more like a service prefix is kind of like a list from an http api kind of perspective and then the more specific service kind of declaration would be something that's more like a get a very a get by id first okay gotcha i feel like this will be something better done to example so maybe let's get the get the um baseline up first and then maybe we can tinker with it yeah because i don't think there's an escape hatch that you'll you can just say just lock down this part of the api uh doesn't quite work that way um if you can think about are you familiar with gcp uh identity and access management engine i am not okay aws aws i am for example right aws i am you don't say i must access v1 slash or i am going to restrict access to v1 right what you say is let me attach an i am policy that's like you know allow read compute instances right yeah or allow vm instances or something like that uh or s3 bucket right read from the s3 bucket right from the s3 bucket you don't declare the ac the the api specifically okay because the identity is assumed on the policy that it's on to the principle that it's attached right okay so they don't let you configure that right i mean aws im doesn't let you configure like you must restrict it to this aws api specific get only put only etc so yeah they generalize it more so this then if we were to slap this on the api uh well on on the the node for the apis this then is implicitly talking about the node it's on and the api because it's there yep so it's the identity okay yeah that makes sense that that makes sense okay because i was reading this as these are generalized across all things and you would need to uh what does it uh specify the service in a specific node okay so does this then mean so we're going to stay at this token for ddns does that mean that we need to begin on the api service or web service node um yeah so you probably should start with the i would say web let's start with web because that's the upstream service so okay and we don't need to do anything for the the any of the servers we should not need anything for the servers okay interesting you should apply wait you should apply the token to the console agent serving dns requests so when the dns server makes a request it will include the token in the request so you can either authorize or revoke the quest okay so yeah let's try this on web yeah to con serving the dns request i guess it would be i guess the the api node would be serving dns requests for the api right but when the as server makes the request includes the token in their request authorize or revoke the request depending on the token's privileges token for dns is the same three-step process you use for agents yeah so i would put it on web first because web has a policy already right the web client uh yeah they they all should yeah yeah it has the how do we get that into that here's just policies um do we not have acl policies uh maybe we can figure that the last time we did did i i put it in here we did configure it so for those who are checking out and we're like where is the policies last time we configured some specific policies oh no okay so we've got that in there is that the token that is that the global management token no no this is the individual one for the specific for the uh for the api um i don't i don't recall i remember just making the tokens but like so here's the identity for the web and the api oh here's where we looked at it yeah yeah yeah we did an inline policy okay yeah we did the inline policy i remember this time okay so we could be so we should and actually it has this already right yeah yeah it does so hcl's then go along with the token yes isn't that how vault does stuff too the it does okay same same very similar system oh okay yeah that makes perfect sense to me so just for my own understanding if we were talking about um aws i've got to draw an analogy for there tokens would be like your user policies would be like the policies that you attached okay perfect alrighty so that means adding the policy i was about to add would be redundant because that is already the one we've got here yeah so that's interesting um let's see what is the uh dns request dns request next create the token apply the console agent serving the dns request so do we have this to default oh you can use the who is serving the the dns request it is console client right that console client there that is a good question so [Music] the service registration has that token but can you show me the console client yes if you go to the sorry if you go back to console ui yep and then you go back to tokens um can you pull up the node identity for which one is web i don't okay uh two five five three eight okay yeah so that does not have node prefix policy in there i think we have to set it on here okay gotcha so just just for my for my underst just uh from my understanding the goal we're trying to accomplish here is getting it so that the web and the api can interact with each other yep now what is the barrier that's keeping those from interacting with each other is it the acl on on the both the uh first up is it the is it the acl on the nodes token or is it the acl on the individual services token it would be the acl on the nodes token because it's saying to add it to the dns server basically the the agent that is serving the dns request yeah and so remember when we set up service discovery the first time the service looks up the dns with the console client console client then returns that information yeah that's right that's right yes gotcha so we need to add an additional policy to the acl for the client gotcha for and for both clients for that matter well i think we could we might just be able to get away with doing it just on web because api doesn't look up web web looks up api ah okay yeah perfect alrighty so that would mean how do we how do we speak that here well if you go back to the acl rule it's node prefix and service prefix we already have a service prefix read so that means that it's saying you can read any of the services in the catalog the node prefix though is something that we're missing we have a very specific node one um we need to be more expansive we should allow it to look up all nodes okay yeah because right now it's you can only write to your own node and we needed to say but you can can come on man can i not edit this right here why am i under there whoa hello no interesting i guess this is a read-only box i suppose it is uh that's a ui thing that we are probably gonna have to go back oh wait here i could just add a new policy right yeah you should probably just create a new policy anyway because if you say it's just for your own readability because then you're going to be like which ones have dns access and then you can get that policy really quickly yeah why don't we just is it okay if we do it in the ui here yeah of course yeah all right so let's call this policy what's good dns i think it was uh the dns requests uh dns requests or you could do dns and then that is it i would add the service prefix as well just because if any future acl and any future node or service identities need access to dns then you already have it configured gotcha and let's say that another ace token has this rule in here they won't they'll just they won't okay yeah they won't over why am i failing to find the correct words for that they won't step on each other's posts yeah yeah but i think it will always choose the minimum like sort of the minimum the minimum privs versus the max no it will choose the maximum privileges i think but don't quote me on that okay yeah gotcha interesting okay well let's go ahead and do this i don't know for sure anyway i think that's all you need for now we can test it out later okay so we have created and applied ah and here is this one yep so when you add a new um policy to a token for that agent does it have to be restarted or will it just pick up on those permissions um i believe we'll have to pick it up okay so this does not seem to be yeah i think you'll i i sorry i believe you'll have to save save at the bottom if you scroll down oh i didn't say it yeah yeah okay well maybe maybe we'll see here though you may have to reload the the client but actually it should just stay here right yep oh set agent token acl set agent token default dns token does the ui not do that for us because it looks like it's applied here go back to the guide well what you're doing is setting the default token you see my individual tokens for them yeah that's fine okay so it should be set and it is using that token the same exact token yeah so perhaps i was just not i was just going too fast or i don't know save and then add the policy to the token go back to the token this one yeah well it's in there it is it is saved we're not imagining that it's saved right when when we go to all tokens it's got the little dns request policy attached here i would recommend that maybe you go on to the machine and reload okay yeah so just do a console reload oh don't forget i don't know yeah i don't have anything good for it yeah so i just guess i can just grab this here are you sure that's the right one though hold on so you're on 38. 38. 38. that is web okay yep could you do like restart console so this is the web one and then i did a console reload do you want me to just straight up do this yeah okay it hopefully won't over i don't think it will overwrite the policy because i don't think you set the policies here so okay okay can you look up on the host can you access it like curl to http api service what is the url for this again console. service dot service service dot console api api.service.console that's right and then we would have the 99 report yep [Music] not resolve host hmm [Music] that's different can you ns look up api.service.console unless we pepcact it again also quite likely uh maybe we didn't do service.console it's not service.console hold on brain one second what is the is the dns api.service.console okay yeah i'm just going to look at this thing let's see if it had any more commentary on this console agent serving dns request well create aim the thing is we added that policy yeah does it so the api doesn't need it then api shouldn't i mean add it to api let's see if maybe it just needs a reverse thing but i i'm not really sure why it would be yeah why that would help yeah helper hinder furthermore is the console clients can you double check that your console client to set as the dns resolver i know we did this before but just like can you how do i go about checking that remember our system d conf the systemd resolve resolve.conf yes uh where was that again here um i should have that in like the uh terms of the files that got set up yeah because we've set it up before but i don't remember if yeah like this either got okay so this is what was set up and all right and it was working yeah when i remove when i don't have the acls and they do talk to each other just fine so so don't think it's that i think it just must be something with the my my impression my limited for my limited knowledge here is that it is just something to do with the token policies right yeah i'm trying to think if there's anything how did how did we generate the uh how did we generate the um node identity policies again did you do those manually generate i didn't know i don't i didn't the the yeah these yeah i didn't for for the tokens i just generated them doing acl bootstrap and then doing the ones for the node identity and the service i did any but in terms of the policies that they're coming with by default i haven't done any of those okay could it have something to do with us needing to do something with the service identity policies um i don't think so because it has the policy in there so the suggestion is that maybe we try setting the default token on there um we don't have the token dns token but maybe we can um can you go into the client console client ssh into that yeah which one just uh web yeah web is fine and then can you do console acl token create dash description yeah well we'll just see if we can issue a token based on the policy that we have oh okay i see you can say token for dns requests or something okay we'll we're just going to reissue a new token and then yeah yeah dash policy name okay is it which one am i oh this one okay i think because we already have the policy created we should be okay we shouldn't yeah yeah you're right you're right you know requests because i don't know export yeah i don't have that yeah i don't have the token set or that one no no no do it with the global global token oh because you won't have permission you won't have permission remember you won't have permission yeah uh what is the name at token console http token yeah there you go okay so there is wait was that the correct token yeah that's it wait oh i guess i wouldn't know yeah okay just checking all right now i should be able to do this yes so now we're just going to set this as the new token with the set agent token default and hopeful that it will work yeah we're we're just going to see what happens here because then [Laughter] uh okay so then you have uh acl set agent token default and then take that secret id yeah that one mm-hmm okay all right so then let's do dig at 127.001 space dash p space 8600 oh port yeah yep and then uh was it web web.service.consular something like that oh wait or sorry api.service.console right so we just want to check if it if we can even access it okay so then we got the answer that's good okay so now can you curl to it so if this has broken uh it is because our resolution is probably broken not because we can't access console now so yeah it works okay so now you should refresh so this should just be working then now yeah there you go okay so we just created a new token and added it so i think the problem is that it doesn't do an in-place update of the policy for an existing token ah that's gotta be it so yeah because of this dns request policy is not okay can you refresh this page yeah yeah here's the new one okay yeah see that's yeah i'm gonna see what we got here yeah so it's just the same thing so the problem then it seems that for the for the uh for the node one that we originally are doing it even though we have the same stuff on here it was not working because we didn't get it updated in place can we go back to using this token again and do whatever we need to do to get it to update um well i think unfortunately we have to just issue the new acl so in some ways i think we even like you would have to just create you could duplicate if you click well oh since this has since this has the policy shouldn't this be usable yeah try it yeah yeah copy it back and then reload it yeah like i wonder if just setting it again will do it where is my theme here there it is all right and it works yeah i guess new token interesting okay so that that makes a question here of uh in this case we had gone ahead and set the token here right is that the correct token uh well go back to 38 and then open yeah six it's not oh no yeah you said right oh because that's the service sorry brain yes that's the service token oh it is so that one is right okay yeah that one's right where is your where is your console config yeah okay so that was there there because that's the that one yeah okay so does anyone in the chat then know how to after you've added a policy to a token making it recognize someone says that it may be that originally when you set up the client you set the agent token but not the default token so we might have to set a default token like force it to have a default token yeah because i'm looking at the i'll paste the i'll paste the uh command in the chat or the documentation in the chat but uh it looks like that there's token types there's a default token type and that's what the agent will use for internal agent operations uh as well as the http and dns interfaces and then there's the agent token so it appears that we set the agent token but not the default token and so yeah that's what it looks like yeah so then okay cool so i want to try something real quick just what i want to do is uh update this with that new value with the default token but then i would like to just for experimentation if this is cool with you unset the one that we've set in the cli and then just mess around with the policy and see if that hot reloads the token okay cool yeah so for those recapping we're seeing if we do a reload uh you know we reset the policy on the token now that we've set it as a default token uh we'll at least reload it and see if it actually picks up the policy real time versus so in practice you would leave both of these then uh you could probably just set it as a default like and just get rid of the agent all together yeah but i don't know if that hold on before we do that acl i have to look at the config i just don't guarantee that anything like i will never remember this it's just like eventually i'll remember i think but it will just it just takes me a while there are a lot of configs there are uh where is the token agent ah okay yeah tokens holds default too so maybe we could use we could set this as default instead of agent and so just get rid of agent all together i wonder in what case you would use a separate asian token from a separate default one well if you're not using dns for example right when would you not do that with console uh for a couple reasons you know you might choose to do it with service mesh in which case your service mesh doesn't necessarily need oh dns resolution right okay well there we go all right goodbye agent all right so we've got that how do i go about doing the opposite of set agent token can i just actually if i just restart console it should lose this right right okay sweet yes let's do that tidbit from chat agent tokens are only used internally by the console agent and the default token is what the agent will use to service external requests okay gotcha so do you so whoever answered that do you need to set both we'll we're on a bit of a play so we'll see yeah i would just restart and then we'll see i don't think you need to set both i think you could just set the default well because like doing that worked basically default will do everything right so if we're doing service discovery we might as well change our config completely to default rather than agent token yeah yeah cool well that worked this is working just fine now so now what i want to see is if we go into here there's one way that we could test this yeah i was going to just remove this policy real quick see if it breaks it and then add it back oh okay does that work i don't know we could try it i just wanted to see it the chat depends the answer is depends cole oh okay depends nice those are my favorite answers depends we're all just learning this last time we we bro like a couple a couple streams ago we broke console and we ended up doing outage recovery for no you know for with good reason but like it was very entertaining so ah look at that it did it did pick up the change so we do need default i think that's what it comes down to if you're doing service discovery you should be setting a default service token probably and now it works again nice yeah so it will pick up changes to your policy that are changes to your tokens policies without you having to reload it that's right we just made the mistake of perhaps setting an agent token uh when maybe we should just set it as a default token because we're doing service discovery and lookups uh let me uh go ahead and fix that in the api or the api's node oh what was that carrot that's what it was all right well i feel like i have a better grasp on tokens and policies now a little fuzzy on the configurations because every time i encounter 80 pins obviously it's always a scary little monster in the in the back of the mental cave but anyhow um in general your token is how you authenticate yourself between the the ecosystem of console and the policies are what determine what it is that you can do with the api and nice cool well where does that leave us in our exploration of console acls uh well do you want to go more into acls do you yeah i would love to learn the language some like like writing a tricky clever one would be kind of cool just so we can explore i think you mentioned something doing with key value store to explore that or or yeah we can do that yes key value uh another tidbit from chat if you don't set a default the agent will use anonymous token for requests which basically means no access thanks so that is why we still couldn't yeah that is why we still didn't access it acls that's so complicated okay um all right so we just to demonstrate acls because what i did explain to cole was that we probably will have a difficult time demonstrating sort of syntax of acls uh with just the abstracted policies so like for example you see note identity service identity those are hard to identify and fully understand um so what we're going to do is use console key value store console has a key value store naturally this is an extension of the service catalog right because services are name of service identity to location of service it is a key and a value thus most of what console includes is heavily oriented around a key value store um and so then you can use said key value store for your own purposes uh cole would you like to know why you would use a key value store or shall we omit that discussion of course uh so you can use a console key value store for a couple things if you are let's say someone who uses vault um you could pre or you could use a console as a storage back end so vault would use the key value store to store information about vault encrypted of course you know but you could use it as a back end and the key value store you make changes it uses the same protocol under the protocols under the hood to achieve consensus on the said values so for the most part you can use it for a number of things it's almost like functionality that you might find in like fcd for example and that's a common analogy you can also use key value stores for various configuration storage and in the past i've used it to store like spring boot application properties for example i've used it as a feature toggling um feature toggling mechanism so i have my experience with key value store comes mostly from a sort of a development perspective and not necessarily from like a systems backend perspective so the way i learned key value store was literally and key value store the way i learned acls actually came from key value store because i needed access to certain feature toggles uh and i might not have had access so you'll see more of it look a little bit like the what cole is familiar with which is the vault policy engine vault acl and policy engine yeah yes you don't have to raise your hand it's just easier that way i'm not like yo what's up okay but um so on the key value storage i know that it's it's not as it's not i don't think it's like touted as the default backend for vault anymore correct okay that is correct if you're using vault uh the recommended uh the recommended store going forward is raft or it's the integrated storage um pretty pretty neat uh yeah i mean for a long time vault and console were kind of hand-in-hand towards storage back-ends you're perfect you are perfectly i will say this though if you are using console as a vault storage back-end please use a separate instance of console for that um i don't really think that you should be setting like your service discovery console as the equivalent like storage back-end console either uh i would have some concerns about that so uh yeah yeah and so from my understanding what makes it a viable back end is the fact that raph storage is kind of hard to kill if you've got five servers up they're going to be syncing that data if a couple of them go down you don't have to worry about losing all your data because they're already cheap they're always keeping it up in sync cool nice all right so that was my first question about that my second thing is what i would love to do uh rosemary is create a couple key values and then make it and limit it so that the web can only access some and the api can only access some can we do that yeah we can do that um so let's do this okay so let's create uh like so let's create the key value okay so you're under the global management token so this means you can write to the key value and you can read from the key value and you could do it whatever you want to send inside key value um so let's create a couple of keys and values because why not that just sounds like the thing to do i was like where's the create button and it was hidden under the little zoom thingy so here it is all right yes uh yes so i will let you create a set of keys and values you can create key and you can create like folders so you can do like pre you know you can do like kind of like a folder directory of sorts so you can do like so i could do like rosemary secret yeah and then you i wouldn't put a secret like that in console because console does it all in plain text but okay oh no no i mean it's not gonna actually be secret i don't know right so my understanding is not going to be secret no yeah yeah okay we'll just say it we'll just call it thing i feel like that's pretty generic so if my understanding is correct this would create a new folder with a key thing with a value hello right yeah so you could do yes that's fine that is good okay yeah all right so we'll make one of those and then we'll make one whole thing and now we've got two rosemary sorry i'm in the chat [Laughter] you know what i'll answer it live because it's just easier uh so we've been we so you know we were chatting briefly about vault back ends vault storage back-ends and someone mentioned that it's better you know in the docks uh it is better to use i guess the recommendation now is to for vault specifically is to use integrated raft uh which is the integrated storage in so yes that is indeed correct it is better now um with good reason with good reason so yeah everybody it is very it is very confusing because raft is effectively like writing to a local file but it's not um and so it's uh it's a little bit more um complex as a system uh so it is you are indeed correct it is the recommended way to to set up vault storage back end going forward sorry yes had to answer that in the chat uh because i know we were there's been some chatter in there so i was like let's we we should just chat we should chat about that uh all right so we have keys and we have values now um all right so let's try a couple things right uh let's go back to your policy or you go let's go back to your tokens and then let's uh create a new token we'll use this as sort of like a um yeah create uh we we are we only by the way for those who are looking at this restrict this token to a local data center we only have one data center if we had multiple data center then you know this would be more important but we don't so we we haven't gotten to that point uh so let's create a new policy right uh you could call it key value testing kv testing something like that okay all right so let's talk about rules um in console the acl system uh offers a set of predefined kind of um rules that you can use so the one that you're going to be using today or let me just find at the actual config uh is going to be kv console kb gotcha yeah so kv has yeah it's on here somewhere oh that's right here it is it's right down under here gotcha yeah okay so here it gives you a couple of things there's deny write read and list right um those are the main rules i mean most of them have these rules most most of the policies have these rules which is deny write read list anyway so you're uh you're pretty much good there some of them have some nuances and some differences but for the most part that's really what you're going to those are the the the actions that you can take um all right so key prefix uh key prefix remember prefix is kind of like everything under there right so in in a sort of file structure it would be like cole star right read everything underneath there uh you might not be able to list but you may be able to read a top level so let's actually give that a try so let's copy that and then we'll try to read from cole okay and so if i do this this effectively says this token has the ability to read anything under the whole folder right but there are some nuances to this so but we're going to just do this by example it's just you know that way you get a sense of what what the the semantics are looking like uh so create and apply yep and save yep uh grab that token um you can choose where you want to go you can go on the web you can go on wherever you want yeah cool you can just go yeah this one yeah so you can do export console http token yep uh and then i believe if you just make sure you do console cave is it kv console kv help can you do yeah help make sure that that is there yeah there we go okay so yeah yep get and then call nice there you go and if i were to do rosemary thing no we cannot which is perfect right yep nice yeah okay that's the reason forward right so the question is why is it allowing you to read everything under coal thing um well it's because you set a key prefix right so if you go back to your policy uh you set a key prefix for coal um and so key prefix means anything under that folder structure quote unquote uh under that api path right so that's really it's just mapping to an api path uh so everything under that api now if you change this up i don't think you can edit them in line here i think you have to just go back to policies and edit the policy but i i don't know yeah i think you're right oh no oh yeah you can wait you're good you're good nice okay um that will kill it yes wait is other other is that your is that does that exist that doesn't exist doesn't it it basically will tell you it won't can you just change it to key so remove the prefix just just oh oh okay yeah yeah yeah exactly and then save that oh it just does this okay yep okay cool and then try it again right so to be really specific you can narrow it down and you just do a read uh you can also do one other thing which is if you go to uh back to your policy um and instead of doing key col you can say key call and why don't you just do um list policy list don't don't touch cold just policy list yeah list okay yeah so what do you think this would do my is there even a list function command under kv okay well what imagine would just give us the keys not the values right so okay let's see yeah okay where is the list command here there is none you just oh or list yeah it's again i see well come on do i need to like oh maybe oh no wait your policy has a slash can you get rid of the slash in the policy i forget about this yeah you need to there we go okay try again well i guess technically that it doesn't have keys because it's a file path so if you go back to the policy sorry okay and then what was the call slash what other or something was that what it was it was it was stain think okay yeah then you can save okay so okay yeah well let's just do with this okay so let's try again so should i do this slash here or do you think okay that just straight up gives me the value right well list i guess is technically the same as read in this situation interesting you can't get that so actually you did key prefix but remove the thing okay yeah that's what i was gonna ask yeah okay here we go let's try this again i expect with this for it to just give me like a list of keys like to just say like thing and other uh can you remove the slash oh just from the command yeah well that doesn't help policy settings in the server excel config i think maybe the slash is throwing it off yeah i mean what is the list thing supposed to do here like what is list supposed is it supposed to am i supposed to be able to oh it looks like a curse right which i don't actually you would get a list of key values i think you have to just remove the slash okay acl i mean the example they have here has the slash in it right but that's for read so it's reading any value underneath it right oh okay yeah okay okay so let's try that again yeah it still doesn't like the can you get coal slash i wonder i don't know lovely um so i'm kind of losing context of what we're trying to do here what are we trying to do basically it's just playing around with the acl policy i don't know to be honest i don't think list will help us very much in this situation let me try something i want to just make some more values under this just so there's more than one thing other other so now someone's recommendation is is list policy setting in the server acl config not sure if that's enabled by default that's a good point so you're saying in the server that the api or yeah that the api lives on yeah so the server i think it's actually the console server itself so console server needs to have acl enable key list policy because by default it does not uh it's not turned on which is why we don't actually have access to it so here no um the acl if you know remember in the server config your server console.hcl console.hcl if you go to console server to the actual server here yeah and then you print out console.hcl or config yeah okay so under acl we need to add a field that is enable underscore keylist underscore policy okay what does that do so that will enable us to use list so apparently list is not turned on by default as of console 1.0 so okay well interesting so sorry what is that field called again enable underscore key underscore list underscore policy yeah i guess it's like if you don't want to recurse allow people to recurse through keys maybe that's yeah that's the point you know you just explicitly set everything to read and write remember you can do a reload console reload maybe oh that works that works for you no i don't try i think you have to set the token anyway oh well that works too yeah i had the tokens so i remember this was also and it depends okay yeah um i think right now it's just the actual policy itself i'm screwing this up uh possibly [Music] let me i just want to see because the example they have here has this under read yeah well we could just do everything with reads we don't have to use lists at all so if we do that you save it you come over you just won't recurse we're still not getting how did they do it in here they have it with recurse okay so this is the actual example so this should do it you curse okay there we go yeah so there okay that was fine okay well you can achieve the same thing so that's good yeah yeah okay we yeah i don't think we could do a reload i think we had to do a full system restart but it's fine i don't think we need it really it looks like you can do the same thing with reed yeah so cool okay okay um so let's actually try writing something you want to do that okay yeah yeah so this would be that would be the put right so kb put something like coal um another one yeah yeah that works and then of course that won't work right so now write a policy that can put another one so if i come into here if i wanted to [Music] curious can you stack same policies like could i like turn this into a list with read and write no i don't believe so oh okay so then i would have to do a whole another separate one oh no wrong braces actually i'm looking though couldn't hurt to look i don't believe you can yeah i think you actually have to set them individually okay so this should women work yeah right someone's suggestion is you another one bites the dust as a value nice and now we have it yay so there's the right so that leaves us with the yeah let's do it so let's let's uh first go over here let's deny anything from rosemary i mean it's already denied right so i recommend you that's true i i recommend you deny something that's nested under coal so maybe you deny another like deny the read of another so you can write to another but you can't read from another okay interesting so we'll do key so we're going to be doing explicit key this time and then policy read no denying does this look great deny it just kidding yeah okay okay so now if we do this for the recurse yes we only need you nice okay well simple but let's change it so if you can do console kv put cole another and then change it to add one underscore bites the dust yeah we'll just do that dj khaled what oh well it doesn't like him it is restricting it so deny right is restricting all yeah that's right it is just straight up a deny yeah so i guess that that takes care of both read and write ah well that okay well that makes sense more you know the more you know yeah okay so then that leaves the list [Laughter] i mean we got what we needed to do by doing work yeah so yeah i mean i think i think i get it are there any other areas of this policy language that you like if we wanted to do something like um what does it break the connection right now between api and web is there a way that we could go into the one of these service tokens that we haven't messed with yet and put a policy that's just like hey don't talk to the api so no right could do it you could do it by breaking dns right so you could remove the dns request capability and that would stop it from communicating um intentions don't quite work this way oh okay yeah so there's a separate uh functionality in console we haven't talked about yet which is intentions intentions is effectively the network policy um capability and that is a service mesh capability um it is not something that will be while acls are a requirement so so the funny thing is in order for intentions to work actually you do need acls you do need some form of uh the acl mechanism right uh on and that's preferable actually you can do intentions without acls on it is possible to do it um you just have to be explicit about the the the rules when you turn on acls intentions denies by default so um it uses a deny a deny by default policy for intentions uh but intentions only works with um service mesh because intentions depends on envoy proxy and the envoy proxy filters so just to clarify my my question here was i want to do something like could i we we like the the api for example has a policy on it right it's got this default one and if i wanted to say yo don't let don't talk to web there's nothing i could put in here to make that happen you could try something i don't guarantee that it will help but we can try something okay you could put service web deny and then maybe what that would do is stop console from resolving to that service okay so service right policy tools deny yeah you could try that i don't know for sure actually wouldn't that need to be reversed so web going to api i don't know would it yeah because like right now it doesn't work anyway i suspect cause okay yeah cause it's still that didn't do anything that didn't do oh wait because i didn't save it let's see well it wouldn't know anything better right because it's not necessarily if you did want to do a deny you would have to go from the upstream to the downstream so that would have to be a policy you put on api or on web sorry on web not on api so let's go to web and then apply this well i guess so we would need to tell it to deny api right okay deny api uh service do we do api here or what yeah it's apk we do the api right because you are on web right yeah i don't think this will work to be honest um yeah it's kind of like cats out of the bag um you could you could tell dns you could change the dns request policy to basically not resolve every surface prefix right yeah and by the way that yeah you're right that didn't do anything yeah so what you could do is go to uh what you could do um is go to the uh client the note identity okay for the for the club for which one weber api for web okay yeah and you can add the deny web deny api sorry add deny web api on here yeah okay yep okay so now try to do the resolution nice there you go okay cool so basically if you add that right it's saying you can't look up you can't look it up so even though it has access to dns you're telling the acl system that console can't uh the node identity of web can't look up api yeah so you can get to the front door you just can't get it yeah however api can still go like it's not bi-directional so api could still go look up web and whatever it needs to do okay yeah so you would have to apply these yeah to the other things so five although would that even make a difference here if we did web like in i don't think got rid of of api i mean if you try to resolve to web it will just not resolve but it won't like yeah stop the yeah i don't know if it works yeah yeah i think still it still works neat so you can to an extent control the traffic um between your services uh with acls it's not i don't even think it's controlling traffic it's that's not even traffic it's just yeah the resolution yeah yeah the api it's like resolution you can control which service is resolved using the acl but you can't really fully control the traffic right so it's not enough it's not enough but it's uh something but you would need to use it in concert with something else to be truly secure and block it off yeah exactly and as someone pointed out in chat thank you if the web if web the service cached the dns response it wouldn't help right so if it cached the dns uh the dns the lookup right it can still go to the ip address it's not actually block you know so it's basically the application is blocked is the application layer is blocking not any of the actual networks and stuff right yeah gotcha okay that's pretty neat um that's pretty much acls like at least the ins and outs of acls so quick question then what like so for these acls i feel like the usage for them around key values is pretty straightforward right because that i really like this as a way to just get our get it under our fingertips and i understand now the usage of these of acls when it comes to the actual nodes themselves i'm still kind of fuzzy as to what the use cases are for acls with the service identity so if you pull up the service identity specifically right it can get up uh get information about itself so it can change information about itself um when we turn on service mesh this will be more important but it can change information about its proxy so that's why you see website car proxy and the service has access to all services so that's service prefix policy read yeah it has access to retrieve information about it it could be for a variety of reasons it could be for a resolution it could be the service itself right resolving to something else um node prefix similarly right is also kind of for resolution not dns per se but just understanding what are the other services available what are the node prefixes of other nodes in this cluster that are available to me gotcha okay so that makes sense um so yeah this is the this is the one for the web here uh i kind of think if i had other questions for this trying to think of an example case um i think that might be all the questions i haven't for some reason it still feels kind of fuzzy because there's a couple of these like the pens and oh i just was the this is the question so the difference between acls and intentions then what is that so it's architecturally uh different so acls are restrictions to acl access control lists are restrictions to consoles api so think of it as more management right it's restricting access to things within consoles http api as well as you know just general console uh related activity so someone pointed out in the chat like there's a we don't go into this because it is more complex um but there's uh such a thing as uh console network coordinates and basically these are ways to identify the nodes in the cluster and you can use them for a couple of different things like failing over there's a lot of complex patterns that you can do and that's where there's a lot of other other deeper functionality that we don't cover um but those things are controlled by acls right so things that are about like um the console network the console management control plane itself uh are going to be based on access control lists the difference with intentions is intentions from a technical view uh kind of only work when you well actually only work if you have proxies in place so uh intentions are made to pass configuration to proxies um and they are they are they are managing control from layer seven and layer four right so they have not much to do with console management network connectivity itself or api authorization itself they have everything to do with service to service communication specifically uh okay yeah so for example it's we've conflated the two a bit in this example because web is on a vm and api is on a vm yeah right and so web and api together uh they are service they're doing service to service communication with service discovery right but there's nothing as you pointed out there's nothing stopping them from communicating directly to each other ip2ip right in the case of api itself it's just the api so intentions what it will let you do um kind of to a certain degree is that if you put a proxy in front of web and you put a proxy in front of api and all communication between web to api is going through the proxy intentions allows you to configure the proxy to truly stop the network traffic between the two okay yeah okay you can think of it as like a layered like firewall of sorts like you're adding a bit of a firewall in front of these things uh yeah you know intentions are very much like intentions are very close to are very close to acls right uh but they're usually for two two different purposes intentions are for the quote unquote data plane so service is communicating with each other of console acls are for i guess the management plane the console control itself yeah cool so we've got two post offices right and then we have a delivery service yeah the acls are like the in-house postal office rules like we're not going to we're going to take packages like x y and z the intentions though are the actual delivery of those packages elsewhere those are like all right we've gotten to the other post office but we're not even going to put this in the door yep cool all right nice is that less fuzzy yeah no it's a hell of a lot less fun it's still a teddy bear but it's a less fuzzy teddy bear well you know it's a little bit like and i'm trying to make it the analogies because it is important to recognize this from a conceptual view of the differences right so acls are aws identity and access management right so aws im policies rules restrictions to the aws api those are things that are management related aws management related intentions are kind of like an aws security group okay that's that makes it crystal clear there you go all right nice anything else um we i know we had a aspirational let's do an external ca but i don't think we have time for that because we have to create all sorts of weird things for it and in a way we've kind of already done that and we could i could just go ahead and explain kind of the bare bones of it but yeah if you could did yeah but i did have one more question on the acl stuff um and that comes to the sort of sort of the approaches that i touched on before that made us 10 minutes late and that is around automating the the bootstrapping of the acl system that seems obviously doable with a set of hacks and retry loops and or other nifty conventions but it seems very difficult to automate and i wanted your just just a discussion around it because there's obviously reasons why it's difficult to automate that that are valid um but yeah that makes sense yeah so it's the bootstrapping specifically right it's not it's not really the distribution of acls because once you have it started you're pretty much okay the problem is seeding the cluster with the right acls in the first place um and so i think the the real challenge is like the the bootstrap acl workflow is is a command um it is a command and you have to you basically have to go in and you have to say like i am ready to set up this acl and then grant me this acl token there's a lot of order of operations here that isn't fully captured when you have a 100 declarative infrastructure as code automation component which is what we have here right um exactly and you may not want first of all you may not want a completely automated bootstrap mechanism uh because you have this bootstrap token that is gives you everything you can do whatever you want to uh and this is something that's uh if you actually look at this workflow on kubernetes even um if you deploy console on kubernetes the bootstrap token is saved as a kubernetes secret right there's no way around it uh you need to create it and you need to start the cluster with this token um and it is automated in that it is like a command that is generating this token um but it is out it is written out to a kubernetes secret to store that token for further usage right um so the i think what we talked about when we when we brainstormed how we would automate this was like well you could get the bootstrap token right run run something in terraform get that bootstrap token and store it in aws secrets manager or or vault or whatever and then read out that token for future usage to generate other tokens um that sequence is something that is as you pointed out requires retries requires a blend of imperative commands in in declarative setup and that's kind of where we got a bit stuck yeah so my question is let's say that we did have that somehow magically set up to where when you start from scratch and it all pops up the acl uh domain master token is generated when whenever we have a new node pop up and whenever we have a new service within that node pop-up those also need tokens generated with that root master token now surely in existing console systems where that's already up how does one go about automating that so some of this is some of this is a lot easier it's done is actually a lot easier than it is uh it's not done manually yeah so just as vault has auth methods console has auth methods okay and the auth methods are primarily organized around automated acl token issuance right so you can do something like the jot auth method in console that can issue a an acl token right so it registers and it says hi i have authenti i can i'm authorized to communicate with console give me a token and it will just get that token um so there are yeah so in terms of node and services both uh can get that it can have that kind of workflow um in fact that is how for example kubernetes approaches it gotcha so just for my clarification that means that there is an api once console's up and running that services and nodes can connect to and get their acl token yeah perfect okay that makes sense yeah okay in in the case of aws that's not really possible uh oh aws is done kind of manually uh you know you have to uh in the case of aws you probably have to do some kind of acl generation using vault um or something else that will issue uh tokens because there's no aws auth method um yeah so that's kind of tricky in that there's no aws auth method so a lot of what you might do on aws specifically might involve automated automated issuance you know or some declarative creation of the token to pass to your node okay so in that type of system it would be you have an existing console cluster you also have a vault deployment you have a new node pop-up that wants to have connect to the console cluster and also have some new services it would need to go to vault first to get its acl right exactly okay interesting so vault then that that to me suggests that vault in this example is configured to work with the console cluster okay it's configured so as someone actually someone mentioned this in the chat so how apropos um you basically would use the management token uh or you would use a bootstrap to generate a management token for vault and then vault would be able to issue the acl tokens on your behalf so you can yeah so because vault has a console acl secrets engine so oh oh okay yeah so so that we don't have a whiteboard for this because it's kind of weird as a workflow but basically you use the management token from console to set up the console secrets engine in vault uh vault issues new acls as well as the specific policies associated with them when your virtual machine comes up it uses vaults ec pro either ec2 or im whatever you decide to configure ec2 or im auth method to retrieve information retrieve a new acl token from console and use that in order to configure its console agent which then you know then it's happy and it's it's going to come up so the automated workflow without a native auth method uh in the case of console means that you need to defer it to a secrets manager or something else right gotcha that's preferable okay so that okay so that makes more sense then um and the reason why that works is because you can have vaults set up to look at for example the ec2 instances im role right and like say yes you are who you say you are therefore i will issue an acl to you nice right yeah okay well that makes sense it's a lot more work than i than i thought would go into it but on others i think you know i don't and i know i don't know for sure what if there are or if there are thoughts on this necessarily i will say you can use consoles consoles acl auth methods uh for example for azure right because azure or gcp they do oidc and jot based uh they offer oidc or jot based authentication uh you know on their machines as part of their instance metadata so federated identity is available in the form of jot noidc so the result is that you can do that you can use machine identity or federated identity on the machine to authenticate to console without needing to do the secret secrets management piece um it's just aws you know if you do this you may it may it works to your benefit to set up external secrets managed somewhere else yeah gotcha okay nice um well that that sums up most of what i had around acls uh obviously there's there's a lot more to the lane with the policy language that we could explore but probably not for today uh was there anything else you had on that uh no i don't have anything else on that we got the basic rules down those at least in terms of usage basic usage um these are the main ones you know you would pretty much do like key probably service and nodes maybe those are the main ones you might have a couple other ones that you use um but those are the the big policies you know at least the big policy rules that you probably would um probably would use so it's kind of tricky but you eventually get a get a sense of like what to do with the policies in the acls um most of the if you're doing this on on other orchestration platforms you know you don't necessarily have to set this up yourself it's kind of automated um this becomes a little bit more important in the mesh space though because for example the gateways whether terminating terminating definitely needs uh acl policies so there are certain policies that need to be created uh and tokens that need to be created for each of these components on mesh which is we're not talking about that right now we're gonna we're gonna layer that soon um yes gotcha that will be november nice so in other words we will revisit these more in depth when we get to that part until yeah cool pretty much all right well i guess that brings to our bid on the external ca correct yeah yeah we can just talk about it briefly we were going we were a little ambitious we thought about doing vault with it but then it's just like between the amount of time we need to take to settle vault and the amount of time left in this i don't i don't think we have time yeah i don't think it after doing it external from from console i also don't think it necessarily needs it just because the way you would go about doing it with any ca is pretty much the same everything's going to begin with your root certificate authority so the thing that signs off all of your other things so we kind of have a root ts ca in terraform here because we're using the tls provider so the the pattern is going to be this no matter what ca you use you first need your root certificate authority so it's going to be your initial public and private key that are then going to be used to sign all other public private keys and well by public key we mean public certificates from that point forward so that's always the first thing you're going to do um who you do that with it kind of doesn't matter obviously you can provision certificates inside a vault you can provision them and anywhere else that's always going to be the first step and then the second step is always going to be to generate a public or a private key and a public key for every node that you have in existence so in our cases here we have server and then we have two clients so each of those are going to need their own public private key right once you've got those you're then going to take your ca key and you're going to sign all of those you're just going to sign all of them and then hand them out to all the nodes and put them in the configurations and uh point to them your configuration files uh when you're given uh using these settings that we have right here and that is literally it in a nutshell obviously the the splatter code on the screen makes it look like more but that's all you're going to do with no matter what ca you use so yeah awesome well so yeah i think we're pretty much good on that we can close early today at least uh so yeah so for those who are um checking it out we'll add the just we'll add the links to the description of the stuff we use today uh appreciate all of those who who are participating in chat i'm getting a lot of interesting information uh and we have a lot of congrats for people who've either finished projects uh wrapped up different projects or like finished their deployments so congratulations uh we you know we're glad that you're tuning in with us uh and and you're we appreciate as you struggle through and and uh build with us so um good luck on i know some folks mentioned that they have other upcoming projects as well related to console and vault and or vault so good luck on that and of course you can always reach out to us if you

Channel: HashiCorp

Views: 158

Rating: 5 out of 5

Keywords: ACLs, AWS, Consul, DNS, HashiCorp, KV, Security, ServiceDiscovery, Twitch, Vault, Terraform, Access Control, Certificate Authority, Service Discovery, Key Value Store

Id: HB4u_C85HV8

Length: 93min 15sec (5595 seconds)

Published: Mon Oct 04 2021