Automating a team onboarding process to OpenShift

[Music] hello everyone welcome to another openshift streaming session today I am joined by two wonderful red headers we're gonna be talking about or the management and team onboarding with OpenShift and I would like you know I am going to introduce myself Chris short principal Technical Marketing Manager at Red Hat CN CF ambassador but I'm one the others on the call to introduce themselves because I will not do them justice so Raphael please introduce yourself for the audience today Thank You Greece my name is rafael especially I am an architect and open ship targeted so I work in consulting a try that helping customers design deploy and design and deploy open ship and then onboarding application and running workloads on other ships and Andy go ahead everyone in name's Andy Blanc I'm a senior principal consultant here at red hats I focus mainly on containers I do a bunch of stuff with automation tool ants both awesome thumbs up to those guys and gonna talk about some of the tools that you can use to not only get your organization I'm bored into open ships but to streamline that using a repeatable process that can help your adoption journey so it's gonna be a lot of fun yeah so just to let everyone know you know I knew Andrew from before I joined Red Hat you know the whole ansible community side of things so it's super cool to have him on the stream every time he's here or whenever I'm in a meeting with him it's very cool to be a part of like bringing that community feel back into my day job it's super awesome Maria like it's the same reason I stay involved in the communities community so it's great to have you both on today so let's let's kick it off for alpha hell you want to get us started right like what are we talking about let's get in it yeah so today we would like to share with everybody an approach to onboarding teams on overshift so let's define what that means right and and we fully automated approach so when we to empower the team an open ship means two provisions all the main spaces configurations policies that are needed before a team connectors start to work of the ship so what exactly those are depend depends on the specific requirements that you may have but in general it revolves it revolves around creating the correct groups the correct airbag permissions for people in those groups and then the correct name spaces which probably map to environment in your social software delivery lifecycle and then so you will have multiple namespaces per application and then you have to configure those namespaces with the correct quarters again correct potentially correct some more airbag permissions and network rules Fargo rules and this kind of thing in theory when that is done I as a user a developer can can log in to openshift and be ready to go like if I have a pipeline I can deploy and start running my application okay so for those of you who are running open check for how many of you still have Cube admins still enabled raise your hand yeah yeah this beyond that what great early on that I think the idea is you know for the audience to understand is right like we we want you to use kubernetes I mean once you use it safely and this is a great way to start doing that so if you're if you're running with cube admin kind of like with the the scissors and running around kind of deal like be careful but you know there's better way maybe better ways for your organization to manage this and this is kind of the topic today I mean I don't mind having that nice banner at the top saying you know this is a temporary password you probably should go ahead and remove that and use a real identity provider that's what I'm gonna walk through today right and and it's probably important to undal I lied that if you have a multi-tenant deployment which it's it's usually the case it's usually what we build with our customers you you have a you probably have a process to onboard teams okay it may be more or less formalized and it may be more or less automated but you definitely have to have a process otherwise this it's going to be complete cows after the 10 team joins right it's you're going to lose control of the cluster so you probably have a process we would like to share a way to fully automate it because that's that's where the Rob is I I don't see a lot of customer I've reached that level of automation where they don't have to worry about a new team onboarding because when when that situation happens everything absinthe matically and everything is configured consistent okay so that's the vegetative of today and we're gonna try to decompose this problem into smaller you know smaller things there's more problems that we want to solve okay so as we said the objectives are full automation and because we are in opposite why Lauren kubernetes now the approach is to build automation through operators all right so it's gonna be an operator based approach we want the team to be self-service so servicing everything that they need there shouldn't be our objective here is really to get what zero ticket situation where I don't have to ever ask anyone to do anything for me be a ticketing system okay fools are servicing but at the same time within the constraint of the policies and the compliancy rules that are in the given organizational company and then we want these to be low touch for the platform team right there is still up the team that is managing the platform a state shouldn't be um boarding a new team shouldn't represent work for them or toilet for them it should should be fully automated again and then we want also some flexibility right some stem ability to model complex configurations and in the demo that we're gonna give today we have come up with the complex relatively complex scenario that we're gonna see and it's just there to show to showcase that you can actually model complex configurations it's complex because while a lot of organizations that I feel and myself work in have complex processes everything from auditing compliant to just organizational structure so every trying to do is emphasize that not everyone has a simple small lean team of a few developers yeah key word mean yeah when I showcase some of the experiences that we see in many organizations and how you can take advantage of those so we have to compose the problem into three or at least these these were the three problems that we went through and we had to solve really and we it took us I would stay a year and I learned out for building these tools for ourselves too and then putting them together to build this entire process this is historically this is an area where open ship as a product does not have features right it's where it's the idea is this this is such a custom situation that it's difficult to come up with a feature the customers will have to solve it by themselves but we over time we build is is you know modular components that we can put now together in a flexible way and maybe cover I think I think we can cover a lot of use cases this way so the three problems that we have to solve are that there is a mismatch between how companies see themselves in terms of organizations and what what open ship expects in terms of of in terms of arbic role based authorizations I'm gonna explain a little bit more what that means then we have to automate group and group so I group and group membership sync from the IDP that the identity provider that the company uses to open shift and then we we have to automate the configurations of the namespace any spaces okay so for the first one in general what we see is that companies a a corporate LDAP okay which is the source of information with regard to the hierarchy of the company or the organization of the company there could be another system feeding the feeding the hell but the LDAP is our integration point right it's it's where we usually integrate it up and ship and held up what we find is that usually contains our your key in our historic information and relatively coarse grain okay by that I mean you may find line of business concept you may find B you maybe we find another layer but not not much more deeper than that and not not and not finer grain than that on the other end openshift first of all as a flight system because open ship you had such roles and permission to essentially a in in space we are using a three-way binding between your namespace a role and in a group right and these spaces are not either article they are flat so that's the first mismatch the second mismatch is a namespace represents a normally and sdlc environment so the software delivery lifecycle environment so it's really like the dev environment for an application for application a so that's much more granular level than a business unit right at least in this in this example that i am here in the picture we we have at least two level that are missing so we have for example line of business bo and then teams and applications are not modeled in the corporate held up and in in in over ship we need to model the dedication environment right so it's below the application even so there is this feeling of a gap between what is modeled in in the application and sorry in the corporate LDAP and what is mod and it is what is possible to model in in opposite and we over you know with the last three four years we have somehow compromised with this gap but if we recognize that there is a gap and we try to fit it with some tool I think we can get better results and that's what we're going to propose here so here you could you could fill this gap in many ways right there's no you know should have your own way this could be just a file in a git repo man it's with the get-ups approach okay but in our example we are going to do this with red artists easel which is a product is a reddit product where that single sign-on it's a little g-well in our portfolio products sometimes not well known and it's free for for people who have a an operative subscription okay so it comes in the bundle of open shipped but for the benefits no key folks only one of several components that come out of the box with your subscription so take advantage of it a lot of customers I know are like how much work to pay for it and I'm like you don't zero you just have to install it exactly so well we're gonna see in the first part of the demo is is that we're going to build a pretend company held up okay deployed inside of openshift we're going to create an instance of red single sign on the upstream is kilo that Andy mentioned and then we are going to import the current existing organization from l-dub to add single sign-on so we're going to see something like this after they import ok and then we're going to model this is the green piece that I was showing here that the missing piece inside of relators is so so we're going to augment the organization model to add a couple of layers ok so this is the first part of the demo so now I'm gonna stop with the presentation and switch to the terminal who doesn't like who doesn't load some demos let's let's make our sacrifices to the appropriate demo deities and everything alright I have no tested this Dame on this current class suite so we have we have good chances of encountering some issues and good chance of excitement you mean exactly so obviously have some scripts here that I can copy and paste and to run some automation the first one we'll just install this Elda it's pre-configured to load a some some data inside the LDAP database and it was an interesting learning for me you can you can create an eldest file with all the information that you know that you want to preload on an LDAP so I'm gonna run that ok and now it should be creating it so let's check that we have some pods running creating this and after this is done we should be able to connect to and up to another viewer line if there is a problem clearly we did not pray hard enough no Chad says maybe to get a repo there we go they run it again it was just a downloading so we should have if I go through the routes Rev scripted also to get there the address but we can just go here and there's a simple UI for LDAP we have to login we is a weird username you wonder why we use me all that but not some other no indication mechanisms I gotta say playing about 90% of the customers like to go to and we even know rocky only I could go to Vishal you know that for everything so it's a good mr. case it it's all old up all around ok so we can see that we have created some capabilities some groups and some users in this example I am for each group for each P be you so it's the second level group we have line of businesses and be used for each be used there is a manager with the name and some developers but I didn't come up with names for the developers there's no this is just because I got lazy with with coming up with names it's not a reflection of myself it is because at some point I I ran out of fantasy and imagination but so the point is we have we have people loaded in that second step is going to create right at an instance of redundancy so to do that we are going to use a the red lettuces so operator Niccolo corporation which i think is going to become officially supported very soon if I read correctly the news and also has been released yesterday so if something is going to break it's probably going to be around here now we say like demos are better because things can break right like if if the expectation is something's gonna break then it's awesome I mean I like I love cooking shows but hey yeah I can pull the turkey out of the oven - all right so let's see a source created correctly actually gone through and look at this Explorer page it is the greatest thing since sliced bread I bought this page you can see all your resources on the cluster search for them and be able to modify obviously operations ok so I see it's initializing so it was just because probably the the operator was still booting and we can see it's now we're postures running and it's really surprising cake look this usually takes a while for key clock to get to become ready once again you say a while I mean how long would it take to go ahead of course a server get a server wait for deck you get stood up and automated I mean Kristen you came from ansible you can do that pretty quickly but still yeah it's a few minutes it's not I mean especially when you're like okay I'm going from nothing to you know a fully distributed you know deployed application that takes a long time right like I mean I've taken you know very very large apps that have taken hours to upgrade right like it really depends on the actual tools you're using yeah you know I I'm just I'm used like it's open ship I yeah I you know what what is it I get most of my clusters via push button these days so I don't want to deal with toil right like if that makes sense right let me aid systems and put them in place here in Red Hat so that we can get what we need very quickly we want everyone to do that for their business right like not necessarily exactly how we did it but we have some to leaning that can help you here so it it might take a few minutes but guess what that's awesome we can what we think we can take a look at what it looks like to stand up Aki cloak so Sookie cloak is not a complex application but it has a couple of components but with the operator the only thing you need to do is well now this we cannot see it very well let's look at it from here maybe Oh too small oh yeah I mean I kind of I mean no I'd like this a little better yeah there we go okay thank you so to stand up key clock this is the only thing that we need to create okay especially we wanted an instance we could have more right for AJ and then the the extension that we want to load there are more features in this operator but basically this is what we need and then here as you can see I also have deployed a real in Creek Road which is is essentially a domain for users okay so it's as often now this reel is going to be empty and we're going to populate it okay yeah go back okay so the next set of commands is about integrating kick low credit assist so and held up as you can see here I'm taking the kick lock password and then I am because I don't want to have to carry all the CLI commands on my laptop I'm actually running those commands from the kick log pod itself so I am connecting to the pod here and first you have to login so I'm just logging in using this there CLI and then the next set of commands here is to process the template here is more template and then again running the kick look at mean CLI to to create what we have in this template so I'm gonna let's open this template and see what we have here this is role okay so this is essentially the configuration to to map rose from LDAP to this domain and we have to provide a couple variables this is actually the idea of the domain if I remember correctly and these are all the parameters I'm just going over a little bit of what we're doing you know if you want to know more in detail we can explore more just let me know from the chat okay so it's not a full dog and pony show only a perfect roll it depends what people are interested in getting if you want to know more put in the chat yes I mean Ecklie that's yeah that was my my point just making sure we have no errors I really wish well I there's there's a chat app I'm using right now that has no font size options super not helpful for me so now let's go to let's go to key cloak the UI so as we we should be able to find around here okay so we want to log in and in this case we have to use well I can't I cannot click on fortunately he cooks a little more secure than my luggage password one two three four I mean my luggage password is slightly more complex but I mean how complex can it actually be it is you know I mean it was a major topic of Spaceballs exactly so we're logging in as administrators of click load and we are directly into the OCP realm but there is always the default reel right so now if we go to seed groups probably they are not important yet because they import function triggers every five minutes but if I go here this is the essentially the connector to l-dub we can test the connection it's successful we can test the authentication it's successful and then somewhere we can synchronize at the bottom yep all the users and that should also 20 users awesome yeah 20 users we can see them here this word we see we have some developers and some users they should also have synchronized the goods so we have a slightly better view now than before on the LDAP UI because we can see the hierarchy right but still very coarse grain right so we want to get the next level this is this is the piece where in theory we are farming out so let me go back to this in theory we are farming out this augmentation of the hierarchy to the owner of each of the leaves right here so take mortgages so we could say to the markets owner Tim Tim administrator we give you permission to create sub groups here and these sub groups represent whatever you want but you know if there is a standard in the company they could be your team and then I could be your application so your micro services and then this owner is also in charge of assigning the people in this in this in this sub branch of the hierarchy in today to the more you know to the individual application or to the individual development teams so this way we farm out we farm out the administration of the finer layer of the hierarchy and we don't have to have a central team that does that because like I said we want my team development team developers opening tickets right so we want to fund this piece out now there's someone that has to organize all this I feel so sorry for them cuz yeah well yeah eventually yeah I mean there is a process right but the point is we want to move these responsibilities to the people that needs to need to do it and not ask them to open a ticket right so they can they can do it by themselves in this demo we don't show the space we we have a script that automates the augmentation but I have talked to the key club developers some of the advanced features of cake lock really allow you to do this so you could you could configure kick lock so that I am for example the owner of mobile banking and when I log in I can only see maybe I see the entire key but I can only manage this piece of the iocai based on the permission that were given to me okay so this is not in the demo but just know that key code could do that if you want to if you want to be able to do that all right so the next piece and sometimes I cannot click on this the next click is Israeli to do the augmentation so we have a script here that should take care of that and it's going to take five minutes this will create subgroups and also create some metadata for these groups and will assign people to do these groups so we let the script run and then we go back to to the the key clock UI to see what happened so in the five minutes secrets we have any any comments in the chat yes so yeah I mean rockhounds asking is this for like a jar or we don't see companies doing my having their HR doing idea onboarding type stuff so this this whole demo is essentially for well well sometimes yes because they have to book it them into the proper group oh yeah they have to be someone in HR that creates an account or something maybe and then put them in the right place but then that's kind of where like there's all this other toil that's involved in getting that person into all the appropriate places development wise or production wise right like getting them in the right environments putting them in the right groups across the organization right across your kubernetes cluster is your whole fleets like that's where this comes in correct so that the idea of this demo this whole workshop today is HR will directly or indirectly manage l-dub right that or better l-dub is the the configuration that you see in and up is the result of what HR has done we need to go deeper than that we need to go great like a charge anything into their you know resource management system human resource management system that then creates this account right say somebody's I so we need to continue that thread right right and they'll fill out a form and some system somewhere saying yes there won't insurance and all this other stuff and this is the right address and everything else what seem they're on let's say this person is there and hired by the landing team is gonna be in this in this group right it's gonna have an account here there'll be someone in that team that has an account or it will this be HR than dumping them into this account indirectly yeah because of what HR does they will these people will get an accounting out of and will be in the landing group team okay so that's what we can expect from an HR but then the landing group team the n group not him the learning group a line of business or business unit let's say has many applications in which application is this new person going to work for the next three months HR does not care has no I have no idea there's no address and you know they plugged in their personal details but we would like to go a little bit different with these two of these levels of granularity because that's how we want to be able to model our permissions inside of a machine right so we want to do that without having to touch the corporate Elda because that would that would impact that would be knowledge or that's the information that is specific for operations just to be consumed by open ship but if we put it in other it will be visible by all the application everybody clutter I mean if from the application started to do that all that would be just a mess right so we want to we need to add these additional layers in in a way that can be managed by the right thief or not not HR but the people that are actually responsible for landing so here there is a manager for the landing meal it will know where you need to work for the next three minutes or team leaders yes somebody they believe accusation yeah and we want to do that without impacting all the applications around open ship right just be consumed by this information is just consumed by or maybe open ship and a few other but not necessarily everybody right so let's see what happens after the augmentation phase that's how I call it so in this little demo okay in this model we have augmented the retail banking business unit and we have said that there are three teams below this business unit there is the online service team which is the one that creates the mobile app and banking and make your font size bigger since we're TV going deeper and deeper down as pool when I open a new window I don't get the right ass okay approach it ask for that sooner sorry yeah you know that's please keep reminding me so yes so we have for example the online service team they create the banking the banking online banking application okay and they have a few micro services below them that they've managed and then we have the large team this is the one that sends you the SMS or the notifications and and then there is the acquisition team which is - is a different application that it's just the one that you use when you open a new account okay as you can tell I have been working in this kind of organizations so so now we have extended the original held up with more fine-grain organization and I'm not sure if there's a way to see this but this group may also have they also have some attributes for example we say that the online bank online banking bill pay service is a large application is an application okay and this is a large application okay so we added some a few a few information a few pieces of information and I think if we go to this develop group we have we maybe some members I didn't I didn't I'll animated it up to this one okay so maybe I have numbers somewhere okay so the developers are added directly to the to the services they they're working on okay so in theory the job that the online banking administrator did was to say I'm running these three teams inside these three teams I have this actual application that needs to be you know develop and delivered so they are on different they are different deliverables and they follow different delivery cycle and I have developers assigned to each of these application okay so he's a manager and is assigning is developers to each of these applications so we this is the job that we want to farm out to the individual team leader and team manager in this case we just have an automation to do it okay so this concludes the first part of the demo let me make sure that but I think that's alright yeah well we can also do the LCP right out SSO integration so these this piece makes makes it so that when we log into OpenShift we have an option to log into reddit assist so as the IDP OS provider so i'm gonna run this once again if you're smart and remove that you bad man you'd have another option if you haven't you're just gonna get to keep that okay we can give it a few minutes to work but essentially now if I go to let's open a new window and I log out from here respecting another provider here yeah sometimes you gotta go ahead and restart those a walk pods yeah I don't know if the pause would actually pick it up would it sure did sure it's really in theory and let's just being slow well let's troubleshoot a little bit so I'll do you troubleshoot these kind of issues well you go to the the cluster setting and the configuration I say rock down yes to answer your question this does provide an additional authentication method on the console so it'll be another provider yeah so what we want to make sure is so I have my password you know password file authentication so I can say remember in the admin password and then we have the new one that we added through this script which is the one that connects to take a cloak which is an open idea type right o IDC type of provider now why is it not working I mean let me try again because it seems yes I'd hit refresh yeah they look in one refresh refresh sold it so at this point we have integrated a new home a new authentication provider and but the users that are coming from them that authentication providers have no permissions right so if I log in right now I think I can log in with a developer I think they all have password well we shouldn't Ciara we should see empty right we should see that there is nothing here for us to do please please log out and try again maybe you have to hit refresh sure yeah yeah that's got a cookie or something somewhere oh it's yeah okay okay this time deb-deb there we go I'm a bit worked yeah working so we can see that now we can log-in through through key cloak right which is essentially wrapping our Elda but we use the password that was to login and but we don't have any permission so we cannot see anything okay so let's go back to the presentation and explain the next steps okay so the second piece of the puzzle here is how to sync with outer sync groups that are coming from external identity providers with ways of a shift and the bishop does you know handles this problem slightly different differently than kubernetes we're in kubernetes if you are if you authenticate with an external provider you actually kubernetes will trust the claim that are passed by the external provider and groups membership are as some of the claims that are being passed in overshift there is the natural internal authenticate else Authenticator and and and the initial authentication that you do with an external provider is just to prove your identity but not to prove your claim and so you have to and so innovation we have to store groups and users inside inside of a human and they are storing etcd and for users it's easy because users are created at login time for the first time that you login the user is also created but groups have to be synchronized or groups have to be created but it makes sense to synchronize them from some source of truth or you know system or record historically in OpenShift we have only supported synchronizing with l-dub okay and so but but now customers are starting to use other identity provider not just another and yeah we were joking before yeah it's mostly and that's true but we are seeing more and more oh i DC provider is now become I mean when open sure first came out we actually had no groups think it was from feedback from a lot of organizations who use well that that that original feature of synchronizing LDAP groups into open ship was actually created to take your you take your feedback so we felt a little bit of a gap and handy here decided to develop the group sync operator so let him talk about it for a while and go ahead and yeah so really a lot of my customers is rondalee mentioned or moving away from all that you know all that is still very prevalent but they are seeing other alternatives whether it be you know mainly open ID connect but other providers like github get lab and a lot of the other supported components that openshift natively supports but they always run into an issue of okay I can authenticate but as we saw earlier I have no access I have to go ahead and you know here we read this together and somehow I try to synchronize it's a pain let's go ahead and use the same model that opens your for uses and use an operator so we can go ahead and provide a pluggable and implementation that allows you to integrate a number of open ships native authentication and identity provider so after they want to just click on the link there and you can just go to the github page and we got some nice documentation and I'm so getting used to ya new UI of github yeah do I changing like in the past 24 hours like kind of like jarring but yeah I had to figure out where some new buttons where this morning it's like oh I woke up early to work on Oh learning the new github UI I thought I was just lucky and they happen to get me for a canary tests I guess I'm special I got the new UI but apparently you could opt into it earlier so I did not get the message or did not see that when it first came out but I guess they'd been telegraphing this for a while now I missed the memo now say man oh wow alright anyways really this group sync operator allows you to integrate a number of different providers like as you saw in our cluster currently we are using technically 2 and a 2 and then an asterisk identity providers we have the HD password what allows you to define a flat structure for your your users then you have key cloak right ahead SSO which we helped set up then you also have the aspect which is Technic you bad in but the idea is that you can integrate a number of those identity providers into into or whether you can synchronize the users into groups that are already defined out in those providers so as you see here we support a number of them right now as your github get github and get lab or a huge I know a lot of people are organizing their teams and github and we took a lot of the same concepts that a lot of other components in the community you use so you can synchronize certain like organizations in github but also filter on team so let's say you have a bigger organization but you only want a subset a bit of those members to get access to your cluster so let's say I have a dev team I only want to give access to my dev team actually better one is it my QE team I have a QE team I don't want anyone else getting the access to there except for the QA guys and gals I can go ahead and say ok allow those from the my org Huey and they only get access to it and everyone else to skip that wonderful sorry don't get access so we are actually leveraging the key clover hat SSO provider and it's really really easy if you just scroll down to the configuration section you can basically just set up a secret that contains out of the heck do I authenticate to get to key cloak and then from that you can go ahead and say what groups we should go ahead and allow what realm we want to synchronize so secret so key cloak has a number or whether you can organize your organization even higher into realms so let's say you had a North American organization and maybe a European you can put them into different realms or isolated rich sorry can you increase the font size again yes so you can go ahead and set that up you can I I always like to bring in scope scope is a way that you can this is an old LDAP and this kinda goes into the hierarchy only do the top-level and don't do any of the nested subgroups we support that as well as this is one that always bites me on a lot of open source software is that oh they assume that we all have science certificates from public CAS and they in organizations this is not the case if you're yellow for now please please keep that in mind have a way that you can provide a certificate so that for those who have their own CA or aren't using the public CA you can at least be able to feed that so we support that as well for all its enterprise use cases so so from that all I need to do is but if you happen to not at least having an ignore function don't do it but at least have it out there you can test it out so for this example this is as simple as it can be it's a color custom resource called the group sync allows you to specify one or more providers and forkie cloak you specify the realm what secret you want to use to authenticate against key cloak and where the heck is key poke located in the cluster that happened to be on a separate server you can support any one of those options I'm gonna turn it back over to my family and kind of walk through more of the demo ok thank you yes so in particular for this for this demo we are going to use the feature of the operator that allows you to sync metadata and hierarchical information so so those little pieces of metadata that we saw before will be annotations in the group that we import from Inglot import to open shape and we also get some information about who is the parents of parent or parents of the of the group and who are the children of the groups so we can use we you know in theory by importing metadata into open shifts around the groups you could you know think about developing operators that navigate the hierarchy and do something about that today like I was saying today that up until today it was impossible because it's in opposite everything is very flat right there is no that opens options for us so in this part of the demo we're going to build this second piece where we deployed a group sync operator here is this is a logical diagram right but the operator is obviously running inside of a ship and its job is to sync groups and then well again we as we saw before we can already login to to local ships okay so let me go back to the console so we first of all we need to deploy the operator simply do this I want to show you how to deploy an operator from the CLI because everybody knows how to do it from from operator hub which is very convenient but if you are trying to automate everything maybe with a get ops approach you need to be able to do it from you know from API yeah and so this is how you do it you know first for this kind of operators you can simply create an operator group with with the target namespace where you want to deploy the operator all right so where you want your predator to watch but really usually it's the same namespace for your deploying also the operator and then you create a subscription with the metadata regarding the specific operator that you want to deploy I usually to get this information I usually deploy an operator once through OLM through the operator Rob look at the subscription that is created and then script it for my you know for my needs but this is only the only thing you need to do to to deploy an operator using the API and then this is our configuration for for the group sync operator so it's very similar to the example that we saw before there is a secret there is a URL for four key clock and and there is a synch period so should we you know change anything on on kek lok every five minutes this is going to resync all of the other groups okay so at this point I think the pairs are triggers immediately on a we have we have two it should be and should be should be already yeah it should happen but you should be running no I think you can blame me now I'm sure we have to know we have to deploy the the group sync oh oh it's actually tell it what we yeah we have to tell it to actually sync something right yeah okay that makes sense the thing is there that can sing something yeah so it's the operator is pretty quick so by it's probably already synced so we can go and do you know see get groups from here so here we are administrator so we're going to see everything that exists come know thing yeah yeah yes let's go make sure everything is fine we have established that is the fun part so he's got some fans out there I'm not a little fellow external Tony and from Facebook live for folks you don't know Alyssa is the one that wrangles all the fun some workshops for us this year so she and I have been working together a lot and it's been like really enjoyable to like to see how the sausage is made for workshops especially on the large a lot of folks I don't know how she does it I really don't know how she does it okay we have some think know if you want to decrease your font size you can actually see the whole air or whatever secret not found secret not found signal phone yep found okay alright so we need to have a secret we should have a secret at the bottom forgive us okay so let me see let me go back to the demo what should I maybe skip this yeah did you not create a secret somewhere secret secret let me see did you fail did that line feel not so it created click on in a different name space on kik log operator namespace and don't double check that no it should be fine okay so we need to check that is in the cloak operator namespace yep exactly there's called keiko group sync up there and looks good alright alright so should I go back to the pond yeah worst case windows get a 5 try again yeah let me decrease the font yeah you get a read it read it but just tell her what the problem tell us what the problem is please yeah everybody several rejected event Oh No so that's a go air oh my gosh go on over see what that it let's say you just couldn't create the event but in theory it should still be able to see the secret all right ok so let's try to understand what the event was the first because we did there is some information that debugging it's half the battle so if you have an engineer to debug go errors this is fantastic yes this is quite ear - holy smokes can I create resource events an API group and namespace default defaults where did you deploy the operator did you deploy the operator on accident to default yeah yeah groupthink waves the same default yeah didn't you actually specify on the command line the namespace I could've sworn you did yeah sorry I think this is something change that has changed in the - I guess in theory they shouldn't even have you event it's bombing because it couldn't just with the secret so in the end yeah where do you see that it's we have a problem with the secret school and I don't see that school but the lesson over no be right not this time before we sell out I swear we saw it mm-hmm let's do this let's restart the policy yeah let's free a much less a clean slate this thing come on okay good good [Music] wait so it sounded it did something yeah so bombing no it's still bombing that okay so wait a minute it found the thing that we have to reconcile but see it says error secret error secret key cloak yeah second line there yeah all the way to the right yeah it's a seeker found secret she heard he cloak goo causes air dirty cloak group sink not found where where was it looking for it's the question hundred thousand dollar question there yeah worst case worst case throw it into the cube sink project just in case yeah also yeah what version of the operator you look redeploying yeah I was thinking that we're in the same thing what is coming down from over at the rubble what is the best way to look at that go ahead no well number one okay look at the pod pod for the operator and people didn't see what pop which persons wing look at the ammo yeah okay all that mitigated it adds just go make something but couldn't you just serve your zero to is it right that should be the right one okay Hertz case worst case I don't advise doing this in a production cluster go ahead and just give that dangerous account no elevated rights yeah yeah we want to see you actually work and then we can go back in this is the best part open source at an issue we'll get it fixed yeah no no no no no no no I don't know egg poster role to user no once again the point again development clusters don't recommend doing this so yeah don't try this at home kids this is why you testing a lower level environment find all the bugs or you watch the live stream when we find the bugs so this account is this yeah like I remember I saw this issue before so this isn't the first time I've seen it I thought I fixed it leymah creator oh really yeah yeah I see nothing okay cool let's see if it is already able to fix itself or we have to tell you that back off I know just go ahead you can just kill it yeah see it's not it's not scrolling arrows anymore so I think it worked but I can I can restart I'd like you restart it that gives a nice ability to see exactly what Roenick will see the progression clear oh right yeah I know that aired out - oh man so this is once again it's cuz the secrets I found we we now got past the API issue of not feeling creepy event but still early in it's still not finding the right request namespace request name okay so where is it looking for the secret error secret group so key cloak group sync is that in we found that right that's in this group this project right so you put namespaces I've seen this error it's like go look at my way is key cloak groups Inc supposed to be hyphenated yeah okay so that's fine and or I go look at my local repository and see a bunch of uncommitted changes right are you serious I don't know I'm checking right now yeah that'd be funny [Laughter] no I know it was fine was working and I know you I know you've had it working on ro to right go to secrets right Kiko groups Inc is the name of the secret correct okay so let's go look at secrets real quick I'm pretty sure we did but why not just prove key code groups Inc so does it need to be in the other project the problem I'm running to if you go back to the to the air its blanket in namespace it's like yeah yeah like that was the other thing there it's no the reading it's not finding it because it's not looking in the right place right I have an idea I still have I think I still have the code of when I was playing with this yeah so I'm gonna cheat here and run operator from my laptop works how many of you are in a demo who are good at giving demos so I mean you know with just pull things out of a hat I know I do okay I think it's interesting it is kind of yeah so so what we're doing now is if for those of you who have never done go development we were actually gonna want an operator locally on his laptop connecting to that cluster and to be able to see the same work but most importantly here if you are doing this make sure or in general don't run to instances of an operator yeah wait yeah okay so I'm just looking for the right way to start the operator which I think it's at the end should be yeah it's a development okay once again got a spin down the other operator and this is so I scaled to zero so okay so I'm running the operator from here well I've never seen this done before yeah I know you you had the ansible operators before this is go operators yeah well I mean I usually test ansible operators and this molecule it worked oh there it works well now wait a minute now you see it was able to find the secret and out indicate to kick log and it found 25 groups think so now if you go here you know I wasn't having nice but now I have to looking debug this thanks a lot wrong yeah yeah I was about to say Andy he goes you got another issue a tear to my list yeah stack okay so we can see we see all the high-level group here we have all the groups that were in inky cloak we see them in a flat and flat view but if we go inside one single group for example we can go to this one there will be some metadata okay for example we can see that the children are this one and the point is retail banking okay beautiful and then we can say that we have five developers now let me find another one I want to show you they made their actual metadata so I'm going to take this one here we can see that this is a type application this group represents the National application and people working on this application and is a size mods so in the next step of the this demo we are going to see how will we use this metadata to to configure the names places correctly because these are hints for for the namespace configuration operator okay okay so we're good here yes we are we can go back to our presentation this is so so we solve this same problem now sorry the second problem which is now we have a consistent way about indicating and managing managing group membership with add additional nice feature of having the right method some metadata coming from the IDP now we need to actually configure the namespaces okay so now we would like these new spaces to be provision automatically and also configure what do we mean by configure usually these are the things that you will typically this is what we find that customers want to configure permissions obviously quota it's self-explanatory right and then we have network policies could be that they want a static and enforce configuration of network policies or it could be that you just want to give a saying you know default to start with but then the teams are in charge of selfie you know configuring their own mentor you know fiber rules and then egress network policies on this one you know by design already know P shape this can only be managed by administrators so these are the ones that are probably going to be enforced if they exist and then you know could be anything but I'm just here I'm just saying these are the most frequent okay right so for this step we use the namespace configuration operator this is an operator that will react to the creation of users groups or namespaces so it watches for those objects and you can configure it to create arbitrary objects based on those so let's take a look at the project you know just to explain a little bit so let's let's say we want to react to groups because that's what we have just created okay through the group's configurator so you could configure something like this okay this is the user so it should be well okay so let's the user so let's say we want to create a nice place for each users user that logs in because we're doing a sandbox environment let's say and and everyone should have their own experimentation namespace so what we could do is to say when you see a user logging in from in this case we have an octa provider not even know this is so but let's say it's an octa provider logging in and it has the label stand box enable true create a namespace okay and then we probably need some something more here we should also give the user some permissions on that namespace but as you can see we can we can just create a list of objects here that will be that will be created by these by this operator and notice that it's a template so if I have 100 users this object with this tablet will be run one under time I'll only changing obviously the name in addition you know we already looked at go development and how to go ahead and build an operative and go this is using go templates so it allows you to merge in the configuration of the namespace with this template so it's basically saying okay tape the namespace at a company or get the namespace as it comes in or you carving the user and go ahead and cuz I say it's a user config and take the name and create a new name namespace called whatever that user name is - sandbox everyone gets their own sandbox personally made for them that's awesome right thank you and I want to spend a minute talking about the the requirements that we gave ourselves in this demo ok so these are the these are the policies that we get our and the namespaces and the users that connect to open striven in space - that we create have to comply with so these basically are the rules of engagement right for our opposite of every cluster so each application needs to be provisioned with 4stsc environments with the following naming convention so there will be a name and then - build dev QA improv everyone in there in the dev team should have you access to all the environments of all the application in the dev team I think that's in the same dev team so what I mean by that maybe I could have expressed it better but what I mean is if I am a developer in any of these in any of these groups in this dev team I should be able to see everything in this dirty you definitely have read only access to everything maybe to see the other ones right right and then everyone on specifically sign every was sent to an app should have added ok every sorry every person that is specific assigned to an app should have added access to dance DLC environments of that specific app so if I have if I am a developer in this namespace in this saree group I should have added to the steals environments that are created out of this group but only view to the other basically that's those are the two first rules and then the build environment is only is the only one but builds can run ok we don't want build or running anywhere else the builds need to talk to the corporate Nexus in this example we're using Nexus for our you know assets and libraries and the corporate gate lab where the horses are okay so we give true we have two DNS here names and those are the only thing that those should be allowed from the Builder namespaces any other communication must be stopped so you cannot connect to anything else from the building spaces the dev and QA environment can establish connection with the internal corporate network this is the cedar of the network and the corporate nexus for image pooling the prod environment can only talk to the protein at work ok we pretend there is another network and the PCI network and there is another network the corporate nexus for image pulling everything else should be stopped okay so we are trying to be very very strict in terms of outbound connections from from these namespaces its brothers we live a difficult Network policy by which pods are allowed to communicate only within the namespace and receive connection from the router pods basically this is about this is to emulate the multi-tenant behavior that we had in opposite three it's basically so so we're saying as a default you get the multi-tenant behavior but then the owner of the namespaces are allowed to modify those policies I mean this is one of the one of the areas that I know a lot of teams still are struggling to understand it's just simple network segmentation they say okay I can see this namespace but why can't I talk across it because at least no bunch of three it was isolated her name space open two four there's a new STM plugin or policies that basically allows you to be even more fun friendly granular into the different policies that you can apply for the communication which is great because you may only want your web app talking to your database and nothing else and then continuing the build projects will receive very limited quotas so there is a fixed size quota for all the build projects and it's very little the dev and QA projects will share a multi project quotas so you know it's you can create a multi project quotas across a set of projects that the developer can allocate based on their needs and but we you know as administrator we don't care if you use a lot of your share of quota in QA and nothing in dev or by severe so do we let you we let a basic it's a single pool for dev enjoy the multi-project coder can be chosen at project creation among three t-shirt sizes and that's the hint that we have on the on the group and and then the prod project will receive its own quota so we don't want to share quota between pre prodding and proud right but still a one to have some Kota and again the quarter can be chosen at project creation and then the dev QA and prod project will be assigned egress IPs and we assume that the user API home operator is installed this is another operator that will assign egress IPS to namespaces why do we want to do that because if you have an ingress IP assigned to needs to your namespace you can identify outbound traffic based on the IP basically that IP becomes the identity of your namespace and can be used in and can be used for outbound connection to configure to configure firewall rules for your outbound connection so let's say only only application a should be able to connect to an Oracle database now application a is identifiable with an IP okay so go back to our presentation hey have you seen anybody using this with ACM yet not yet um I know ACM for those who you don't know what a CMS sorry advanced / management I mean I'm following evidence - yeah everyone has this is read ahead advanced cluster management for kubernetes allows you to connect multiple open ship clusters but even beyond multiple open clusters you can go ahead and connect any type of kubernetes clusters whether they bring in the cloud stand-alone etc I haven't they are very much into get ops in the get ospa model so we have a great team that it's working on that right now and they're working also with the reddit developer game be able to really showcase how do you get out properly now Chris and I done a few meetings earlier in the week ya know so yeah if you're if you're wondering about ACM right now next Tuesday 11 a.m. to 2 p.m. Eastern Time 1500 UTC we'll be talking with the Technical Marketing Manager of ACM himself Jimmy Alvarez so super excited to finally have him on the channel and get him teaching us all the things they see him he said he needed three hours and I was like alright let's go let's do it and we might we might only cover one of the buckets of ACM even so it could be pretty intense it could turn into its own like series on governance and policy that's a big one yeah so we'll see how far we can get on Tuesday and then we'll go from there you know but everybody's trying to get their feet wet with ACM right now so we are definitely we definitely hear you and we'll have content in the pipeline for sure is they seem available to public yes GA no no tech review sorry off I always live in the future here yeah I feel like I yeah yes running together sorry my counter saying yeah let me ping I can find out yeah but I mean yeah like it reach out to your rep if you have one you know or not if you have one if you're ready at customer you know you can reach out to your person that you usually talk to you about redhead things and ask them about it and they'll be able to hook you up potentially okay so in the third part of the demo we are going to install the namespace configuration operator set up some configurations and then it will react to them and create the necessary necessary resources and then we we will try to conclude the demo by logging in as one of those users and see that we only see the right thing based on the you know on the configuration that we have on requirements that we have explained and that we can immediately start working if you wanted to okay so let's go back to the script I'm back in a bunch out here I'm gonna lie operator the date same okay sorry no sorry I'll go ahead I was just reading it out loud my bed yeah so it's the same procedure we create an operator group and a subscription and the operator should be already being deployed and now we have a bunch of configurations I'm gonna show some of them I don't think it makes sense to show all of them I think it will pick up very quickly how this work works let's say let's let's take a look at this one if I remember correctly this one is the one to create creates the namespaces based on the group's okay I'm gonna go on the code just remember okay so I need to close this okay so we are going to watch four groups okay and we are going to match all the groups that the type application has their annotation if you remember some of the groups had had that annotation in particular the groups that were at this level the services had that annotation okay so here we are selecting all only those groups and then what do we do with those groups remember we we essentially run a template merge with these templates and the parameter of the of the template is the group itself okay and so what do we do we create a namespace and this namespace is what we could the build the dev and the QA namespaces okay so each the 4stsc names places but then we also add some annotations and for example we we had just some some record to say this is the name name her name is actually the name of the group right so and and then we had the team which we can figure out by using the ir key parent annotation right so we can go up one layer and say okay this application is really managed by this team right so by doing by using that we're going basically if this is the group that is being sync where we are going up one level and looking at this name and then what do we do then we add the type okay this kind of metadata is then going to be used again by the same operator when when we actually go and configure the namespace so maybe in the next example we will see how we use this metadata let's take a look at at the and another one for example let's take a look at prod so proud we named it prod obviously team and up we have already explained we say that this is of type run remember that we hear the rule that said bills are allowed to run only on namespaces on the building spaces so we we're going to latch on this label to apply that configuration that says that prevents build from running into a namespaces that are not of type build so we need to call this namespaces something else so we call them run and then we specified at this stage prod we specify the size and the size was chosen by the developers right so we look at the notation of the group where where the team administrator or the developer has chosen the size for for this particular namespace a based on this there will be a rule that the creates the right quarters right and then we have this annotation here which is the one that assigns the egress IP let's go and take a look at the quota configuration so we have two configurations for quota for project quota we have the build configuration and then we have the prod configuration remember that the driving QA get they don't get an individual quota they get a multi project code so how does it work well again we this time we are watching for namespaces and we select all the namespaces that have the type build okay so that's the notation that the label type of build so that that's the label that we assign before when we created the namespace and then we say we want to build a resource quota object and these are the these are the dimensions or the sizes okay so relatively small doesn't really matter this is an example and then for the prod we are going to create different quota rules one that selects project of type run in stage prod and of size small right and we'll assign a small quote and then we have the medium and then we have the large because this is three three we have only three t-shirt sizes okay we can make another example with for example the network policy okay so here we have four we collecting everything here actually so basically anything that has it is in type type run or build so basically everything that we have created will get this default Network policy which is the multi-tenant essentially as we can see you can only talk to yourself or to in the same namespace or receive connection from the namespaces that are annotated this way which which is what they're the ingress namespaces where when ingresses are deployed let's take another one let's take the multi project quarter okay so this will be applied to feeling something is missing okay this is will be applied to the side project that are of type application and and are small and well and it will create a small quote and then those are medium we create a medium quota and and those are large will create a large quota but I feel I feel like I mean because this should select only the one thank you what the demos today right yeah but now but I mean is it's an opportunity to to talk about of this work I could you know I need to add another annotation selector here that that's a better selects exactly the projects that I want but we had enough metadata to do that right I think we showcased almost all of them we could look at the egress biggest rules what do you know you know this is not not a common object because it's over ship specific but it allows you to control our outbound traffic from an in space so if you remember the rule from we said that from build you can only you could only talk to the Nexus and to the guide lab so that's how you can do it nice okay and then we said from projects that are in dev and QA so this is the annotation I feel was missing before it's the selector - to narrow down on their only dev and QA excluding prod so we said we we were going to connect to two Nexus but also to the internal network I didn't I'm a huge fan of those matching rules that they have in kubernetes I mean that allows you to be really really flexible about how you want policies and be able to as we've shown tagger different resources effectively build grab them and this one is for prod yeah I like I like them too and very easy to use from the code perspective also this is from prefer prod we said the rule was slightly different and it was to allow still Nexus but they only be proud the prod network right frame okay what so and that that'll really like if something gets compromised by chance right like this is a higher level policy that kind of you know stops that process from you know breaking past you know the barriers of the namespace essentially right so yeah and this one in particular is about you know you may want to say I want to keep fraud and non prod separated from from right like you could actually say like to allow no cross traffic between right and Matt's yeah and still but still they run on the same cluster this is not typically what we do but you could do that right in theory and you have all the tools to to configure that okay so we can run the script all right so this is a little bit of work for this for this operator all at once but we should see already that if we go back as administrator in Streeters that we go to their to the project list we start seeing already this project okay so we have online acquisition credit score service build the product q8 right so those those were the projects that we were expecting and you know all of them are already created that was pretty quick but so basically all of the process were created we can now try to connect and let's go back let's say we want to be investment so far man security feature number then I logged out that's ok ok so what I want to do it was is just to see what who is the developer in this so depth 3 is the developer in online banking investment account service so let now let's go and login as Deb 3 I'm still confused but by what happened before so yeah I log out here what else will I be the administrator it's all set then 3 you have a nerd magic 8-ball and we'll see what it says when it's wrong so let's see what we see we see all of the projects which is fine that's what we said in the rule but what can we do so be wearing online banking investment if I go to that one sorry online banking investment and I say I go today's I am a developer so I'm gonna go to the developer view and let's say I want to deploy hit database no I don't remember this you have options I need options in my catalog then we're gonna show how much they don't do these things know what where can I find like a little build of maybe a go back go back to go back to the ad we're at the head yeah we're going to the catalog gonna count there you go okay yeah and okay I would I happen to wipe the dotnet example this one yeah builder image okay cool so let's create this one so we are in a build project so the bill should work yep go ahead click on that sample by sample you know i below under you know it's very simple okay yep a lot of matically if we populate that for you set everything up and innate it out-of-the-box easy customizer click create and one of the benefits of open chip is through a process called towards the image go ahead and build that in the deploy it alright so if you're lucky this should be doing a build yeah so as you can see the build started now while this is doing it let's go to a different project let's go to build payment okay let's do ad and let's do from catalog same thing right and we expect this to not work right because we don't we should not have permission to do it this is trying to do some thinking hard yes thinking hard well let's Don house'll points out that like network policy is opaque ish sometimes the folks right like they can't always see that and I'm trying to convey the argument that like if the egress network policy is in place right like there's usually a business reason the compliance reason right like it's not it's not like something that you can just like oh yeah let me walk around that right like there's a lot of consideration that had gone in the place before the and what I like to emphasize is documentation yeah explain why because developer disco they don't want the platform to be a black box they want to be able to understand why because they'll go oh I guess I can't go ahead and talk to this server because it can talk to the production network and you know cause millions of dollars of financial know right so yeah the the key thing is you know if you have a policy in place already is to like make sure that is documented and the Y is documented always document the why and if you can label like the Y in the policy as well or any somehow or annotate it somehow so that you can say right like sarbanes-oxley or whatever right like whatever the reason is put it in there so it's uh yeah no I get that too from developers my experience it means come in a few things like it can mean I don't know exactly what the policies are in my recommendation and that's to fix or it could mean I don't it's difficult to read what network policies are never policies object and that's this for us to help people understand how to define them and how to use them and enable your developers folks enable your developers yes a lot of this is very new to them and try to make it as easy as possible that's what this that's what our user interface here is trying to do is to make it as easy as possible but then is a first step to getting your manifest because everything in Cooper Nettie's be expressed and manifest if you go ahead and walk through the console the first time and then if we talked about oh goodness it's an hour ago get ops being able to take those manifests and put it into a get outs models and the third interpretation could be and this is usually comes from security is once a network policy rule is define we don't really have a way to see what's happening right is it denying stuff or like so they would like to see a sort of an audit of the deny of the net that the network policy did and well that's not that's not there so I'm not sure why it could be that it's failing because we don't have enough permission even to create the newsreader three cannot create resource yeah this piece Wow we can't even go back you have tried to do something illegal be assured that someone will be at your desk or late but as you can see this this bill finished completely finished correctly and and and so so I'm here I'm impersonating developer three right and as a developer tree I connected for the first time and everything was ready for me right that that was the final point the final jetty that they tried to oh so Lunik sand on Twitter said it'd be neat if there were there might be a central internal log for complex actions like firing off a deployment there is kubernetes Event log is if you want to be able to audit those actions you can go ahead and integrate open open open Eddie's audit API to be able to track down who did what great yeah the who did what is the big part right like the the event log is just a like deluge of just everything happen in your cluster all at once yes you can you know go through that but adding adding a component on top kind of connect the dots for you really really ills okay so that concludes our demo also the floor is open for questions honey Python this concludes the movie is that nice what is next so some very it's a very cool demo you know thank you very much for that the audience agrees a lot of chat action activity today well yeah do we have a rough or do you have all the links available to all the stuff we touched today but I could just like spam chat with its if you can share this presentation it's here otherwise yes let's see just drop them I mean slack wherever you want to put them that you can communicate with me it's fine it'll pop up eventually somewhere this is the demo itself I'm gonna give you the two operators that we saw a lot of these operators are part of the redhead community of practice github organization octave content there sweet I will and the chat yeah RS Ari's right like I'm friends with some of them they love love love because if if you as a customer are having problems and we don't know about it then we can't help you the tree falls in the forest it and no one's around to hear it didn't really fall not until someone comes around to find it yeah like if you're if if you have questions or anything we're working folks reach all if they want additional interrogations the first one if you're starting to play around with some of these operators and components and things break or you love them obviously adding issues to the github repositories is one of them I know I'm on social media so feel free to reach out to me yeah at Chris short on Twitter for me and most of the other places Chris - short on github if you want to ping me in an issue or something you know I had been known to respond complicated Twitter and handle saver SABR e10 for one across all the channels well you made it unique across all the channels mine very slightly yeah I am on LinkedIn she want to contact me or through these repos make sure have a specific question around those topics awesome if you I will continue the chat I will continue dropping links in chat after the stream is over I believe I can do that because the restream chat tool has all of a sudden decided you're just not oh stuff even though I'm heading in there I think it's an electron app so I'm I just need to hit refresh or close it or something yeah refresh there we go so Thank You Raphael Andrew thank you both for joining me and the rest of us today you know collectively we all the hard work you're doing I will continue to share links and so forth in chat after everybody logs off but thank you have a wonderful rest of your day and look forward to seeing you again to morrow at see the first one tomorrow kicks off at 1400 UTC with everything about infra nodes myself Christian Hernandez and maybe Jeff our trial you'll be on to talk about how to build in for knows why to build in for nodes so for so long Eero Christian Anna and guess again yes number three or four at this point oh he's up there trust me yeah you might I think we might have more than Eric right now and then after that is the OpenShift Commons briefing what's new an operator framework so I will obviously be very happy to see that and that will be it for tomorrow but subscribe to our calendar go to red HT / Stream Cal and you'll be aware of all the upcoming events right there in your own calendar also hit follow on Twitch YouTube or Facebook and you'll be alerted as you see fit with your notifications so thank you all again for joining really thank you thank you you [Music]

Channel: OpenShift

Views: 1,106

Rating: 5 out of 5

Keywords:

Id: M1BU1ztNWRs

Length: 120min 46sec (7246 seconds)

Published: Wed Jun 24 2020