Concourse and our journey to production with Spring Boot Microservices in the land of Kubernetes

so good evening and thanks for coming and in particular thanks for having us here at the Congress Meetup I think this is for both of us the first international meetup experience normally we have only spoken at meetups in in Germany and it's also the first time for for Concourse so tonight we're gonna talk a little bit about our experience of using conkers in one of our customer projects where we develop a micro service based application on on kubernetes the the background of this is I met and I know some of the big four lovely folks from engineer better and we met at at the airport flying back from CF summit in Philadelphia and we had this idea that we could talk about that so this is where it all started a few a few words about myself I'm Matthias the one to the left and this is Adrian still not accepting - you have a Twitter handle so this is why it's only one listed I think Paul has already asked so so who in here things Adrian should get a Twitter handle yeah well see I told you you need that and we both work as consultants like cloud consultants at a company called Nova tech based in in southern Germany and I well do you what yeah thanks so this is I mean we're not going to talk much about Nova tech but I we have those intro slides that I normally should do otherwise I'm gonna get problems with marketing and then probably won't sent me all over the globe again to doing this so I'm not gonna talk much about what we do internally or in Nova tech I mean we both in the area of IT architecture and cloud but Nova tech in general there's a lot about software engineering in general HR med methods we do a lot of coaching and training and these are all our locations so there's no UK location yet this might change maybe one day so they're all in Germany except one in in Granada more important that I want to show we're very much engaged in community activities so we are run the local dacha meetup the commune de meetup is a scrum roundtable and I'm also the organizer of the Cloud Foundry meet up in the area of Stuttgart so we do quite a lot there and we're always happy to speak and participate in those so if you like this or you want to hear anything else from us just like let us know now we dive right into our project so first of all I want to give you a bit of a background idea of what the application that we try to develop is actually supposed to do and then I will hand over to Adrian for describing the sea ICD and in particular concourse activities and by the end we might be able to give a quick demo on what we're doing depending on how good week you're gonna be with the time so yeah if you have any questions feel free to to interrupt us we're happy to to pretty much answer everything so yeah this application the in a very very simplified view we're from Stuttgart there's a lot of like automotive industry and I I can also tell this is from the orden from automotive context there is a big big database which holds all the various configurations of the vehicles like as large objects and the idea is to bring that configuration into a fast consumer way to like clients and consuming systems and the original architecture that we found before we started was the overall application running on a WebSphere application server in a local data center and the client asked us for like a concept to migrate this to like a more modern architecture more modern platforms and also to like be able to get out of the data center this way the original architecture was developed in a way that the WebSphere application server would basically load all those configuration objects to like an in-memory kind of caching system and a registry would be responsible for maintaining the like cached instances now and there's like an API where the front will a consumers can access those cached instances now you might want to ask so why did you just not use a like a regular cache and did have all this architecture basically that's what I asked when I got in so there's actually a transformation happening and those objects look fairly different from what they are in here but they had some issues with the overall with your infrastructure basically this is a full monolithic like a single year deployment application that means whenever you update the code you have to fully like deploy everything onto new application servers which switch the load balancer or the routing and start all over again and that means always load all those objects back into the application server which is in about an hourly range which can be quite frustrating if you'd only want to do a minor change the other thing is the end of this was kind of foreseeable as those config objects would get more and more over time and right now or like at the point when we started Webster we're still able to cope with that but in the future it just might not those were two things there were more yeah just like a bit of text for all that so this country objects were pretty large in size they're many they change often and they need to be transformed the good thing with this architecture was the performance the downsides were as I said the large startup time on a losing approach this registry implementation was also kind of error prone and they were about to run into capacity issues with this set up now they didn't like give us all the freedom of using technologies and frameworks of our choice so one of the like the requirements they came up with said they want to do something on a kubernetes and docker based infrastructure and implement the application as a spring boots Springwood microservices so we said ok we're gonna divide this into like a loosely coupled distributed systems environment to give the benefits that we can update services individually while the others would still continue running and also to be better on the scaling side of things basically in kubernetes terms put all of the code each each configuration object into a known deployment with scaleable pots and then dynamically add routes to an ingress controller to make things accessible and take them out again initially we did this as a SPOC implementation and we were able to to demonstrate the advantages there so we had in we could shown individual treatment of like the startup of the overall architecture and then just start up of individual components while the other ones work same applies for the updates and individual scaling so some of those cached objects would be called a lot more frequently while other ones just idle and we're scaling we could we could go in the direction a lot better another thing that we did in there we leveraged the kubernetes api to to re-implement that registry I mean we found out that the kubernetes like service registry was pretty much delivering exactly the value of like how many how many instances of a certain object do I want to have in my environment so we could basically throw the old registry away and use the kubernetes api to access the kubernetes registry and serve exactly that purpose also the system kind of like navigates itself so it schedules new deployments or based through internal API calls and as I said eden removes the routes from to and from ingress so this is just to give you an idea what this is kind of about we were mostly involved of developing all those services right here like the RET like a registry an admin service and some like components who would schedule all the other parts and then we came to the point where I said how we gonna build and deploy all that and luckily they left more freedom to us in the decision of the tooling and that's basically the up to the point where we put any product concourse and said well we've seen this especially around the Cloud Foundry community we heard great things about it but we never didn't Life Project but we would like to give this one a go here and they let us do that so with that I would base it we're not definitely not conquerors experts we have applied this for quite a while now and I'm fairly happy with it but also we would definitely like you to if you see something and say well we've seen this different before and you could probably do this better in a certain way feel free also to give us feedback I mean basically Adrian will walk you now through the lessons that we learned on our way and show the state where we currently are and hopefully provide you some insights of what we discovered yeah now that everyone knows how interest our architecture looks like and if we know that how our architecture will look like it's actually time to build the apps and yeah we're only four well we weren't only focusing on building the apps but we were also focusing right from the beginning on building a ICSD pipeline because this is something that we and our also a client lock luckily what was really important to start with and the thing is it's not just the first question we had was not just about ok how do we go and build the pipeline the first question actually was what kind of pipe lag technology we are going to use and this is what this were basically the for once we are thinking about initially of course we're going to talk about pipelines we are obviously talking about Jenkins it's all reliable one we're like material said talking about concourse because it's new it's cool and so pivotal and then over there also a spinnaker and Jenkins eggs and right off the beginning we discarded spinnaker and Jenkins eggs and just because pinnacle was a little bit too big for what we thought was what we needed and the thing with Jenkins existed at the time when we develop the application in Jenkins X was still fairly new I mean it still is fairly new and it's like very heavily opinionated and we didn't like this thing and we want to like be flexible with how we build the application how we build our pipelines because we really weren't sure which direction we would eventually go so basically there were like Jenkins and concourse and the thing with Jenkins is that we wouldn't use Jenkins like just plain Jenkins and Lloyd in our cluster but we would have to use Jenkins as I manage Jenkins because our client and has some sort of IT infrastructure service and he hoped basically hosts Jenkins by himself so we would have to use this kind of Jenkins service and we really didn't like this kind of approach because the provisioning would take like two weeks or one month and you really had access to the Jenkins service and as I said at the beginning the really the goal was to start with the pipeline now so that was wasn't really really good and also one thing is that the manage Jenkins was sitting inside the data center of the client and our application was sitting somewhere inside IBM cloud but it was a private IP I'm private IBM cloud with all the public endpoints shut off so really wouldn't even know if there would be a connection between our Jenkins and our actual environment and so there was a really big problem and and yeah we would have to use to still the old-school style of VM workers with Jenkins and and this is something we really didn't liked and the thing with compris is that we would get Congress now like we know there are hand shots and we know how to use our charts and you know how to apply them and it's very easy to deploy Congress on kubernetes and yeah we would have full control about conkers and the thing we use with conkers and that's what we really liked and of course the whole concepts about containerized builds and that we have 1/5 our whole pipeline decoration sits this was a something really liked and but of course one of the cons was that yeah we weren't really sure at the time how Congress actually works how good it is and if it would constantly die on kubernetes we would have to take care of it and if it would suck like we were just doomed like it was our idea I take conquerors it's very awesome our kite love small like yeah but you know Jenkins it's yeah everyone uses Jenkins and we were like no and this is actually really like not the best idea just to take it because of that and yeah just as I said we really wanted to take Concours and that's where we actually you can use concourse and spoiled us we don't forget it or anything yeah yeah not at all so now we have our build server chosen like we take concourse okay it's actually time to build up to build a pipeline so our initial goal was like okay we have a code repo and we have our cluster yeah the first thing of course we want to do is actually get the code and pull it into a grenade into conquerors and once we have it in concourse we just built a code and we built the image because we are all deploying on kubernetes need to need to build a docker image out of it and then we push the image into a some sort of image repo and then we just trigger the deployment the apps the apps appear on the cluster and everything will be fine so this is what we had in mind initially and this is also where our adventure started so about pulling the code this was this is how our git flow looks like so basically a develop we have our tasks we're working in scrum and we're the task in our backlogs developer were to got some tasks assigned to him and then he would create a branch from the dev branch and create a new feature branch and if he had this feature branch he was working on that feature branch and once he created really the task once he successfully ended it and he were just like making a pull request into death and if this would be accepted inside of like a code review it would we were done to death and the task would be finished so this is like basically how we're yeah how our get worked and now the question was okay how do we actually integrate our pipeline into this sort of get flow and the thing was that we decided to actually build every poll request so just to verify that no code gets merged into death that is some sort of broken or that can't be built or deployed on kubernetes and for this we use the conquers result for from te OSS to get up PR resource and it worked really well like we were really happy with it and like one thing really liked about it where that we can use polling I know you shouldn't use polling but the thing is that our github repository where as we're lying where was a self-hosted get a repository and it was hosted in the data center of our clients and we didn't have a connection from github of our clients to conquerors on our kubernetes cluster which sounds a little bit weird but there was diversity case and we just couldn't fix it because it wasn't out of all it was out of our range yep and of course another thing we like to set we could we trigger the pipeline on existing committee so this means I make I think I am ready with my task I make a pull request and then I call it comes ahead and says no you know that it's actually not done you have some mistake inside of your task and I say ok yeah that's true I fix it I make a commit again and because I already made a pull request the whole pipeline triggers again and it's replayed and this is some nice feature we really like appreciate and what they are obviously downsides as well one is that you can't build all pull requests so if I make a pull request and five minutes after noon my colleague makes a pull request and my might changes burn be built again like they're not getting build again this is just the case I have to live with it and this is not yeah this is some unfortunate but it happened sometime and yeah how to fix I don't know if it's really flexible but yeah this is just the case of it looks like and another thing is that you can't always true get manually so if the if the repo is still really new and there hasn't been seen any it doesn't haven't been any pull requests wire the pipeline has been deployed you can't trigger it manually if you make a pull request and the pipeline actually sees the Porticus then you can afterwards trigger again even if the puller case has disappeared emerged but it can't be merged if you apply the pipeline and they aren't currently any pull requests of all they have been in the future and in the past so now that you get but now that we have the code it's actually time to build the code which is actually really easy like we were building sprint applications and we build it our cup of maven and yeah maven boots are actually pretty easy just maybe Keaney installer that's everything but one problem is that it's actually really really slow because we have to download every dependencies and we have an offspring applications which was yeah we which kind of suck but you actually can really easily solve this by caching the images into your interim image like you build a custom image with the dependencies you need and then you have then you don't need to download it every time so what building is actually really easy a little harder part is put a building the image of all it's not it's not really correct like this little twist with building the image and this is because we were pushing the image into a self-hosted a frog and the self-hosted a frog used a self-signed certificate and the thing is with kubernetes is that it only deploys images or basically pulls images through HTTP so there's no HTTP and the problem of that is because we have a self side self-hosted a frog in the self-hosted a producer cents a self-signed certificate we didn't have the certificate in south art inside our kubernetes cluster so we could probably fix this problem if you would have liked just and use a self-signed certificate and if he a certificate signed by a trusted authority but the guys who were like running the cell for such a frog one didn't wanted to do anything and we couldn't put their certificate into our kubernetes cluster since it was a self-signed and since it was a managed service and yeah that just they just didn't work so yeah this is this was the setup like pushing the image into a frog but you can't get the image from the j frog CHEO kubernetes to work to your core burn it is cluster if it uses a self-signed certificate yeah so what did we do we were on a VM cloud and because of that we use just the IBM container registry which has a certificate which is signed by your tarsal and trusted Authority so we wouldn't have any problems of that okay now we get to the exciting part deployment part and at the beginning we use the kubernetes resource and from this repository slap JP and what it does it basically just it basically just applies sample files or you can execute any sort of kubernetes command with this resource it's really easy as I said for applying singing download files but it gets really really harder if you want to do harder stuff and the lines of code you see over there and basically the lines of code which I use to set up the solar conquer at this whole and image a resource and because in order to like top of kubernetes you have some sort of permissions you need to have some sort of permissions and in order to get this sort of permissions into one thing and five ones in cluster I execute like the following one consider config view such as minify minify only take the configuration part of your current kubernetes configuration but only depart from your current cluster and flatten really makes it self-contained so you have any secrets which are appointed to inside the configuration it just puts the secret and puts it into one file so basically with this command you get one cube config file that has all the configuration needs to talk of your cluster and you just apply it and that's it and this you can like deploy to kubernetes and it's really easy but the problem is it really gets harder if you want to deploy and more stuff but recover back coming and yet going to that back later so now we thought okay we can deploy the kubernetes which is pretty nice this is what actually what we wanted but we still have one problem left and that this thing doesn't really support our hope flow that well and what a meme of that is and as I said we trigger the pipeline with a pull request and by making a pull request it just redeploys everything so if I make a pull request it gets deployed and I can look into my application which is deployed in kubernetes or if my tears five minutes later makes another poor request and the whole deployment gets overridden with this stuff and he deploys with his pull request so we did like that and in order to solve this kind of problem and we just said that for every pull request to every application and not say that later every pull request gets its own namespace so if I make a pull request it gets its own namespace in kubernetes but here's makes a pull request it gets on his own and names based on kubernetes and I can look at my namespace Matias can look at his name space and and this brings sort of a sort of grammar granularity where everyone can view his changes independently deployed on kubernetes and yeah this is basically what it looked at at the end so for every pull request that was it there was a own namespace and every app would be redeployed into this certain namespace so now that we fly all this kind of stuff in certain namespaces this okay so this worked for a certain amount of time what this really didn't work for a long time because as our application grows our Gamma Phi definition grow and we had we didn't had like only deployment fights we had service fights we had like service accounts because we had some sort of M security thing going on and then get roles and role bindings and all the kind of stuff and the whole thing was basically really really big and because of that we decided to use hand shots at the end and the whole thing with we deployed the whole stuff of ham shot with this kind of resource and from drinking out and we basically did the same thing but now doing it with hand shots and it worked better because in this with this hand shot we really could and configured the namespace it should deploy to like we also could do this with the old resource but with this resource we can we could configure it in that way that it would really check if the changes made to this that specific namespaces really worked so if I say okay I want to deploy to p2 in a space Peter minus 20 it really checks okay are the names other applications from Pierre minus 20 actually running because the thing of the other resource was that it only would check like one namespace which you would heart out and that really just didn't work for us okay so yeah I'm not on now to some more nasty things and the problem with that was with this kind of with both of the resources was that the thing is for in our like our cloud provider IBM cloud used like all of tokens for the access to the kubernetes cluster what the thing was that they just expired after some time like four weeks or a little bit longer and whenever expired we need to push the applique the configuration for the cluster again and we also needed to refresh the whole configuration for the up for the pipe ends again which this kind of sucked but this is this is really where the case for our setup or basically our authentication we have and yeah we also try to implement key tops and the thing is I know if everyone heard about get ups basically in like really short it was basically it's the whole idea that you manage your whole infrastructure or the whole apps that run on your infrastructure through bit and you version you and have a description what is deployed on your cluster and this the description gets deployed into kit and based on that your whole cluster changes so and yes then you have this picture which describes a little bit better and we try to integrate and get opposite into our pipeline and we did this by saying okay now we deploy with a hand shot and a hand shot has a certain configuration this configuration decides which apps actually gets deployed so let's say I make changes to the code repo and macey make a pull request and this triggers the whole pipeline it gets version it gets built and in the next step it takes the current configuration of the cluster it takes it and then it menu Polly manipulates it in such a way that it takes the newly built version like the newly built image puts it in the configuration and reupload reuploads it into the config repo and this triggers a redeployment of the of the of every app because you have changed because you have changed this one thing so let's say I have changed one thing in my app it basically says okay this is my new app with this version tag I take it I float into the hand shot configuration and our everything gets really ployed with this one help with this one configuration like obviously not everything on every app just that app that I changed because hence modern enough to like see okay he only changed this part of the configuration so I only redeployed this part yep and if the thing is that it's it's actually a little bit of absolute key tops and because we say like in this step like the deploy step that it it will actually only trigger if I run the pipeline like if I buy myself make sure this to be good to the config repo it it won't affect the cluster at all so that's why it's kind of get ups but it's not like hundred percent get ups in my opinion and yes now we at the end of our of the presentation and this is like bits of basic light and summer we have Tov it and like we pick Congress to have to full control and it's actually lightweight and it we really did have to full control and it was lightweight like it wasn't it wasn't really hard to manage it at all and yeah we were running good on kubernetes which was really easy deployment was really easy and the thing with the major maven builds it's a little bit problematic but it just takes you a little bit of time like it's not that much of a problem and we definitely recommend the whole TOS SPM the TOS as github pull request resource it's it's really good three awesome and we get some certificate certificate issues if you use a self-hosted yeah itself was image registry and of course the whole thing with the snapshot the namespace and concept was very helpful because we had to go narrow T helm was awesome it was really easy to apply with it and yeah we were still not sure about the github thing what we are actually keen that if you look into it a little bit more tips will work and that you can obviously am use key tops or implemented ops with conkers yep so yeah that's busy and now it is what I brush kind of brush over it a little bit or actually very fast and so if you have more questions and yeah we can talk about it afterwards and I'm happy Ansem [Applause]

Channel: EngineerBetter

Views: 1,202

Rating: 4.4285712 out of 5

Keywords:

Id: fYNLAXvCe50

Length: 31min 12sec (1872 seconds)

Published: Tue Jul 23 2019