Kubernetes, Ingress and Traefik Usage at CERN

hello everybody i see people coming on in and i am so thrilled to welcome you to today's traffic online meetup thank you so very very much um this one over to you ricardo thank you so very very much for joining us and um [Music] i personally am super excited for today's webinar or online meetup it is with cern who does amazing huge experiments with stuff um about our great grandiose beautiful universe which i'm totally into um and so we get to learn how they're using traffic to manage all this data what their stack is and we get to hang out with ricardo rocha who is a really dope guy and so uh to get started i'll tell you a little bit about me i'm patricia dugan and i am the head of community for traffic proudly i represent our huge community huge and growing community and um basically i put together this program to showcase the ways that companies are using traffic to solve interesting technical challenges and the things i'd like you to know about our three-fold one is we just came out with traffic pilot which is a sas um control platform that can manage all of your instances for you and give you security alerts so if you're not familiar with that you should definitely check it out and i also designed the traffic ambassador program which is a group of people who are very radical and they're contributors of code content and community and if you are a traffic ambassador you earn uh rewards promotions of your efforts and also you're invited to a private traffic discord server by which we do very cool stuff and you get direct access to many cool things so if you're interested in participating please drop me a line and then the last thing that i'd like to say is if any of you have amazing use cases around traffic i would love to learn more about how you're using traffic and see if there's collaborative things we could do together with you and your company if you would be interested so you can find me on twitter at patricia underscore dugan you can find traffic at traffic and um today what we're going to do is listen to a ricardo uh talk about his work at cern and he's going to show us some beautiful demos and and we're going to be blown away by just the scope of how big their operations are and then as far as q a please drop your questions in the chat box or the q a module we'll answer them at the end and if we don't get to them we will follow up with the answers to those in a gist and i will be sending you the recording of this meetup via email so i look forward to getting to know you please email me if you need anything at patricia contain.us contain us and now i will say thank you again for joining us and hand it over to ricardo and uh enjoy your time today with us thanks again for joining us cool so i hope you can hear me hi everyone um i'll share my screen quickly cool can you see my screen hopefully yep for some reason i can't it's black right now though yeah for summer it's floating [Music] again okay we're just talking about something not working quite as expected i forgot to pray to the demo gods yeah actually um i can probably share the slides in another way if you can't share we you can always if you need to go out and come back in or or what not yep thanks for your patience everyone the other option is i just share like a not full screen that will probably also do it okay do what you gotta do we'll be here you can see this right yep yeah so let me see if i can do this in a way that works because for some reason okay is this good enough if i leave it like this i'm okay with that if if everyone else is unless you want to try again um anyone who is not in favor of this holler but otherwise let's tr let's just do it if you can do it we could do it like this okay i mean as far as recording goes it's okay yeah i think so just try once more it's not gonna work okay welcome michael everyone okay so i'll start right here here okay cool okay i'll do it like this then um so hi everyone um my name is ricardo i'm i work at cern in the cern cloud team i'll start with just by introducing myself quickly um i'm a computing engineer in the cern cloud team i focus mostly on containers kubernetes networking aspects of our clouds and as well as things around accelerators so that means gpus and other types of accelerators we started doing quite a bit of machine learning so i also work in that area to support our users previously i worked in the wlcg which is the worldwide lhc computing grid the lhc is the large hadron collider which is the largest experiment we have today and the computing grid is a large network of sites that we've built over the years i'll talk a bit about that as well i leave you my contacts here so feel free to um to um reach out if you have questions after um or yeah we'll we'll cover quite a bit today hopefully but um we can then go follow up after as well so i'll start by introducing a bit cern and what we do uh and then describe a bit uh our infrastructure and how it has evolved with time and then i'll jump into the the containers and traffic topic of today so one small second uh there are at least there are two people saying they can only see black um can you see though can yeah i can see it and i think maybe also demographically okay we many people can see it too so it's a i would go out and come back in and now i'm going to stay out and let ricardo rock and roll but should i get out okay yeah i think we're good uh do you want me to get out or or just share it no not not you people who can't see it should probably go out and come back in okay fair enough so hopefully this is all good okay so i was introducing cern so cern is the european organization for nuclear research we are based in geneva in switzerland um founded in 1954 and our main purpose is to do uh fundamental science and what that means is trying to answer like uh big questions about the universe uh which is what is 96 percent of the universe made of what is dark manager matter and dark energy we don't quite understand completely uh a lot of what is made of uh what was the state of mata just after the big bang a state we call parkland plasma and why don't we see any antimatter so theory is that we should have uh the same amount of matter and antimatter but actually um we only see matters so we try to do some research in that area as well to answer these questions we build very large experiments scientific experiments the largest one today is the large hadron collider so in this map you can see the drawing of the particle accelerator the lhc it's 100 meters underground it's 27 kilometers in circumference um and in this accelerator we inject two beams of protons or one clockwise one anti-clockwise and that's some specific points we make them collide and we build these very very large experiments and you can see the names here the four main ones alice cms lhcb and atlas we we make these protons circulate to very very close to the speed of light and we make them increase in energy and then we make them collide at these specific points to have an idea of the size you can see on the bottom right the geneva airport the runway so it's pretty large and the one thing that is also quite interesting is that this is in the border between france and switzerland so actually depending where you are on the side you might be changing country as you move along so if we would go to the tunnel so actually uh we are currently under maintenance for upgrades and it would be a great way a great time to come and visit because you could go down and visit the experiments but actually with the current situation uh this is not possible but um we are doing upgrades but if you go to the tunnel it looks something like this you can see uh the particle accelerator these are superconducting magnets this is where we we inject our beams and one thing that we often say is that cern is the coolest place in the universe and the the the reason for that is that um to have this magnets being superconducting we we lower the temperature to very close to absolute zero uh to i think it's 1.7 kelvin um and we have a lot of cryogenic systems to to achieve that um then when we make these beams collide protein beams collide we have these massive experiments so these are also 100 meters in the ground this is the compact mullen solenoid cms which is not very compact the cavern is like 40 by 40 meters the detector itself is uh 14 tons and you can see for scale um a couple of uh human uh like in the picture you can see the bodies so it's pretty massive and how this works is uh this detectors act like a giant machine [Music] camera uh it's not an actual camera making pictures but it it's pretty much acts like one and it takes 40 million pictures per second and basically what we try is to to see what resulted from the different collisions that are happening and it's billions per second and then from here we did this this data it has to be stored and collected so this is an example of a collision as we reconstruct them but to actually get to this where we reconstruct the collisions and we try to see what happen we need to record a lot of data and analyze and this analysis gives something for that for physicists it's quite spectacular but for the rest of the world is a lot less which is a plot so it's not as grandiose as other domains of science displays but this for example is the cms plot that showed um proved the existence of the higgs boson back in 2012 and that led to the nobel prize in 2013. we can see here that the blue part is the simulation data or the known physics that we know uh already pretty much everything about and then you have the black docs dots which are the actual data that we saw in the collisions and then you see the red bump which is the uh predictive physics for for the existence of the higgs boson and you can see that when the data matches the the the prediction then with with uh enough data we can can claim discovery so this is what we all work for pretty much we have other experiments also on site apart from the large hadron collider so one of them is the antimatter factory which is very close to the building where i work where we basically create antimatter when we try to see to understand better its behavior and try to understand why we don't see it in the universe and then we have an also an experiment ams that is installed in the international space station and does other kinds of of scientific experiments really using cosmic rays as well because of all the people we have on site working and the requirements we have to exchange information um cern also develops quite a bit of technology around i.t uh one of the most important ones was back in 1989 when tim berners-lee uh created uh what would be the world wide web um and the goal was to have a way for physicists to easily share information between them um and the standard way to access this information as well this is the paper from the initial proposal and then in 91 cern uh decided that this was a technology that would be useful for a lot more people and then made it public domain and it grew quite a bit of course so to analyze this data uh one thing i mentioned when we saw this detector where we detectors where we do all these collisions actually the amount of data we generate is uh something like one petabyte a second this is of course not something that we can process so i can show you this again so out of these detectors we have like a small pit that will do the first step uh filtering of this data and which will put the data down from one petabyte a second to something like 10 gigabytes a second and this 10 gigabytes second are something that we can store and analyze but we still need quite a bit of resources to do that so we generate something like 70 petabytes of new data every year and we don't uh delete any data so we are close to half an exabyte these days we run a private cloud in on premises which is based on openstack and here you can see that it's quite large we have like users above 3500 unique users lots of projects define more than 30 000 vms and close to 10 000 hypervisors this is just for the compute and and the cloud then we have a lot more servers that are dedicated for storage one relevant uh number here is these magnum clusters which are actually the kubernetes or the container clusters that we use which are quite relevant to this talk um will mention more but they've been growing quite a bit in popularity all this containerized infrastructure so this is not enough so we have something like 300 000 cores we saw here in this number uh close to 300 000 cores on in this private cloud but we need more so well in the last 20 years we developed this great um infrastructure um if you were around this area in this domain 20 years ago you probably heard about grid computing so we have more than 200 sites around the world that participate and collaborate with us to to help us analyze and process all this data so we basically double our capacity thanks to this if you look at it at any moment you will see something like half a million jobs being run um and we developed all this uh infrastructure that is connected with the dedicated links and has a bunch of orchestration to decide where to send these jobs around and uh i'll just finish this overview of cern computing this is quite interesting i think to see the evolution of computing at cern uh in the picture on the left you can see how our data center looked in the 70s so it's a pretty old building you can see some cray mainframes on the back uh then around 2007 i was already at cern uh we decided to start doing rack based uh deployments and but we actually used commodity hardware like the stuff you get from the shops and you can see the these boxes that were not the most um like efficient power and cooling efficient uh way of us having a data center so they actually didn't last very long one interesting thing you see here on the right picture is also this silos for our tape robots so one one interesting part of our infrastructure is that we still rely on tape so all the data is written to tape and we use it mostly as backup but uh we we have very good experience with tapes we have some tips that are 30 years old that still work quite well and it's a very cheap way to to have long-term storage as well so we have a lot of storage in disks but we also have quite a bit of tape and then of course this is the how it looks today so it's a much more uh recent way of structuring a data center with a bunch of racks and cooling aisles and all these normal things you can see though that the building is quite big still so it's not the easiest one to to manage the cooling but uh but it's a quite modern data center um in in the way we deploy things and it's fully automated uh using openstack and other tooling uh so one curious machine that we have in our museum now it's uh this uh next computer from that was what uh tim berners-lee used to to create the world wide web um and if you come and visit us you can see it uh the interesting thing is that it also had the sticker that says this machine is the server do not power down at the time it was one of the two machines running the world wide web so it happened a couple of times that someone in the office would power this down and uh so he wanted to make this clear that this should not happen again if we look at the evolution so i talked a bit of how our data center evolved it's also interesting to see how our computing evolved as well with it so if we look at what we used to have with pure physical infrastructure provisioning was kind of problematic um it could take days or weeks to get a new machine you will have to open a ticket wait that if the machine is around it's set up or maybe it has to to be bought as well so this was kind of uh um could could take quite quite some time any kind of maintenance was typically highly intrusive um this idea of having multiple replicas was okay but actually the downtimes uh could be quite important because they were basically shutting down physical machines where a large fraction of your applications were running uh deployments and updates we were already using uh automation tools configuration management tools like puppets or i don't know in the house we use pop it but it could be chef or ansible so it means that it can take minutes or a couple of hours for everything to be set up uh the utilization um was quite poor because very often we had to isolate the applications which meant that even applications that do not require a lot of resources would be dedicated in a physical machine so uh virtualization and especially this api based way of interacting with uh with computing resources uh was really a breakthrough at the time this meant that provisioning of new instances like virtual machines now could take only minutes the maintenance uh because we have in a lot of our capacity the ability to do live migration this could be a lot less intrusive and before operating um doing maintenance on on a node we could just like migrate the instances uh deployment and update is pretty much the same as for the physical machines and utilization uh increased a lot not only because you can have multiple virtual machines in the in the same physical box but because you can over commit the resources and and like pack applications that do not need a lot of resources and then containers which is the reason why we are here as well in this case provisioning is is quite a bit quicker it can take seconds especially if you're just scaling out an application even new deployments just take a couple of seconds maintenance is definitely less intrusive and here the main reason is because um if you use an orchestrator for containers you're expressing a lot more about your application than what you would traditionally do with a virtual machine you will tell us and and the orchestrator a lot more about what kind of uh things your your application can can deal with in terms of moving it around so we can automate a lot more in this in these cases uh deployment and update is uh literally seconds to to to do upgrades on on our applications and utilization i put here that is very good because there's an improvement by not having uh this kernel layer and the improvement is both on the compute side because for example for our batch systems we saw that by using virtualization even after a lot of optimize of work on optimizing the configurations we still lose to almost three percent of the compute power and if you have a large infrastructure like ours actually two percent can be something that you would be very happy to get back and also because we don't have this extra kernel we also get a lot of memory back not only compute so that's uh that's why i mentioned here that containers actually have there's a motivation also in this area so the three key aspects that we see in containerized infrastructure the first one is simplification because you know more about application traditionally applications will also have very good integration with the tools for monitoring if you're using something like prometheus the life cycle is is uh integrated into most of these orchestrators the alarm is also common that it comes in the pack then the simplified deployment so we have this simplified this uniform api that is kubernetes these days which means that we can really focus on one way of handling most of our workloads replication is also part of the definition of this deployments load balancing is also part of it so all the fact that the applications come with all these definitions already integrated in one standard and uniform api helps us a lot in simplifying our operations and then one thing that happens at cern is that uh we have um eighty percent of our capacity is what we use for batch which is uh basically like the main processing of the data that is coming uh and then the rest of the capacity capacities for analysis and user analysis and other types of workloads but we have spikes if there's an international big international physics conference it's very common that the weeks before people are like jumping on their analysis to finish their papers and also every couple of months or a couple of years people do reprocessing of all data to try to change the calibration of the detectors as well and see if the if the results change so if we have these campaigns we also need extra capacity so the fact that we have this uniform api it means that we have the capacity to go look for resources elsewhere and try to just expand our infrastructure and this is much simpler when you have something like the kubernetes api available in in this other infrastructures so that we can burst more easily our our capacity so i'll just mention a couple of use cases that we have been using for containers um so the first one is is really cool so this is uh the picture on the top left is the atlas detector which is uh one of the two big ones with cms as i mentioned it generates something like one petabyte of of per second of data typically this filter is split in what is done purely in hardware which is the first uh filtering and then uh the second filter is based on software with a as a form that is very close to to the detector so we i mentioned we do something like uh 40 million interactions per second which is what i mentioned as pictures there are 3 000 physical nodes and 30 000 applications running in this cluster so it's it's a very cool use case for something like uh running containers and kubernetes so we've been doing studies um with them to to to try to to see if a containerized infrastructure could help with managing the their farm especially because a lot of what was doing being being done by hardware is now moving to to software in some areas as well because of the advent of gpus and other accelerators so what we've been trying to do is study to see if in the next future runs for for our experiments if we can change this infrastructure and improve the setup what happens with this experiment is that once you decide what is going to run during the the period of this run which can take like one year and a half two years you don't touch it so we make a decision and we if it works we keep that set up um during that period so we've been doing this uh for a couple of years already we started with kubernetes 1.5 we are actually redoing the the study now to see um and the results are quite promising another cool use case um that i think is it's quite um in it's coming up in other uh members of the community uh around kubernetes is um that we have all these sites collaborating with us and we need to distribute the containers um to this side so we we with time we developed uh we deployed a system that we call cern vmfs where we would push the experiment software to all these sites in this hierarchy uh where we first do a release of the software push it at cern and then there's this kind of cache layers uh to act like squid caches all around the world and we only pull what what is actually needed for for the local site this is a quite efficient quite uh nice and efficient way of of distributing our software now as applications move to containers this started posing a problem because like traditionally docker will just pull the image and then launch the container so we started looking at this we first developed a component that was a craft driver that would basically it's called what was called this was done by the cvmfs team and it was called a thin layer which basically instead of pulling the full image you just pull this thin layer that tells you where to find the actual image and the and then you do kind of a fuse mount of remote file system and you just accept that the full uh image is available in this mount and as the container itself will require files it will start holding them uh from the cvmfs file system so we did this as a graph driver for docker and their support now in container d through this remote snapshot and so i'm i put the link here it's a quite interesting work uh there's an um a plugin for what's called star gz which is seekable tyraj said as an alternative for docker images uh but uh we the team of cvmfs has been developing also a remote snapshot for our system uh the results are brutal we have very bad users that uh have images of like 10 15 18 gigabytes this can take like several minutes and bring down our registries if you have a large enough cluster uh with this uh with this driver what we've seen is that the time to start a container is pretty much flat at five seconds five six seconds and then you you only pull the jobs actually only pull like six percent of the data so we actually gain a lot in also in network and bandwidth and i mentioned machine learning so that's also something that we've been investing a lot in [Music] in in our containerized infrastructure so we are pretty heavy users of gpus these days with kubernetes we are expanding our uh gpu availability in on premises as well and we do here the example i have is this deep learning for fast simulation so we need a lot of simulation data i mentioned that we generate a lot of real data from the detector but actually we have a lot more simulation data than that and generating this data is very expensive um compute intensive so we've been exploring machine learning tool to help with that as well and then i mentioned the grid so one very cool thing is that to run a grid site you need a compute endpoint you need a storage endpoint you need the monitoring endpoint so we started realizing that actually kubernetes offers most of what we need in this grid sites in this large distributed system we have so um this is the example of one of the experiments that started trying um running a grid site just as a kubernetes endpoint and these are the results so actually we already have production sites running which uh behind only have a kubernetes and kubernetes cluster we find some funny things like here uh if you have eight cores and one core jobs you can see that you will have some challenges to to pack everything together and make the best of the cluster so there's there's a lot of work going on on scheduling in kubernetes that covers for this as well so yeah if you need any more information about these topics um just reach out as well and i'm happy to discuss them so i'll jump now to like our kubernetes infrastructure itself and how we use traffic and then i'll i'll hopefully leave quite some time for questions so this is an overview of our kubernetes infrastructure i mentioned that we offer it kind of uh like a public cloud is doing we offer it as cluster as a service for our users so they can just come up with their apis create a cluster and play it around and then of course for the largest clusters we are we are a bit more attentive at making sure that they have things like high availability well well set up and kind of look closer to them but they have the ability to go and and do this uh so in addition to defining like how big the notes should be and things like this they also can enable disable a lot of different features and one of them is of course the ingress controller so i put here like there's a lot more features that can be uh tuned but i i highlight here for this stock uh the ingress controller so traffic out of i don't know 500 clusters we have uh 390 are using traffic today so this uh something i took a couple of days to build this screenshot uh so it's uh by default setup uh uh the defaults the ingress controller for our setups and i'll briefly mention a couple of things we did to to ensure that integration works quite well so i mentioned to create a cluster we have this openstack based private cloud and uh the tool we use to orchestrate clusters is also coming with openstack um basically what we do is we offer these public templates that are keyed on the version of kubernetes but actually internally they have like a lot of features enabled disabled depending on what we want to offer to the users these are what we give them by default and then users can can customize uh their clusters as they want when they deploy them so for example here i have an example of a cluster creation where you pass the version of kubernetes you want or the template and then you say i want 10 nodes of this kind of this flavor but i actually need to enable gpus so what this means is that in the deployment we'll have a component that will make sure that the gpus are well configured and exposed to to the containers that request them and also that all the monitoring is is including gpus as well and then we also have the ability to partition the clusters so it's very common that you don't want your all your nodes to be the same you might want some nodes with cpus only or another with gpus or you might want to split the node across availability zones the cluster across the webisons so this is also something that we support with these node groups and the stack looks like this so we our base image is a federal core we used to use fedora atomic but then with the corrosive acquisition kind of a merge so we are now using fedora core and this immutable operating system was something that users were kind of [Music] not too happy at the start but actually they they got used to never logging into the notes and doing a pure um kubernetes based interaction with the cluster and then the runtime we are using container d with run c and uh we are using the cri interface so we have cri continuity uh we do uh kubernetes is our main container orchestration the reason i mentioned this here is that at the start we actually support it as well uh swarm and dcos misers so we deprecated support from misos we still have a couple of swarm clusters mostly when people just want the docker api and that's it but kubernetes is like i don't know 99 or 95 of the clusters today we use fluenty for log collection and aggregation and prometheus for monitoring and metrics the way we do this is we have the logs and metrics being collected inside the cluster with a but lasting like two weeks with a high granularity but also we also give the option to publish to a central system uh for long term storage of the matrix and the logs and here we kind of do some kind of we often do aggregation and the reason we need this central collection is that uh the way people have been doing is migrating their services uh to to kubernetes one by one and often they need to do correlations between metrics in different services so having something that is not self-contained in the cluster but also outside kind of eases this task so for traffic this has been our default ingress controller from day one um the reason we we chose it is because when we are starting to use kubernetes and we did it quite early um it was the one that was most dedicated to integrating with this cloud native uh tooling uh had a healthy community we met uh people at a couple of conferences uh the interactions were very good and also our initial use cases it covered all of them um so i'll give a couple of use cases that we use uh the most simple one is actually we'll show this one first so the the simplest uh use case is this uh simple http or https so in this case i would say like 90 percent of this of our services this is enough um so that's all you need not no no need for a lot of annotations it starts um the only annotation is to expose it with https but actually this made it made kind of the point of using something like kubernetes also for ingress because it was so simple to define and all the magic was was hidden the reason all the magic is hidden is because we have this kind of extra integration so one thing that is common is that you you just want to have some kind of dns based uh selection on the ingress which means that this has to be some kind of dns integration so we have internally this network database that we call land db so what happened is the the way we do this is we deploy traffic as a daemon set and then but uh with replicas only on nodes that have this role ingress uh so we need to select a couple of nodes that run ingress of course if we would do this just statically and this is the blue dots i have here the blue circles if we would do this just statically this wouldn't work because if nodes crashed then we would lose the ingress controllers so what we have is a small component that runs on every cluster that basically is watching changes in in the via the kubernetes kubernetes api and what it's doing is watching the nodes in the status so if it sees that some nodes are unavailable or crashed or whatever it will move to this role ingress to other nodes to make sure that two or three nodes in the cluster are always serving ingress traffic and when it does this what it also has to do is make sure that our dns database is updated so that the ingress deformations are are are propagated so if we have a bunch of ingresses that need a specific alloy alice dns alias then we just update the network db so that those aliases point to the current nodes serving ingress this is uh not ideally the ideal of course but the the main reason for this is that uh we have a flat network provider network at cern across data center so we don't have this capability of doing a software defined network yet which means that we can't virtualize the networking very well so we kind of have this this setup here what this means also is that there can be some some hiccups when dns is being updated because of caches and and time to propagate the dns updates uh so it's not ideal we are working on on improving this but it's pretty easy like it kind of shows the point of of doing this having this way of defining the all the pieces of your cluster via these resources then you can very easily integrate with things like this same as you do with operators and other kubernetes components so then the other very very popular option is uh to auto generate this sell certificates this was also like a huge uh change for us it's very popular now the issue we have today is um is that dns challenge the using the dns challenge of me is not an option today uh we don't have an api to update the txt records in our dns so we rely on the http challenge which means that we need to open the firewall to get a certificate and to renew it and this is fine but it kind of limits the usage we have for for acme and let's encrypt internally and it also means that uh um we have kind of a chicken and egg of like the security team will want a certificate before opening a firewall but we need a firewall opening to get a certificate it's kind of has to be negotiated but this works extremely well we are working on the integration of dns as well which will make things much easier and then one use case that is also quite common and this is mostly because of this great services that we have we across the grid we use x 509 certificates for a lot of things which means that we have to push um the certificate all the way to to the to the application so that they can validate uh the users so we do this with annotations uh with traffic uh where we do ssl termination uh but then we we push the the client certificate uh along with the http payload and this this also works quite well there's a couple of other requirements that we have and this is the reason why we support other ingress controllers as well so we need to do in some cases expose directly the tcp ports um and we also need to to do ssl passthrough in some cases so this so this work uh functionality that was not supported at traffic when we started looking at it um it might be something that recently was added but i actually had a look quickly and i'm not completely sure it's it's there already and then one kind of dubious use case we have is to do a redirection based on headers uh we have i see dws it's it's a real use case but we have only one user that is requesting this so it's not something we've put a lot of effort in because they they managed to find a solution uh for themselves uh so they're pretty smart they found the work around um so we don't we don't put a lot of effort on that so i come to the end hopefully leaving some time for questions um so basically what i would say is the traffic has been really stable in our deployments we basically have no problems we are still using one x versions we didn't transition to two zero yet uh 2x um but we we plan to do it soon um again it's the most used ingress controller we have um almost 400 clusters are relying on it if i look to the next steps i would say that the main one is to do the integration of ingress with this external bouncer so now we have an ability to have an external load bouncer with the virtual ip assigned to it which means we can stop having this need to do the update of dns and just uh rely on a virtual ipe and just manage the members there so that's that's a very something we are working right now and the other one is that all this work that is being done on the new service apis in kubernetes so this is something that we are pretty excited about i think this is uh this will be very helpful um especially in cases where the number of use cases is quite broad and you end up relying on too many annotations it would be nice to to kind of have the next generation service apis uh being a bit more uh allowing all these use cases without having to to fall back to these annotations so that's so i have i put a link here in case you're interested we actually run some webinars sometimes as well so we cover all the topics including like ingress and little balancing as well and i'm more than happy to answer all your questions you have my email as well at the start so hope uh hope this was on topic um hey great job could we put that link in the chat and then we have several questions to go through if you're ready to take them we'll get started uh you mean this link or yes cool okay pardon um so thank you everyone for all of your questions we had some great ones and i'll i'll get started uh now um there's your link to you abhishek thank you so much so the first question we have is what is your kubernetes upgrade strategy and do you have a dev and staging area as well also he asks don't you use grafana for viewing yeah um so regarding the upgrade strategy um the recommendation we give to all our users today is as much as possible to to because we have this new service that is load balancing to expose their services via a load balancer as well and this can be with ingress and then when they want to upgrade actually deploy new clusters and we have some tooling to migrate the work uh to kind of migrate the the nodes the size of the cluster gradually from one cluster to the other and the reason we do this is that we had some some issues with upgrades in the past um we like for our critical services we run multi-master with hiv-ability with the load balancer on top but we've seen cases where the upgrades didn't work quite well so our main recommendation is to do that to have some kind of uh load balancer on top and migrate from one cluster to the other this is also quite popular for people that are migrating from virtual machines to kubernetes they will have a load balancer where members will be backed by virtual machines and then they are their cluster and they slowly migrate the workloads from virtual machines to kubernetes by killing the vms and and increasing the size of the cluster we have support for in-place upgrades but they only take care of um upgrading components they don't do reconfigurations today right on and the referenda question so we have prometheus and we do graphene on top yeah okay cool and and actually um michael had asked what are you using for prometheus aggregation is that thanos or cortex so um the way we are doing uh we are not doing like federated prometheus we have a centralized monitoring infrastructure where they run their own prometheus instance and they configure each cluster as a parameters uh endpoint and there's a set of metrics that are being collected and aggregated so it's kind of uh in-house solution in-house solution right on okay um and do you have any ci cd tools yes so we um we push uh everyone to use um like this new githubs model so we promote this very much internally with its training sessions and we do our recommendation today is to use uh flux uh with helm so we we try to get everyone to deploy their applications with home and to automate this with flux and we used uh subs for for handling secrets as well in the configurations so basically we are doing flux and some people are doing like releases in single branch some people are doing multi branch uh to manage the different uh deployments uh there's a couple of people using argo cd as well and there's some effort to kind of merge a lot of the functionality into a single library so that's something we are very interested as well cool and uh how do you push the locally hosted cluster logs to centralized infrastructure yeah so we do fluentd in the cluster and then we have an http endpoint or several in our in our cloud in our infrastructure that basically all the services can push the logs via this http gateway uh so basically this is what we're doing fluently http but then the backend is actually elasticsearch so we could eventually just bypass this hdb gateway the reason we do this is because we store an elastic search but we also store in hdfs for long term uh log analysis so it kind of give us the option gives us the option to push in one place and then propagate to multiple back-ends cool thank you um and how do you set up https ingress using cert manager and traffic crd uh so we are we're it's some people are using insert manager so but we are using acme directly so we basically maybe i should i should have shown an example here but um we just configure the act acme in the in the ingress traffic demand set uh and then yeah we leave traffic to do it we are not doing insert manager in by default some people are yeah that answers the question um and then the baby wants to answer too um have you considered something like metal lb for your ingress controllers um no so for the load balancer we are relying on this external load bouncer which is also part of our openstack deployment so it integrates very well the reason is because we use uh openstack to manage these clusters as well we can also do we have all the tokens that we require to talk to the rest of the services in the data center so we don't need to add any kind of additional token to sp to talk to specific services we just need one token so it kind of makes our life quite easy to have this single authentication authorization mechanism very cool uh how do you manage cern how do you manage at cern the different networks isolations between your projects and the fact for traffic to access your projects across the clusters and vms right uh that's a very good question again the answer is we have a flat network and for most of our data center it's a flat network um so basically we can route across um and the other part is that uh i guess because cern was very early in the internet we have a very large amount of public ips so it's it's pretty easy for us in this respect to to just rely on a flat network and pretty much everything we'll get a potentially publicly routable ip for some some like critical systems we have like a technical network that is isolated from all the rest but in fact with the in addition to this load balancing service that i mentioned that we introduced recently we are also adding the possibility to defining private networks and we have a software defined networking layer in part of a data center now okay thank you pierre um are your nodes with the ingress roll doing only ingress and how big are your nodes um and then are they just network optimized to forward traffic along uh so they are not uh dedicated to ingress um by default so usually they will be randomly selected the size of the nodes is really dependent on the on the workload so we have people doing i don't know the internal map service or uh our pension fund or and then we also have like physics services that handle a lot of data so it really depends the main point here is that we use ingress mostly for services for any kind of data intensive workloads and because we have to to read a lot of data we actually have those containers um skipping the the network namespace and just using the the host network and we basically skip all the all the network isolation in those cases where we really need the bandwidth for the data cool very cool thank you um who manages the dns uh our network team so we have this network database and then the dns is uh populated from this network database so we kind of have an api to interact with this database uh but it's um it's not like each project can have their own um like sub domain or anything we have one dns so that's why we don't have this flexibility for example to to like decide which records we can edit and things like this but uh like the use cases are real to to and in this case we are working with them to to extend the api to allow uh editing these txt records as well uh but yeah it's a separate team from from ours cool let's see did did you have a use case for stateful workloads running in kubernetes like any database with headless service and tcp based ingress also how does traffic support these services right so we have a couple of uses of users uh that require access to databases and in that case uh it can be like in some cases it's more traditional uh databases but in we also have this uh um i i don't know key value store distributed databases that still need uh persistency in this case there are two options um in some cases they run the front ends for the databases in the kubernetes cluster and then they rely on a central storage system to store the data for each of the replicas and in this case pretty much the service endpoint is is uh stateless so in those cases you can still run a database of course the performance won't be ideal for the other cases where you need to to run multiple instances and store the data in the in the cluster and you need tcp access we are using in those cases we support nginx and at the time it was a solution that worked for our users so we are exposing tcp ports using nginx in those cases very cool and then uh just a couple more and then we're on perfect time how do you manage and segregate the cluster and related resources like ingress in the thousands and is it divided among a group of people um so is it like multi-tenancy club multi-tenant clusters something like that let's let's say yeah okay so we we traditionally we do multi-tenancy by having multiple clusters and recently we started having large enough clusters and use cases that justify uh this kind of uh like multi-tenant clusters in-house um but this is quite recent and we've been exploring um things like resource quotas um uh to to be able to to to kind of partition the the cluster resources between multiple tenants if the question was more like splitting traffic across large clusters we have i talk a bit about optimizations we've done for the scheduler and the kubernetes api to support a large amount of pots being created and deleted and also when we split across availability zones we also uh take this into account when defining the ingress node so that we split the load also across availability zones okay beautiful um and uh what are the strategies followed on this deployment to limit the max header size and buffer size uh yeah i'm not sure actually i don't think we hit too many too much of those issues i i can go back and check if we actually have any any tuning for that not that i can remember right now cool and then we have one last question which is uh what is the database in use for kubernetes clusters or for the services so for kubernetes we are just using hcd for for the backend of the of kubernetes and then for databases you can name it like it's a big place we have a lot of projects multiple teams and uh you can find mysql postgres some people use mongodb cassandra yeah you name it you'll find it we have everything well beautiful um well done and and as suspected the questions had a great breath and and you you did a wonderful job answering them so thank you so much um thank you to our attendees who stayed with us this whole time and for the show um and thank you you're welcome daddy um and uh ricardo everyone is saying thank you to you so thanks for working um so what happens now is i will produce this recording and email it to all of you um uh feeling the love everyone thank you so much be well and uh and um you can find ricardo on uh on twitter uh what's your handle again i'm following you but i forget it put it here a-h-c-o-r-p-o-r-t-o so we'll see you on twitter and ricardo um the floor is yours if you have anything else to say and uh then we do a virtual fist fund no i think that the main the main thing is they are thinking that traffic for all the cool software we are like large users as you can see and also thanking all the community for from traffic for for um all the efforts and uh we look forward to continue like collaborating with everyone in the near future cool okay everyone stay very well we have a big life ahead of us and uh ricardo virtual fist bump thank you so much yeah cheers

Channel: Traefik Labs

Views: 1,228

Keywords: docker, kubernetes, microservices, golang, traefik

Id: WZ1CzvoUmik

Length: 61min 31sec (3691 seconds)

Published: Fri Aug 28 2020