Docker Networking in Production at Visa

so thank you so much for joining us we have a special guest with us today of course you've all used visa before you may have used their card or paid them before very happy to have them here so Thank You Sachi for coming in to tell us your story so specifically that we're going to talk about networking today and everybody has a different relationship with networking it's love/hate and it's complicated your systems people don't want to admit that the network exists and the network folk want to count every single packet and this is also the use case track so we're not going to tell you that docker is going to solve all your problems we're going to talk about the visa story and some of the challenges they had and some of the unanticipated problems and resulting architecture that they ended up with so before I start though we'll do a little story a history of in a bridge history of networking and yeah it's a networking fun I've got some more so what networking is so challenging because it's distributed in nature we have to manage and touch all these unique components that we talk to differently that we configure differently and don't talk to each other very well either so who can provision a new VLAN or virtual network pretty much instantaneously by a show of hands all right we got like seven people you guys are in the cloud we get it so who can do it for a new service maybe within a week what about a month two months we've got a lot of people don't have their hands raised I'm scared to ask how long it's going to be that's okay that is the challenge of networking and it's tough being a network engineer because sometimes these things are very very difficult to manage in a cluster see it's because we speak a different language and networking we speak in the language of subnets and IPS and Sider and that just doesn't work so well more deploying applications our goal is to deploy an application uniformly across the cluster while using all these discrete components so docker made everything better right containers came along and everything just works in the network well not quite because in a lot of ways it made things worse because now we have hundreds or thousands of containers per host so we have a higher density so we have more IPS and more MAC addresses in the network we've containers that exist for minutes or months and so containers coming up and down developers that are deploying 100 times a day or more just going nuts and we also have micro services that are more horizontally distributed adding more network traffic this is all challenging and this is worse so we're in a worse situation and we can't use the tools of yesterday and yesterday's networks to manage the containers today it's just not going to work and so back and docker won not seven which seems like an eternity ago we came out with a construct called the container network model and what this was is a contract it's a contract between a network driver that does the heavy lifting for the network and docker itself that's doing the scheduling for your container and there are two very important principles with the container network model one is put users and applications first and what this actually means is we want to have portable networks that are defined in terms of our applications not in terms of subnets and firewall rules and load balancing configurations and it has to be portable so that it can be used in your cloud on-prem bare-metal vmware and the second important thing was a plug-in API design and that's because your networks are all very unique they're all snowflakes right we have different constraints and different features that you have in your networks and for that we need different plug-in drivers that can accommodate that so there's three more things that are important one has to be scalable and secure it has to be scalable and we have to use algorithms that can allow for convergence in very quick times no matter how large the network is and also secure by default which is the docker mantra of securing everything by default so you actually use it support across OS ecosystems this is critically important because you want to use your network whether it's Linux or Windows or in the cloud or on Prem so you need to have a portable network that works across the operating systems lastly it has to be easy so you use it and so you can deploy it docker manages your search for you you can do we we manage the search on your your control plane across your data plane on your networks and then we're also managing the bring up of the control plane itself and everything that gets distributed across it so what's now possible so now we can describe everything about our application in a single file everything about the network and how the application uses the network for instance different tiers of applications and need different network policies or different features that they're using of the network like overlay encryption or the driver that they use in addition to that ingress and egress policies that are different for different applications all described in a single file and the result is that we have a multi container multi tier multi node application that can be brought up and brought down and what's docker doing it's managing your IP tables for you it's managing the load balancer for you it's managing all these pieces without you having to think about it anyways I'll move on and Sasha will come up and he can tell you how you actually use these features at visa in production next look all right so I don't have to give him intro but WETA so everywhere you want to be you guys know so um Flickr sound of metric alright let me give you a quick background about you know dark or how we brought in darker into Ouisa so we started looking into darker in 2015 and then after a year in late 2016 event live so this is not about a cold winter night application that takes care of all the visa payment transactions this is the application that does payment processing for variety of e-commerce companies globally between in production for the past six months we started with a 100 100 containers we tasked the ability you know scalability capacity up to 800 to 1000 containers to data centers trust us you know multiple clusters across two regions so you could right from the slide you could ask me question you know what took you guys so long it's been a year you started looking at it in 2015 you went live in 2016 right but looking at how we handle things you know the boss said you heard for me talking this morning that is how these guys set the bar really high I don't know if any of you know about we saw score system availability the core system has been running with zero downtime for the past two decades yes two decades not two years right that's kind of a level of expectation this guy said on certain applications not all applications right so with all that you know any application we look into we run into a lot of those challenges so I'm just highlighting a few of those challenges that we had with the critical ones talking about security Ouisa is obviously paranoid about security you know security is kind of a job number one for us we have very concrete requirements set for every single piece of software hard we have in our ecosystem so that means every single piece would need to meet those requirements before it goes live so when we started looking into darker it did not meet some of those requirements we already had in a sense it had other mitigating controls that we had to update we had to discuss and change things so that you know we can bring darker into our ecosystem the other thing is infrastructure you know the whole industry started with physical servers few decades ago and then the virtualization revolution came in everybody which alized we don't have all core systems on public cloud we have our own private cloud and the whole infrastructure is based on virtualized models so what this means is when we wanted to have physical servers for docker we had to really justify with the rationale behind the benefits of darker running in physical servers instead of running those in VMs I mean it doesn't make sense to run docker and VMs in production right you lose some of the benefits Dhaka offers so that that is again months of discussions you know justification and getting a buy-in from infrastructure team the variety variety of teams and we set to large corporate again the tools you know I'm sure everyone I've been working with the CIPD tools and how you automate things in your agile process right so there are so many such - you know every single stage option your sdlc you need to change things to accommodate your darker ecosystem and see how you can you know get things done the same we have done with the traditional way of application development right operational tools another dimension that brings the complexity and how you manage things so all these were a top challenges that we had which took us a year to really get to production right now based on all they see the other question you could ask like why did you get go for doctor what really made you guys to think about this you know we had our own goals just rewinding a little bit you know it's not that our applications we're not scalable not you know working well so we looking at euro year-over-year growth you know double-digit growth on the traffic that we get at or system and we also see you know the incremental changes that we have done for the past decade you know you don't really take it chances are opportunity to revamp or redesign the whole platform every time you have a business requirement you tend to do in criminal changes some of the applications becomes larger over the period then you get to a state yeah I need to do the refactor across the board right so that's how we are in a state we we can't go with this state for another decade so we need to take the opportunity to refactor to a very different architecture now right so we came up with these goals these are again highlights of the goals with many more line items with what we have so micro services mean not to mention that every single line item this goal is kind of a difference of opinion that you may have right I may have a definition for micro services which would differ from what you already have right likewise scalability and operational simplicity and all such things so we decided to go with micro services based on our definition and we decided to have a dynamic scalability you know our transaction traffic is like it's fluctuating unlike many of the other companies it's weighting in degrees seasonal you know different bursts out of marketing from different companies and all impacts our traffic patterns right so we want to have the ability how we can dynamically scale and we wanted to simplify all these operational tasks that we do when it comes to operational tasks you think about how we apply patches how we apply you know any of the changes to your application what happens today is like you know I would say for the past few years the malicious users or the hackers world is getting really sophisticated so what makes this to do in your environment it's like you need to have the ability to quickly push the changes the patches into your environment so you can't take time you need even to apply product or preventive measures you need that ability to you know what with a push of a button I want to have all these patches apply and Linux oils in a JVM whatever you call right with the current set up we did automate to an extent we do have you know scripts automated that pushes things but still you run into issues tool there is some time taken by all these to push any of the patches so we wanted to see the ability to you know push these things like you know the doctor benefits off you have everything in your image except the host yet there's another way to do that in the host - um so that's another simplicity we're looking at the last one load balancer let's architecture so this is what load balancer les doesn't mean that we do not want to have load balancer capability at all in a micro services architecture absolutely you need to have the load balancing capability but what we don't want to have is the complexity of managing the load balancing configurations like you may have a device and you may have a proxy that has all those load balancing configurations to the instances of your services right because anytime you say I'm in micro services are you even a typical SOA right you will have a service cluster on the service cluster is load balanced by a load balancer you will have a web that you know gives you the load balancing capability so to eliminate all those complexities we wanted to go to that you can see the right side of the diagram that eliminates those complexity logically right so that means the application which ever wants to talk to the other application within your ecosystem should know which instance it needs to talk to at least transferring to that application right that's our goal so to achieve this we decided yes darker fits in well we have analyzed how darker simplifies every single one of them besides some of the application changes we had to do to fit into this model right so based on the first gen darker container networking we decided to go with certain tools to achieve what we wanted so if we look at this yes darker gives you the ability to schedule containers now without a load balancer routing your traffic through another software or proxy software like ketchup proxy engine X whatever it is how would you get this done right so we decided to go with console registrate err this is pretty much you know standard practices that people use today so register helps us to register the services and service discovery is done using console dns agent and we didn't go for a console template and hecha proxy kind of a model what we went with is like ability to use the console DNS interface to get the load-balanced IP whenever we wanted to make a call from application 1 to application - we'll talk about more detail Linux upcoming slides basically this is the choice we made to get this done the another important item I wanted to note is like we started using darker bridge driver not the overlay when we started we had the darker one dot one that already had the overlay but we made a choice to go with bridge because of the maturity of the bridge trailer okay so this slide this picture represents or setup just in regards to how this you know registration and the flow happens so if you look at this you can see every darker host has counsel agent and registrated that does the registration and does the gives us the ability to discover services right so council server you also notice another component there custom lifecycle management so we built our custom code to do some level of lifecycle management so we'll talk about a little more in detail in upcoming slides and what doctor offers what we built it so as you can see whenever a new service is created your application container comes up registered or listens to you the doctor events and registered that service into console aging console agent syncs it up with server server you know syncs it up across all other agents ok so once your service is registered how does the discovery happen right so as you can see see the the color for a little pale I don't know if you can see those pale blue lines number one or number two so what happens here after registration when application one wants to talk to application to the application one there are multiple ways you can integrate with counsel DNS service their DNS kind of a plugin or API that you can use so we chose to use the DNS API from our application itself we built the common framework so that Curry's DNS agent from console hey give me the next available application to instance so then it gets the the next available instance it knows the exact port and IP of that service then calls the service so it's pretty much the setup in lights nothing new from many of the folks will be building with first gen darker even some of those with the second gender darker so we see certain problem with this first gen architecture if you look at this console register there are really good tools proven tools that there's nothing wrong with that but there is some complexity that we have introduced in our platform if you look at that you know we needed to manage multiple components yes darker manages those instances but when it comes to we need to manage those control agent and register ourself in every single host you have to have this is even before the global container came into picture right so you have to manage these containers explicitly in every single host you need to make sure these are healthy and available so that your application to application communication happen within your ecosystem now beyond that what if that goes down so when it comes to a four nine five nine available system you need to make sure this consul agent register is not really going down you need to keep them healthy you need to make sure even if it goes down bring it back up so we need to build some custom glue code listening to their health of the system pinging here is it there because if registrate er goes down if any of new container scheduled in that host it's not going to be visible right so likewise if your console agent goes down if the application one here one of the hosts wants to talk to the other application it would be able to find out I mean we do have some caching mechanism to mitigate that so you need to bring all this into picture to get this high availability there so those are kind of a problems that we are seeing with this architecture all right so let's look into the next capability the darker bringing mark is going to describe about the second gen so what Sasha described is um is not is not unique because the console and registrated deployment model is something that people have been doing for a long time because it works well it does its job of doing service registration and service discovery for a dynamically scheduled application so it does this well except the thing is that service registration and service discovery are fundamental to any application especially micro-services application when there are cross hosts but also when they're in just a single host as developers are developing multi container applications that still require these features and so the docker network engineering team worked really hard to integrate these features directly into the docker engine so that they're portable and there weren't any docker engine exists it makes a lot of sense for these features to be there wherever you're using the docker engine and so in this picture here I have a single host with the docker engine running on it and every single docker engine there is a DNS server running and this is by default turned on that runs whenever you run docker there's no extra configuration from your side to do this and inside every container there's a DNS listener that will forward request the DNS server in the docker engine and so when you schedule applications or when you schedule containers on a docker engine we instantly propagate the DNS server with with the identities of these containers so they can find each other so that app one can resolve the name of app two and receive its IP address and then also for DNS requests outside to your whatever external DNS service that you have running and this is great even for for just a single docker engine this works for you now this also extends nicely across to or to orchestrated and clustered environments for instance in a multi host scenario if you have a docker engines clustered together in a swarm we automatically create a network control plane for you and this is again is done without any intervention from you and so when this is created you may create services on top of your swarm and every service container get DNS entry of course corresponding to the IP address that it has but furthermore and what's really important critical to know is that every service also gets a virtual IP and this virtual IP is load-balanced internally by docker in the host and so without any extra configuration of VIP exists for every single service that load balances between services that you have in your swarm to take that one step further in docker 112 we introduced the health check and so now you can configure a health check for your applications some URL or anything that you're using to check the status of your application inside the container and a doctor engine will by default check the health and then send that back to the cluster and so the cluster is aware of the health of any of your containers so that if a service goes down well then the cluster is aware of it it can remove it from the docker load balancer and then so that service isn't contact anymore and then the service will be rescheduled depending on whatever scheduling whatever scheduling policy you set for it and so this is really powerful for because we're managing the entire application lifecycle just with a single engine or across a swarm cluster without you doing any additional configuration and so this is good for applications that might die and/or freeze up or host that die we're putting hosts into maintenance mode to get containers off of them the life cycle through all these steps is managed by docker without anything from your side and now Sachi is going to talk about how they use some of these pieces at visa thanks mark alright so we didn't use you know doctor because it's a shiny new object we knew what we are getting into so it's a evolution we need to keep working with the team we need to get to a state where it can be solid it can be you know highly available and meets all our requirements so looking at the oh man I'm having trouble with this okay so this is the choice we made earlier with first gen right so now looking at the capabilities the darker described you know we have options to see how we can take advantage of those newer capabilities built into the darker itself not necessarily using other tools out in the market right so scheduling yaks UCB takes care of it in service registration is built into your darker engine itself that's because in the new version of darker you have the ability to define your service and you have the ability to define you know load-balancing characteristics like its default you have the web defined as part of your service definition and the discovery is also provided by your daughter engine DNS or service itself right and the other important thing I wanted to note here it is on the overlay Network these are built on the rails of Orly right with - we also decided to switch to overlay to take advantage of all these capabilities and the other information that I mean the other advantages that Orly brings so if you look at all these load balancing capabilities you know it's logically that's nothing new it's pretty much same software-defined load balancing capability that you would get out of any of those tools right but what this gives you is like you don't really have to manage council agent your own how to manage registrate or yourself right you don't you're free of managing such tools you're relying on the same piece of software that is managing your instance isn't that really good it's a good option right so we decided to go with that so it takes care of keeping that you know highly available so because if your engine is down although yeah there's a capability or container can still up which means your core system that takes care of your application containers or is also providing you all these you know load balancing capabilities and service discovery you you know so so with the service definition that's built in registry you define a name for your web which is what your application is going to use to discover the other service within your ecosystem I mean obviously there are multiple other options we're just going to focus on this problematic area right so there's variety of ways you can configure like exposing port you know exposing external services all the you know ingress egress network that mark talked about specific to this when the application wants to talk to application to yes all you uses but look at this picture now it becomes nicer you know what yeah less colorful not much of red in there red orange but still the component you manage becomes much simpler you worry about one of your business applications not a infrastructure there is no load balancer here to write there is a capability exists but there's no explicit load balancer that you need to worry about so what happens here is like your the configuration is transferring to your applications you don't really worry about it get me the next available IP calling console agent within console too you can really define those integrate console like a transparent dns agent right but we went with a different mechanism for a reason so we also do some custom lifecycle management in the earlier version because you know you need to manage your instances what if something goes down let's your application goes down for whatever reason right what happens is you need to know yes my application went down the console we we implemented using console watch you know we get the trigger yes the application is done then we spin up another instance right so with this newer architecture what darker gives you is the built-in capability to auto recover your service so what this means is if your application - for example is down for whatever reason darker takes care of bringing your application up either in the same instance same worker node or another worker node so in your definition you might have already said I wanted to have these like ten instances of the service of this container for example here for application - I'd say two replicas so now all darker notes is like any time it sees lower than the set number it's going to bring up that many number of instances again based on health checks standard practice in the load balancing industry right capability so that's another cool feature that took away some of your you know our own you know custom implementations all right so talking about this connectivity if you look at the darker bridge driver you know you every time I mean obviously you can set fixed ports but you would quickly run out of ports so the better option is to let the doctor decide the random ports and then you use the ports so that's why we needed to go for you service registry and registering the port you all those things and overlay simplifies that a lot I'm sure you guys have gone to some of the black belt sessions that the document working team had had yesterday I guess Madhu it was so it's pretty much you know taking advantage of that capability here so overlay simplifies a lot you just have you configure your old and it work you know or your application to have the fixed port that is mapped to the actual physical port I don't really have to go into the details of it you guys know pretty much all that so the end is like we discuss you know the advantage we get is like fuel components to manage that simplifies our operational capability that also gives us the you know high availability goals you know meets it's easier to troubleshoot if you look at the earlier first gen architecture you know that anything goes wrong in the communication from application one to application you need to figure out whether that really fail did the consolation fail register to fail to come up to register my service what happened where right so that troubleshooting issues are going away with this I mean obviously you may argue yes the same number of components exist here but your troubleshooting mechanism becomes really consistent with the same set of tools produced by same set of folks right it gives you better visibility you know where to look into not like three different tools and again some of the custom glue code goes away right so you don't really have to build those custom components you know you can focus more on your business with that you also have additional integrated capabilities we had Whipple load balancing that we talked about you know that's something that you know backbone for all of these discussions we talked to we had here that's pretty much it so there's um there's lots of resources for docker Thank You Sachi there's lots of resources for docker so I would implore you guys to check some of them out docker labs is one great resource on github as well as success.com which has some excellent Prinze architectures for networking storage security CI pipeline go there and check it out and then lastly don't forget to vote for the session if you liked it if you didn't like it don't vote I've got some side bets going and I really really need this so please go ahead and vote I hope you guys learn something and we'll open it up for questions Thank You Sachi thank you there's a mic over here by the way hello could you describe your underlay network and both within the data center and also the way in links in between data centers so we have you know well-defined subnet for each region if that's what you're asking for like you know it's a physical network but virtual you know we lands based on our own definitions okay again can give you more details it's we specific questions layer 2 layer 2 the data center and then MPLS in between them or but I'm just curious about any details about the actual infrastructure part of the network there's probably a lot of know how much I can discuss about it so I'm kind of imitating to say it's a pretty typical multi data center set up and within our data center we do have you know completely isolated networks for application groups and also platform as well we call it as a part of secure parts with the different firewalls around it if you really need more depth of information here you need to come to pub chassis sorry hey I'm sorry a question so those second gen network architecture features Sachi you mentioned in his flies how much of that is exclusively only an Enterprise Edition and how much of that is available out of the box in the community source yes so actually a hundred percent of that is part of swarm so you could take da course EE right now and use it so as much as I love the simplicity of getting rid of console and things like that for service discovery how are you doing service discovery for things that are not running in swarm so if something in swarm needs to talk to something outside of swarm how are you finding it so for services outside of darker ecosystem we already have in the typical SOS set up we do have load balancers in front of it so all you need is that DNS for that service right we're not until we move them into the dock or ecosystem you're going to use the typical communication channels so your services in swarm just have a hard-coded load balancer to the service they need to talk to or what are they finding that other thing so we use console for configuration properties console server is not going away it does play a very important role in our ecosystem right this the capability that we are taking out of console like the service discovery in service registration is something that we do not want to use for this dock or ecosystem we would use rather the Dockers built-in capability but the consoles configuration management is still there if we what we do is like that web name we talked about is configured with a key value in console so when our service needs to call from application to application - it gets that whip from console property got it thanks I was wondering if you could compare the performance of the bridge network versus the overlay network that you saw so it's a virtual mapping I would probably get more details if you're looking for a performance difference we haven't measured the performance yet but we don't see that much of a performance the second gen is not in production yet so we're working on doing this performance testing and security testing on top of all you doctors done some performance testing and with both bridge and overlay we're rewriting the packet and we even with overlay there's some load balancing going on and so both are going to be obviously slower than using host mode or Mac VLAN but overlay performance is pretty it's pretty on par with bridge but a little bit slower like 1% slower or like 10% slower or like 50 percent slower I I think it was within 50 percent range than Ridge as far as latency is concerned so from the discus we had the same question so from the discussions we had the kind of the differences you would see you know microseconds when it comes to your internet traffic how much of a compromise that you can make for the flexibility is something that you need to make a decision on thank you what are you guys using for DNS and also related question I looking into NFV products to run in darker at some point in the future where you are with that but for DNS just for just for DNS service you do have your own DNS so we have our own DNS servers in our data centers hardware yeah so ours is we're not running in public cloud we have our own private cloud and we have complete set up and just integrate to whatever is there I got it oh thanks okay we'll take two more questions and then we'll all take all the questions off the side right hey does this swarm load balancer support the stickiness so I said again Israel any stickiness capability in this swarm load balancer Oh stickiness yes so in the the version that we're using here it's all its round-robin although there's a layer seven load balancer that's part of the Enterprise Edition that does include stickiness all right thanks yep that might have probably answered my question but my question was how do you route traffic to application is already running in swamp from an external load balancer so so one method is using layer seven load balancer that you include and then you set DNS name for it or URL for it and you can set that in the configuration for compose or in this part of the Enterprise Edition yeah okay cool well thank you so much [Applause] [Music]

Channel: Docker

Views: 12,417

Rating: 4.8550725 out of 5

Keywords: docker, containers, visa, docker datacenter, Docker enterprise edition, cloud computing, devops, microservices, Docker networking, Use Case

Id: k3SeQPt0f0o

Length: 42min 33sec (2553 seconds)

Published: Thu Apr 27 2017