Finally! An Open Solution to Solve for Persistent Storage

brief introduction I'm Clint my background is is operations I worked in the casino gaming industry for a bit open source before that and most recently I've been working for Dell and EMC working with a lot of storage automation and infrastructure and virtualization for the past 10 years or so so I have a huge background in figuring out kind of new emerging markets and how storage plays a role in those markets I'm also the technical director for the code team the code team is open source initiative at Dell technologies we focus on building community around software based infrastructure which is you know exactly what docker is fundamentally you know you find us in the community contributing to open source projects you find us building our own projects where we see there's some material impact that we can have that's different from other things that may exist you also see us engaging the community so here we are at the conference engaging you all you also see us in Twitter in slack we have about a 3,000 person code community slack group which is all focused around topics containers DevOps and all this good stuff that's going on you also see that we have a you'll see that we had a catalyst program which is a lot of the industry thought leaders in this space where we promote their activities and so in all it also SSE's we're trying to contribute and and grow the community around software based infrastructure the last point is that you know when we engage the community and we work on projects we tend to we definitely think that a rising tide raises all boats right so you're not going to see us go out and create a duration directly for our technology we try to engage a community and create something that moves the community forward as a whole and that's really what I'm here to talk to you about today some of the initiatives they're going on in the cloud native space specifically to storage another interesting for us is that we are a platinum sponsor for the cloud native computing foundation so this is a nonprofit organization that's all about sustaining and and making the environment that containers can it can grow in and be used in more available and more consumer-friendly for a lot of enterprises that are out there so we play a large role in bringing our technical solution leadership to the Table four that organization so let's do it so cloud native and persistence like what's going on here I think a lot of you have previously associated cloud native to things like containers continuous integration and other types of patterns or ways of doing things that you want to build embrace as an organization but in in this case you know what are we really thinking about we're thinking about making better applications we're thinking about dynamically managing applications so that they can be smarter and that they can take advantage of the benefits that clouds can provide so that's really what we're focused on as we look at the applications right there's many types of persistence that these applications tend to need so we're talking about traditional applications we're also talking about modern applications here in the modern applications it's more likely that they've been architected and built to split out their data services to different types of you know focused services so you may have log streaming you may have time series data you're going to have these purpose-built services that provide the persistence needed for those modern applications in addition like below the containers we still see a lot of your typical file storage and block storage right so it's very important in these environments to not only like satisfy the modern applications and they do certain things in new ways but also the traditional MRR applications that still use file and block storage so that's what we kind of define as the the storage area when it comes to cloud native now in this space what's really important is that we want to be able to build these docker container images they essentially are portable so we want to have a docker image that I have in dev ships out to testing out to production to do that we want to be able to shoot we want customers bill to choose the right container Orchestrator for the job you know that may be doctor may says kubernetes right they're all going to have different qualities and capabilities we want to leave a choice to the customers in this space and then we also want to have the you know to enable that you know what we're kind of defining as a universal network and storage interface and that's you know one of the basics of and one things you should take away from this talk is in the industry there's a collaboration going on between the Container orchestrators so docker maysa kubernetes Cloud Foundry to generate this container standard for storage to make it better and more interoperable with all the platforms you may want to consume from so this layer is very important for portability and it's very much emerging and it's something that some of our projects are directly going to be working with so the idea we want in cloud native I want to be able to have applications running in public cloud consuming data services to support my applications I want to also have my own cloud in my own data center which is able to consume those services in a very common way so essentially a portability of the container image and you also have portability to build leverage the data services that make that container container happy that's what we're driving towards in terms of the design pattern right in cloud and cloud native you got to think about the advantages and also the disadvantages essentially like what is it that cloud is doing that I have to prepare for storage wise or for my environment this kind of depicts that and this is very very important and this is what makes storage critical in cloud native environments in cloud right on the right side you've got an area which we want to perceive as where we plan for failures right and something that's temporary or ephemeral and that includes container orchestrators engines runtimes containers themselves and the applications right if we want to be able to leverage a cloud and say this stuff's going to fail I don't need it I can spin it up somewhere else but under the covers right to make this all work your data has to be stored somewhere so what we're really accomplishing with cloud native storage is that we've got container storage Orchestrator which I'll tell you guys about which is responsible for connecting the dot between the storage resources available in a cloud and the container that needs it right and this is much different than where we were with virtualization virtualization the storage and the container and the VM and we're all intermingled and here in cloud data we got a plan for failure so separate the two and rely on storage resources that a cloud provides and that may be in your own data center as your own cloud or maybe a public cloud like AWS GCE etc so how does this work today we've got a little project called R x-ray you guys may have seen it in the the main demo area Kenny was given a talk on that a minute ago there's some details there were swarm I'm gonna go through some pretty basics for what this is about but here's the logo for it that is little Rex he's been around for about two years now and actually this may be Tex Rex we've kind of redefined this one he's been her up about two years and he is a managed plugin in the docker store that's providing the storage orchestration to connect your containers to the cloud resources so here's how it works a request is going to come in from a container works trader or runtime it's going to say hey I need a volume Rex ray is a managed plugin that's long-running that is on each of the container engines he's going to get that request and he's going to go work with a cloud provider he's going to get that volume that where the data exists and he's going to orchestrate it out to the container when that container is done it stops then Rex Ray's job is to detach and give that ball you back to the cloud resource right that process plays on over and over but that's his main job is he's going to orchestrate resources and he's also going to handle the lifecycle of the storage the lifecycle is can I create remove snapshot you know clone etc like that's his job he's a container storage Orchestrator he interoperates with upstream platforms so today we said up yesterday we have press release rex ray is in the docker store as a managed plugin and a docker certified manage plugin he's got five or six different plugins today on docker source you look at the amount that are in there I think there's 14 or so Rex rates about 50% of the docker store and I'll tell you which platforms those are we're getting more and more every day but here's what happens under the covers is with docker typically you do docker run that's going to launch your your user space containers with the manage plugins what happens you do a docker plugin install that's going to launch the manege plug-in from x-ray which is essentially invisible to your doctor PS commands but that's going to kick off using run see under the covers there are three different types of plugins the doctor has in the store today it's the storage networking authorization this is one of the key plugins here's what the actual commands look like so very very easy if you need to actually use persistent services and this is common across all the platforms you do a docker plug-in install that's going to do all the downloading of the plug-in and it's going to start running it in the run C container once that's done you can see that a plugin is listed locally and you can start using it so you see a docker run command re specified the volume drivers are x-ray EBS and then you specify text as the volume name from the EBS storage system so that's all you need plug-in installed downloaded ready to go and the orchestration in the background is done to get that volume to the container that's running Rex ray has compatibility and interoperability with a ton of different storage platforms more and more emerging everyday I'll tell you why but here's the ones that we support first of all it's for any type of source platform block file objects anything that we can really put a file system on top of and anything that can have LBA or theological block addressing those are all Rex rate compatible the the ones in bold are the ones that are in the docker Store today you'll see we have all the abuses storage whether that's block Nazz object we have all Dell EMC storage so that's scale io Isilon ECS these are our software based platforms mainly and then you have Google compute engine which also has block in object and then you have just generally s3 so there's six plugins that exist in the store today the other ones listed are available through R x-ray and a typical running scenario where it's not a plug-in or it's not a managed plug-in it's running as a service locally on the system and you start it install using what we call a curl bash method so still very easy but that's the way you did install some of those other ones all right so why should I care about a container storage Orchestrator like if you go to the doctor plugin store you see other plugins that exist from different providers like what makes us different why should I care it's all about the users all right like if you're going to go consume persistent storage and you're going to use a plug-in you know what's better that you have a plug-in which has been tested that has you know eight to ten different integrations Plus that has a mature CI CD system that has a focus and intent on the on what it's doing or that you have a company who's going to build a plug-in specific you know for a one-off I think it's much better to rely on you know the mature Orchestrator then to have like a one-off plugin the other things that you think about are you know under x-ray it's got some other value out of features so it has a centralized in standalone mode what that means is that if you have an environment with a thousand nodes you probably don't want a thousand docker runtimes alt or orchestrators all talking to sf3 or EBS or anything like that like you're probably not overload the API hit the limits so you really need like a centralization right so you need the doctor engine to actually talk to something central and then that to be able to do rate limiting credential management etc before it talks to a back-end storage platform right so records got that inherent in its architecture you also probably if you're doing that Pinilla I think about encryption like that's when it starts to get more and more complicated you know once you get R X rated there your scales or plugins at bigger scales gotta think about some of these other things he's also you know has interoperability against other container works traders so docker is the first and the main one that it's focused on kubernetes we have a flex volume interface for x-ray so any of the any of the storage platforms that I just talked about they're all kubernetes compatible along with mesa the other thing is that it's a tool right so it's not just a plugin if you decided you wanted to perform some type of orchestration operations and you said hey I want to use your x-ray for some unplanned you know thing that we're doing an organization you know he is an Orchestrator that has its own seal CLI tool that you can interface with so very very powerful for any type of use case that you guys would want to have now you know in terms of the the user experience I talked a little bit about this a minute ago but this is like in terms of generating a plug-in right we want lots of them in the curse or write we want to have as many platforms as possible I think that's great for customers in cloud native and customers of doc but this was the reality of what it takes to build a plug-in right then you need an organization to focus right and you need them to be able to create a quality user experience and these are all the things they'd have to think about so if you go to the approach of creating a plugin all on your own you've got to handle all the stuff on the slide if you go their approach of using R x-ray and just contributing the storage driver interface semantics then you don't have to worry about the stuff that's in brown right you just do the green parts and that's the stuff that's closest to the storage platforms so the user experience for people that general organizations decide to contribute or x-ray plug-in is going to be much much better for their customers and this is really what this ends up looking like you know you satisfy the core capabilities as a storage platform that creates the storage driver the the abstraction is what we're studied in the term in industry as a container storage interface so one of the benefits of also being in Rexxar is that we're going to follow and interoperate with any of the standards that emerge that the container platforms are adopting so as docker continues a volume API or they start thinking about the container storage interface you know we'll follow that standard same with all the other container platforms so the drivers end up getting packaged with rex array which is the tool and then we create the manage plugin from there so that whole workflow build process is all done automatically for the people who contribute drivers so very simple process and to do that this is what it looks like I do a simple git clone go in the rectory directory specify a driver and this would be like if I'm creating my own plugin right I'm going to contribute one specify a driver I do a make build docker command all of a sudden in my on my docker engine I have a volume plug-in that's brand-new from the code that I contributed that's available and you can see that there's a couple files that are created that really support that process you know as a contributor of a driver I got to create of course the bindings and what I called out in green on the previous slide but you'll also create a docker file you're going to need to create the plug-in and then a config file that supports some of the the plug-in configurable options so very simple thing that you know it's going to greatly enhance the user experience that was it so quick little lightning talk on what Rexxar is I think it's going to be you know it's a very important tool today when it comes to persistence and cloud native very much an emerging space so expect to see more and more out of it in the docker Store I got a few minutes for questions you guys have any questions about it yep so port works is a I would say a we let's separate out data plane and control plane port works is providing a data plane which is a storage platform and then they're also providing the driver interfaces like a managed plug-in in docker this is purely an orchestration tool so it's a control plane orchestration tool at some point if port works decides they could always contribute a driver and then they would as well offload you know the burden of the plug-in development to the rectory project does that make sense so the question was sorry I should have repeated that ahead of time how does this differ from port works right so this is a control plane Orchestrator it really has nothing to do with the data plane itself and port works as a control plane and data plane storage platform we're questions the so the end goal that we see right is that in this space we want to expose as much storage functionality and volume functionality up to the applications as possible and so that means making volume functionality storage functionality like a first-class citizen inside the platforms and if we can do that successfully right we can start to create you know a lot of what we'd call operator automation which is focused on these applications so in the case of Postgres right in the future I would have something that's really automated for Postgres so it's able to you know leverage snapshot features available from the cloud and he can actually create you know replicas for Postgres in an integrated fashion alright so there's a lot of operator knowledge around you know managing persistent applications and we can actually you know contain all that stuff programmatically and we can leverage the cloud features and services by way of these interfaces in the future so today you know people have basic volume functionality tomorrow we're looking at Rob lock devices which means the no file systems just the raw blocks and certain applications want that kind of stuff we're looking at more features like like snapshots and cloning so it's emerging it's moving forward or questions it's got a CLI tool right so a docker volume plug-in I'm sorry yeah thank you so why is it why are we calling it a container storage Orchestrator why isn't it just a doctor volume plugin that was the question thank you it's a CLI tool right so first of all Rex ray you do a curl bash and you have a binary which is written and go that you can run and manipulate and use in any way that you want to so it's all about how you interface with it for one because it's a CLI tool tool with the docker endpoint it's a flex volume interface three different interfaces today and then it's also got an abstraction to handle multiple platforms so it's not just a plug-in that handles one platform it's built to handle whatever storage platform there's a driver for alright so it's really the front end and back inside of it that makes it a tool versus just one interface for one platform so the question is can it work with the vmware vsphere plugin yes does it today now we don't have that plug-in today but I think as organisations who are building or thinking about the plugins that they want to create for the docker soar like that slide where I had here's the things you should focus on and here's the things that you probably don't want to have to focus on companies like VMware and that's of course close to to Dell technologies you know we're going to work with them and figure out how to get a driver in there because I think it's beneficial for for the customers and we're hearing that feedback that it's the right thing to do - so the you know recs are a and live storage already for it and we're you know eagerly looking for people to come join us on making it more functional yeah yeah yeah and we're always looking for contributors so it's actually pretty easy it's a matter of you know the testing for it to make sure that it's working as expected but Asher is the question of sorry the question is is as your support coming and the answer is yes it is a driver today for unmanaged discs we got a question about managed discs actually yesterday I think but the plug-in will come soon after yep any more questions yes so the question is can I run Rex ray and have it work against you know docker swarm or you know GCE and cloud platforms all and and the answer is is yes and you can do this in a couple ways there's two operational modes of rex ray one is a standalone mode which means that you have a a rex or a stand-alone instance that includes the storage driver which gets deployed to every single engine in the docker world and then that controller or X ray instance would be configured for whatever storage platform you want to use the other alternative though is if you use the centralized controller mode of Rex ray you would deploy their x-ray client or agent which is a lightweight Rex array with no storage driver that would go to all the nodes and then you'd have a centralized controller which you then configure and you say he's going to handle you know GCE with these parameters he's going to handle another platform with these parameters and it's all about the client accessing the service that that controller x-ray is advertising so yes you can intermingle services platforms like it's all going to work that's what it's built for the question we're working on the the manage plugin for that so the question was under the centralized deployment the controller method are there manage plug-ins for that today it's expected that you're going to use the the typical install method which is just deploying the binary and starting it as an init service or systemd service but we are working on where you can have the the manage plug-in be a rest right agent and another manage clip plug-in being our x-ray controller and the configuration or environment variables kind of determine the configuration of the point so for now like a controller mode is not manage plug-in but it will be yeah like last questions is that okay last question so the question is other than the orchestration you know when I have a large environment is we actually doing anything else for me so other than attaching detaching mounting unmounting so there's two sides of rex rates functionality one is the orchestration which is what you've covered there the other is of volume lifecycle operations and the volume lifecycle operations would be can I create remove volumes can i snapshot can I clone stuff like that you know the create remove are the additional lifecycle features that Rex rate does today okay that's it you guys have any more questions feel free to see me thank you everyone my name is Chet Thibodeau I'm a product manager from Veritas technologies and I'm going to talk about predictable performance and data protection for docker container volumes so I know this is 20 minutes I definitely want to get through this so we have time for questions at the end so let me start with many of you may be familiar with Veritas many may not but just a kind of level set so Veritas the company actually developed one of few file systems and I just had to double check and confirm for myself so back in 1991 so we go back that far in terms of our storage experience we recently split from Symantec and as part of that activity of splitting now we have a very singular focus around data management so there was again a time when there was security involved in that with the Symantec play but now we are singularly focused on data management so basically as you kind of look at this what we're looking to do is through kind of these three main areas provide a 360 data management solution for customers now we're not going to talk about all these of course but this is just to give an idea of kind of the breadth of our portfolio so if you look at all the different products again you have things ranging from backup to disaster recovery to information visualization to storage and again obviously today the key is to talk about storage for containers so I put together just what I would say is a very high-level kind of evolution for container storage so you have starting from your left I guess my right that's F so that's Red Hat right and basically red hats been working with containers for well over a decade its docker that really brought containers to the forefront brought all the excitement around containers but nevertheless they've been around for quite a while then as you kind of progress hopefully that should be familiar that's kubernetes kubernetes now you know again being open source that they are definitely getting into more advanced storage services that they're looking to provide on storage persistence as an example for their pods as you progress again that is cluster HQ unfortunately now a legacy company since they folded last it was right around Christmas but what basically cluster HQ did is they provided this storage management layer that sat on top of other storage providers so you can run a heterogeneous storage container environment a more container storage environment and then as we go finally to the far right that is our logo so this is the Veritas hyper scale for containers which again I'm going to get into here in a bit but now what we're looking at doing is providing the enterprise class storage services and performance that customers not only expect that they really need to be able to deploy their applications in production so kind of taking maybe a little sidestep and again I don't I don't expect that this is very or should be new to many of you but if you kind of look at different workloads that are running in container environments right so whether it's databases that are going to need persistent storage whether its Web Services that are going to need to address management and scale online transaction processing right you see scale security and persistent storage etc and then like homegrown applications which probably need some or have some form of challenge of all of those what we are addressing with our hyper scale storage for containers is the persistent storage problem there are other ecosystem partners from docker that are going to be addressing some of those other areas but that is really our focus today is to solve that persistent storage problem for containers so now getting into it so I know there was a question from the previous session around how recs are a differed from port works for example and the gentleman discussed data playing compute plane so let me discuss our architecture again at a fairly high level during this quick lightning talk so we have the concept of a compute clan in the data plane and the idea is really with hyper scale as per its definition is to separate these two planes so that you can actually scale either one independent of the other so if you need more performance you scale the compute plane you need more capacity you can scale the data plane by having this two plane architecture what it allows is it allows the ability to define quality of service and go into that in a second but traditional quality of service where you can define you know some threshold of performance and then that actually is all staying with the application up in the compute plane so what we're trying to do is make sure that the application gets the highest performance storage that it can and then down in the data plane you actually do the other storage services so it could be snapshots it could be additional replication it could be eventually an integration with backup for example so so that's the concept I will maybe kind of preempt a question around how we differ from some of the other storage products out there I think again the idea of focusing both on quality of service as well as data protection is one thing that is unique with our solution and in addition to that again Veritas has been in the storage business for a long time so we have not only the pedigree but we really have the understanding of the problems that customers face with their storage I mean in supporting their applications so predictable performance via policy so I mentioned the idea of you can actually define a quality of service um so could be for AI ops it can be for latency it could be for both so you can define both a min and a max and again that's to address the often term noisy neighbor issue no different than what happens with virtual machines it also is configurable per volume per container volume so again by creating these policies you now could apply it to one container volume in your environment or you could spread it and apply it to multiple container volumes in your environment and then the thing that is the last bullet in italics there is so we are also looking at evaluating what the overall capability performance capability is of the node to ensure that you don't actually over provision the performance for those container volumes so basically making it a lot more Hardware aware if you will or performance aware and then just a quick screenshot of the user phase two just reinforced so again this is where you would define a storage policy or you could leverage an existing one and apply it again on a per container basis so then in terms of data protection and resiliency so again you have the ability to define a replication factor for both your compute plane and data plane and so this is again to provide that high availability so in case you have either a potential node failure or corruption and either playing you're able to instantly recover from that in addition to that you have the ability to configure your snapshot frequency and so and again I'll have a screen shot here in a second and then I would say probably the most important thing here is that last bullet where because thinking back to the data management that I showed you earlier Veritas has a broad portfolio of products and so the idea is we will be integrating these additional prom products of Veritas into that data plane and what's important about that is by doing that in the data plane the intent is to have no impact on the performance of the actual applications those container services that are running in that compute plane so all of this would happen down at that data plane layer and from my understanding and kind of being in this container journey for the last couple years there really isn't a good backup solution for containers and so this is again a great opportunity for us having been a backup provider for again many many years to actually fill that void and fill that gap and then again screenshot very similar to the one you just saw the differences it's just showing a gold policy where you have you know a different availability factor as well as a different snapshot frequency so just trying to kind of highlight that and then finally simplified management and API support so let me explain quickly the way that we actually developed hyper scale for containers was through a micro service architecture so we're kind of coining the phrase PowerPoint to product in a four month so we literally went through an agile process using micro service architecture develop this product and it is delivered as container images so there's basically I believe it's five container images overall that make up for the product there's a volume plugin and then there's different services for whether it's the i/o or device management etc but I think that's actually important and in addition to that we have full API support so we were very conscious of knowing how people actually consume and adopt these new technologies is you have to support api's and restful api for that matter and I think the other thing I just wanted to touch on is around the self provisioning so the idea is to make this is easy and user-friendly for the consumer as possible so if they want to use the docker Universal control plane to provision the container volumes they can do that they will then automatically show up in our UI or from what I showed and I guess I'll pull it up here if they wanted to create a volume container volume from our UI it will then get mapped and show up in the docker Universal control plane so again wanting to make sure that there's that full visibility for the user so that you know however they're most comfortable working with containers we support and then kind of just to close on this in terms of so we announced our beta version at this conference so I've actually attended the last four docker con at Wallet Veritas and myself as the PM super excited that we finally actually no our exhibiting at docker con we have a product again it's in beta we have the ability for people to be early adopter so kind of the marketing pitches you know please come by see a demo so we have demos so you can see you know what the app product actually does how it works do a survey you can win a GoPro get a t-shirt etc and I think with that I want to thank you guys and then I'll open it up for questions yes so the question yeah no good question sorry so the question was how is this different from hyper-converged so here's my take so this is chad opinion but so you have hyper scale you basically break apart right so you again have this ability to where you can scale compute and capacity independent of one another hyper-converged is really the opposite of that right the intent for hyper-converged is you're trying to bring everything together so that you have what claims to be a better overall user experience or easier deployment model so now you've combine compute and data in a single appliance right so again as we can say you know think Nutanix as an example for hyper-converged so again i would argue there potentially competing architectures but you know they are at kind of opposite ends of the spectrum that answer your question okay other questions ah great question so the question was what's the difference between volumes and just regular data in terms of backing up so first and foremost if you think of a micro service architected application it's kind of like a spider web or a nest of all of these different services that may have some dependency on one another right so in terms of backing up the challenges I think exacerbated because let's use the example of I want to back up some monolithic application maybe it's Oracle it could be you know some other application I basically have to back up that application I'm going to have to do certain things like you know takes state and everything else but you're not going to necessarily have all of these separate little dependencies and modules and so again I would argue that backing up of micro-service applications is exponentially more challenging than traditional backup so that make sense thank you any other questions I think it probably had just one or two minutes any other questions okay I think we're good thank you [Applause]

Channel: Docker

Views: 13,239

Rating: 4.47541 out of 5

Keywords: docker, containers, CLI, REX-Ray, Operations, interoperablility, interfaces, storage problems, Ecosystem

Id: AXWfdu9f8sU

Length: 38min 12sec (2292 seconds)

Published: Mon May 08 2017