Feature Friday Episode 66 - Object Storage Extension 2 1

hi and welcome to feature fridays my name is guy bartram director of product marketing in vmware and today i'm joined by galena slavova and your glue glenna why don't you just give yourself a quick introduction seeing this is your first time on feature friday so welcome and thank you thank you guys my name is galina svalva and i'm a cloud technical marketing writer at vmware today with my colleague york we'll present to you the new features of object storage extension 2.1 excellent and uh york just a quick intro for those who haven't seen you yep well yeah welcome i'm a technical product manager at vmware and first of all uh i guess a short apologies to everyone because we haven't had a feature fridays for a few weeks um it's been a very crazy time with vmworld and other events going on but we're going to get back on track now and try and get a feature fridays every friday because there has been quite a few key announcements around vmworld um vmware's have been you know pushing out a lot of new product updates and object storage extension is no exception um i think this came out at the end of september was that right galena 2.1 yes it rolled out on on september 24th and well and it's available to our partners to start using it and take advantage of all its new features yeah great so i mean it's a great update as well there's a lot in it and we'll get to that in just just a second um i think this is a good time because you know with vmworld people may have lost a lot of the announcements just before vmworld because there was so much that we announced this year really great stuff and great innovation so let's dive straight into it then um 2.1 um is the version we're talking about uh released in the end of september like lynn just said let's start talking through what's new in uh 2.1 gleaner uh the new the new features and and enhancements in object storage extension 2.1 are as shown on the screen are the following for a provider administrators there is a global bucket sync policy that can be executed on demand or scheduled and also provider administrators can can create storage cloud and storage policies uh that can be edited by uh tenant administrators later and uh for tenth administrators the highlight of this subject storage extension uh releases the kubernetes backup and restore feature with with this feature tenth administrators and tenant users can backup and restore kubernetes clusters another feature available to talent administrators is supporting a true roles that can be uh applied to talent users with that support and throws tenant users can do specific things on the apps catalogs and kubernetes external kubernetes clusters uh as i said uh uh in a second uh tenant administrators can can edit a default global global storage policy set by provider administrators and new specific storage policy for uh for the for the tenant organization and uh for tenant administrators they can also restore archived s3 objects in aws what is specific here that the s3 objects that are part of the glacier and glacier deep classes cannot be viewed in object storage uh extension in case they're they are they are restored and after that they move from archived objects objects to frequently accessed objects and their content can be viewed on object storage extension uh user interface and another uh the list is long and another feature available to them administrators is that they can directly import and export the apps from cloud director this uh this allows them to to to quickly export and import their important the apps without without without switching to object storage extension and another enhancement is that there is an s3 api endpoint available for s2 client access for tenant administrators the list goes goes on and we have we have more features available in object storage extension i will mention a few more we have cli support for for s3 cli support to configure object storage extension when there is when it is configured behind a network using a proxy server you need to uh to provide a few settings to to configure object storage extension so that it can it can connect uh in a network using a proxy server so um another feature that is available to the cloud providers is integration with donald with vmware analytics cloud this is related to to the vmware customer experience program and cloud providers can accept the the conditions of this program and they can with this they can they allow vmware to collect uh specific data related to the product usage and the vmware only collects data that will improve its uh product offerings and the the final uh feature that the the the final enhancement that is available in object storage extension 2.1 is the improved retrieval of object of a bucket object count now it took a lot of time in the previous releases the bucket obj account to be displayed on oc and now uh this object bucket object count is now much faster to be retrieved and with that i can i think i covered the the features and enhancements in object works 2.1 yeah there's a lot there so thank you very much for going through that um maybe let's look at this a little bit and look at some of the key ones here and talk about the kind of new value that they're bringing for the provider um let's start with and you know we're on on topic of bud uh buckets as you finished up there um let's drill into the global tenant bucket sync now what traditionally you may have tenants in multiple sites with multiple different buckets um previous to 2.1 was there a sync capability there and and did you have a single view of these of your tenant buckets or was this all distributed per um cloud director instance uh it was uh now we have global 10 bucket thing that uh that spans across sites and it's available to all tenants and it instantaneously syncs the the content of buckets in object storage extension it sinks the the content of the the buckets in object storage extension with the storage platform so now we have this capability that was not available previously and this this is sorry galena this is just syncing the the metadata view right this isn't actually syncing the real data uh yes it thinks uh it's uh it syncs the data between oc and the storage platform okay but that's not the actual data that would be on the object storage itself it's it's the metadata view of that data right yes yeah okay that's cool um and i guess the next kind of big one i'm looking at is the contain um k8 kubernetes cluster backup and restore um we haven't had that capability before and obviously with you know tanzania being introduced to vcd times with basic now included in flex core um there's a lot of capability there that providers can start to sell to their customers and backup and restore is obviously going to be an important aspect of it york do you know you know what what's the kind of capabilities of this um functionality in relation to uh the kubernetes clusters that would be running is this sort of a ui base api well it's uh actually both so it's ui based you can um select your kubernetes clusters um and define some uh backup policies for them and then agent will be installed a regular i think valero backup agent will be installed into the kubernetes cluster and use the buckets in object storage extension as an endpoint to store the data of the cluster configuration and the cluster data and the interesting part is that it's well it works very flexible with kubernetes clusters that are managed by container service extension in cse but you also can provide a generic cubeconfig configuration file and point it to any kubernetes cluster that runs well either in vcd or anywhere else so that doesn't really matter where the kubernetes cluster runs and that's something that can be it's uh in the self-service capability for the tenant so as a service provider you can use this really to offer automatically additional capabilities for your tenants to have a backup and restore for kubernetes cluster no matter if they are part of a container as a service or kubernetes as a service offering or if it's just something where as a provider you know that your tenants are running kubernetes clusters and offer them that as an additional um functionality that they can use yeah that's cool and then does it have the same sort of scheduling capability and also the backup functionality that you'd expect yeah yeah we will see that later in the demo that you have some pretty flexible um backup scheduling options okay excellent um and i guess the i'm just looking at the next thing here would be um and i found out quite interesting galena you were saying about the archive s3 objects in aws so there's this very very long term archive where data is you know not going to be accessed frequently and we can't actually view that right until we make a request to move into the frequently accessed bucket yes this is how this feature works uh you select such an object in the oc interface user interface and then you apply the restore uh option to it and after depth this object can be seen its content can be seen so this would be a really useful use case for something like medical records or police videos for example criminal videos you know where you're gonna have to keep those for quite a long time but you know you need to be able to instantly get them back if you're you're in court or you're in a surgery um that makes a great use case for those type of verticals um what is the uh v app export to object storage extension about york can you can explain what yeah so that's some uh functionality that we already had in object storage extension for quite some time now to use the buckets as a um just a destination for exporting and importing back the apps that are um otherwise running in your arc mdc's regular reaps and but in the past you had to go or you had to navigate to the object search extension user interface and then select your v app in like the drop down or the list for of all your v apps and now you can do that in the context of the vm itself so it's a improvement of the user experience that when you work in your org with dc and you manage your vms and v apps you can directly um trigger the export to object storage extension from there oh that's great because that's something that you know has been asked for a long time in various plugins that we have is integrating them better into the the vcd experience um so you don't have to then jump into the plug-in then refine the vr whatever yet yeah absolutely excellent i like some of that um and i guess the other the other thing i just want to quickly touch on is the s3 api endpoint and the cli support um is uh the s3 api endpoint i guess this is uh you know particularly important for developers who are going to be you know programmatically wanting to uh store or manipulate data um galena what what are some of the kind of use cases you see for that endpoint api with this endpoint api actually tenants can access the content of their s3 buckets and this is what they can do with that then manipulate them and apply some operations on them so they don't even need to go to the ui at all it's all available now in api and that i think that sort of belongs together with the [Music] increased granularity that we have for role-based access control and the application and subordinate roles that are now available so you can define finer grained access management to the object and the buckets and this functionality is also available through the api and that's why it's related to the s3 client access i mean object storage extension or to access the objects themselves to retrieve them to upload them that always has been possible through the api and if you hopefully you remember some older episodes demonstrated on this in the future fridays but now it's also well it uses the s3 like api capabilities to also work with the the role based access control and access rights and further features of the individual objects and buckets oh great so we've basically got a lot more control and capability to restrict or provide access to different api calls and functions that you can do on buckets awesome and lastly i just wanted to um touch on the vmware analytics cloud this is the kind of first time i i've seen this uh mentioned can you just give me an explanation what is the vmware analytics cloud well it's related to the collider program and we basically collect the data only if the if the partner agrees uh for the usage of the vmware products and this is only uh for for for uh for making decisions on the products that we offer to our our clients and the data is only related to the product so that the the providers are using we don't okay so so i think it's part of the the product usage how they are used it's part of the customer experience improvement program and it helps well our product managers for objectives extension object search extension to see for example which different endpoints are being used by our partners and to prioritize um yeah which features are heavily used which are not so heavily used and just help to prioritize the improvement of our um products in the future that makes a lot of sense okay yeah so um um and super collider in case anyone's wondering his um you know our internal database of all of the uh utilization um and it's important to recognize that no proprietary or configuration or actual data is sent through it's just simply the functions that are being called okay great um well listen that's uh i think that's enough on this slide should we uh dive into the demo galena i know you have uh um a video you want to walk through or for the functionality okay so just a second i need to switch to i'm sorry to the demo here i will start with with the global bucket synchronization policy that is available to uh to provider administrators here you as you can see uh we we showed you a bucket synchronization policy uh we need to provide the sync interval and the threshold for the uh for the s3 objects uh the after this when this threshold is uh is um is uh achieved so then the the objects that are over this threshold won't be uh won't be synced and we also provide here the scope of the of the bucket synchronization policy let me uh the tenant scope is uh basically about if we if we will apply this bucket synchronization to all tenants that are part of the local site or if we apply this uh policy to the tomb to the moot site tenants here i will uh present uh how a new cloud and storage policy can be created as a provider administrator and for this uh the storage the cloud and storage policies about replicating the s3 objects across the data centers uh that where the the claudia nodes reside and here and if you're not using cloudian galena if you're using like a another third-party s3 object storage this is only uh for claudium okay for clogging and it is not applicable to the other s3 platforms that can be used with object search extension okay basically what we see here is creating a cloudiness storage policy for the tenants it's a default storage policy which can be which can be edited by the tenants uh here we need to specify uh the data center where the the claudia the claudia nodes reside and uh we need also to to select how many replicas of the object storage of the of the s3 objects will be uh will be created and stored on on the nodes on the claudia notes and as you can see we the the replicas are are related to the number of uh of the claudia nodes so we need to create replicas in each uh each at least two replicas of uh of each object uh of each s3 object and uh the replicas will be uh spread across the nodes the claudia nodes in the in the data okay is that a cloudian resilience thing as well that's yes specifically yeah okay let's specific cloudiness and this what we do here is to specify the the distribute scheme is automatically presented to us and it is uh related to the to the number of data centers uh where the claudia nodes reside there are other distributed schema of schemes available and they will be uh shown if there are more data centers so where the where the cloud nodes reside yeah so that is something where claudian specific functionality gets like available or made available in the object search extension ui and that allows service provider partners who use claudium as a start endpoint to yeah just create different offerings service offerings with different availability or durable durability settings on their glaudian storage and publish them to the different tenants so it follows like the same paradigm as usual that we have in cloud director that you can offer services and functionality on a per tenant base and then of course monetize these features based on your yep your business model as a service provider partner here we can see how um how a v-app can be exported from cloud director to object storage extension we need first to power off uh the v app before we we export it to cloud director and after the the apps powered off we can proceed to the to the export of the app and that all can be triggered from the regular app management yes so you don't need to yes i'm guessing there'll be something in the actions menu here yes the actual procedure you select the the power of the app that you want to export to object storage extension and you then wait until it is exported to object storage extension that's taking a copy of all of the vmdks and all of the configuration everything within that vm yeah yeah that's a very powerful mechanism that allows you well because once these v-app uh v-apps are stored on object storage or on your in your object storage bucket you can very easily i'll share the links create access control policies and migrate them or move them yeah just access them via http which is much easier than sharing like v-apps across different vcd installations as a provider if you want to offer that like as a service or as a tenant share catalog items or v-apps between different environments perhaps download them to an on-prem environment and import them back to on-premise center for example so this allows you to just use the object storage buckets as a store for the v-apps for these procedures we will now continue with showing kubernetes backup and restore functionality and here we have uh kubernetes clusters that are either detected by object storage extension or have been uploaded to object storage extension here we have two types of of kubernetes clusters that are shown in object storage extension object storage extension automatically detects kubernetes clusters that are managed by container service extension and also object storage extension can backup and protects kubernetes clusters that are external and and have been uploaded to object object storage extension here we will present how how we will backup and restore an external kubernetes cluster it was previously uploaded to object storage extension and here we will demonstrate this uh backup and the backup process sorry kalina just pause for just uh when you say external versus native native is discovered via container service extension cse it's a grenade cluster which uh cube config yaml file has been uploaded to the object storage extension and the with with this object storage extension connects to that external kubernetes cluster okay that's pretty cool that's a great feature yeah absolutely because um i mean we um use the power or the functionality of api uh of kubernetes as an abstraction layer here and um it allows you to just import when you add that external cluster it allows you to import kubernetes configuration file from whatever kubernetes cluster and wherever that runs so as long as that kubernetes cluster has album network connection to the object storage endpoint of vcd this backup works which creates a lot of cool new business cases for service providers to um yeah offer backup for kubernetes clusters that your end customers might run on-prem for example or even in other clouds yeah exactly as long as there's a route to it i like it i will continue with the demonstration and we specify what will back up and when this backup schedule will be executed we specify here whether we want to uh to back up the whole kubernetes cluster or part a part part of it so here we will will select a few kubernetes namespaces that we will backup and we can also specify labels which can also help us to select which kubernetes cluster the objects we can backup now there's no i'm not familiar with the agent i guess deployed at this point but there's no interruption to cluster operations is that why that's being deployed oh no the backup agent is installed when we uh when we select it for protection it's it's uh it's invisible for the for the end user and it basically does the backup and restore and the mechanisms that we use here is called valero which is like a backup solution for built for kubernetes an open source project but uh sponsored by vmware by our tonsil team yes it believed us the backup and restore and we'll see what will happen after this backup backup policy is created after we selected the external kubernetes cluster for protection we now see that it has has a different label and it shows that it is uh that protection is in progress and we also also see the backup timeline details here when this backup schedule was was created so now we have after the the status of this kubernetes cluster changes it it will show that this in protection and the the backup has been uh triggered again that's a schedule yeah here i will uh demonstrate how how the restore process can be done here we will connect to that external kubernetes cluster and i will list what's inside it what are the namespaces part of it and then we will select and we'll see what what's part of this specific nginx example namespace we will now delete the the namespace tangent nginx example namespace and we will again execute the cube ctl get ns a comma full show uh the namespace is currently available part of the cluster sorry for this i uh and no problem it's catching it isn't it in time right yeah you can see there the yes i know njx is gone here so uh here we will see that this uh namespace was deleted and it's now missing from the kubernetes cluster that we that we previously protected that we have previously protected and i will now continue with the restore process restore the a backup of a kubernetes cluster you need to use the this restore button and you need to select the the backup that you want to restore here we have just one backup and but the providers can select which backup to restore and we need to to wait until this process completes again we see the label for the cluster that that has changed that it has changed and we see that the the processing progress it was during now it is restoring the the backup to the kubernetes cluster i'm assuming from the the backup menu here where you've got the action drop down you've got things to do with scheduling and other things to do with the agent right yeah it shows the the list of operations that has been that have been done on the on that particular cluster and here we will see if the the restore has been successful we'll again check the the namespaces that are part of this kubernetes clusters and i will see that their store was successful and the namespace was restart the deleted namespace was restored i will now show the the content of the of the of the s3 bucket that that was selected for the backup of the kubernetes cluster we will see what has been uh backed up from that particular cluster the backup actually saves the etcd keystore which which keeps information of the kubernetes configuration and statuses and it also captures backups the persistent volumes that are part of kubernetes here we'll uh here we display i'm sorry uh here we uh we show the the content of the restore uh folder part of this street bucket and it includes the files that are related to the restore process configuration and log files yeah that's a very again powerful mechanism because it really works independently of where the cluster the kubernetes cluster is running as long as there is network connection to that typically public-facing object search extension endpoint anyway you can have as a tenant administrator you can configure these backups and restores so as a service provider you can there's not a lot additional effort for you to just tell your customers who have object search extension available that they can use this for backing up any clusters be that kubernetes clusters that run within the vcd environment managed by container service extension or even run in any other environment or even on-prem environments which is a yeah very nice add-on or functionality to capture new workloads and offer additional services to your tenants and your this is um obviously we've talked quite a bit about claudium but this is also available for native aws s3 this yeah and it uses or as we mentioned it uses the open source valero project for backing up so even if we currently don't have any ui integration yet to [Music] restore the clusters to or restore namespaces for example to different clusters um as galena showed in the demo you still can access all the like the backed up data including the valero json file and download that and then um yeah if you want to restore it to some completely different environment uh you can play around with that as well yeah so yeah i want to mention if you look at the valero documentation be aware that the backup of persistent storage it is available right now but it's currently marked still as beta quality so have a look at the valero documentation if you want this feature or if you want to use this feature or again as a service provider you don't really have to do anything here if you tell your tenants to use this feature they should look at the valeria documentation to figure out details about the the other requirements how it works and what kind of information is in the backup bucket center restore yeah good cool okay another feature that we want to demonstrate today uh the new supporting trolls for tenant users uh as you can see there are three three different supporting controls that are available in this release we have contributor catalog contributor and kubernetes contributor the the different support and the trolls define what the talent users can do with the apps catalogues and kubernetes clusters here we demonstrate how how we will apply kubernetes contributor role to attend user and what this user will see after this role is assigned to him this is the kubernetes contributor ring yes it's the grenade contributor role that we assigned and as you can see the the user can only work with external kubernetes clusters and to work with with them the distance user needs to to add an external kubernetes cluster here the current user needs to upload the tube ctl the qtl yaml file to connect to that kubernetes cluster that he wants to to backup and this new this another feature that i want to demonstrate no it's not it's uh uh it's part of the the cloud and storage policy but here attendant uh administrators can edit the default storage policy that was applied to the organization they can even select a specific one that they want to apply so tenants and administrators can select a different storage policy for their for uh for their users for their customers and uh that's great you can really let the tenant loose with this now i mean there's no need particularly for a service provider to be stepping in and creating all these things for the tenant it's now uh pretty much self-serve and self-fulfill yeah that's the case so uh here this is the the last feature that i will present today this is the aws object restore as you can see here we have an object that is part of the glacier storage class which means that it is archived object and cannot be accessed on a whim it's uh it's uh it needs to be its steps needs to be changed from our car to uh to frequently accessed and this is what we do here change its text status and in order to to be able to be able to see its content so once you do that you're then basically going to go to the normal restore process and choose the name space or whatever it is you want to restore right yes and with that my demonstration [Music] has completed completely this last feature it's very important if you well it only applies of course if you use native aws s3 as a backend for example in a cloud director service environment but also as a service provider and with that you can define in the aws like policies to have some automatic archive functionality to keep or to have some steer tiered storage offering so that certain objects that haven't been accessed for a certain amount of day get moved from the rather expensive s3 storage to the glacier which is much much cheaper for um yeah as a data archive but you still can use the ui now to retrieve these buckets be aware that for all this functionality well we are using native aws s3 so all the fees apply for restoring and the data transfer that are documented in the glacier and aws services yeah that's a good point yeah so moving between one and the other may may impact your cost and also the selection of how fast you want to access that so during the restore you saw these three different options for mobile restore and yeah speed high speed restore they have different price tags yeah there you go expediated retrieval fast versus bulk yeah yeah it's interesting everything is chargeable right when you start looking at public cloud um that's great uh galena that's been really good i've got a really good grasp of the the new 2.1 functionality now i'm particularly pleased to see the the kubernetes backup and restore stuff i think that's a great addition to this this service that cloud providers can now instantiate for free um you know we've got tanzania included in flex core tanzania basic and no additional cost for that entitlement object storage extension is no additional cost and now you've got backup and restore services of kubernetes clusters at no additional cost so this is really um i think providing a full kind of life cycle end-to-end to that cluster management service which was definitely needed before okay any uh final ones galena york i do have some um of course um have a look at the official product documentation and we also have a white paper that explains again the architecture of operator object storage extension there haven't been any changes from the older versions also the upgrade is in place upgrade is pretty straightforward you just well follow the upgrade guide by installing the new rpm on your object search extension servers but still of course released uh read the documentation about the upgrade procedure especially from older 1.5 version of object storage extension if you're still on old version because they have some database changes that you have to apply and yeah otherwise for the tenant policies which are these claudium specific features we have some extensive documentation as part of our object search extension documentation so that explains the details how to set that up and why it's used and yeah the white paper explains the overall architecture how you can use and install and architect design object search extension if you are net new to it excellent i'll make sure those are in the uh the description of the video as well well glena york thank you very much for your time today it's been great to run through object storage extension 2.1 and look forward to seeing 2.2 thank you good thanks thank you bye [Music]

Channel: VMware Cloud Provider

Views: 6,786

Keywords: vmware, Object Storage, Object Storage Extension, VMware Cloud Provider, VMware Cloud, VMware Cloud Director

Id: EPRkiYqELH0

Length: 43min 10sec (2590 seconds)

Published: Fri Oct 22 2021