Azure Storage and Disk Encryption Deep Dive

hey everyone in this video i want to dive into the storage and disk encryption that we can actually leverage in azure interesting tidbit when you use archive storage your data is actually copied to floppy disk drives there are millions and millions of three and a half inch disk drives and they copy the data and if you mark it as immutable it just flicks that little black tab up anyway enough foolishness so i want to dive into really thinking about what is the encryption options we have available for our storage and i can think about the most basic level of storage is well i create a storage account so that storage account lives within a certain region and into that obviously i i copy my data now by default since i think it was october 2017 storage accounts are just encrypted by default now if we think about what powers a storage account i can think about well there's a service and then underneath that service there's a set of infrastructure now in reality there's a whole bunch of different kind of partition layers and streaming layers etc but for our purposes i can really just think about it well there's the service and the infrastructure that enables me to have this storage account into which i write my data my blobs my tables my files my queues and as i mentioned so by default this is encrypted now it's encrypted using this 256 bit aes it is fips 140-2 compliant now how is that actually working what's happening behind the actual scenes so i drew data here so if i expand this out just a little bit i can think about well remember those layers the service and the infrastructure so the thing about my data is sitting within a storage account now what's happening then is that data is encrypted now it's encrypted with something called just remove that a data encryption key now a data encryption key is a symmetric key i.e the same key that i could use for example to both lock and unlock so this is very efficient for kind of those encryption decryption type of operations now that data encryption key that's actually used to encrypt the data is then protected by a key encryption key now by default that key encryption key you can think of it as a platform managed key that goes in the microsoft key store and if i do nothing else that's what i get if i go and look at a regular storage account so over here i'm just going to pick this very regular storage account i can go and look at my encryption and i can see it's just a microsoft manage key i'm doing nothing special here i just have this key available microsoft are responsible for rotating it based on whatever the regional compliance guidelines are i really don't have to worry about it it's just kind of done for me now maybe i don't want that i may have a requirement where i want to control that key i want to be in charge of holding the key so i could do something basically revoke all the access to it i want to be in charge of when i rotate that so in azure we also have the idea of azure keyboard now in azure key vault i can have secrets bits of data that i can uh write and fetch and then keys where i can do cryptographic operations with them and also certificates now remember that key encryption key that is asymmetric so in that world whichever key is used to lock it it's the other key that can unlock it and if this key locks it this other key has to unlock it and so the whole point here is if i want to send something to protect something to someone i would encrypt it with their public key so only their private key can decrypt it so that key encryption key is an asymmetric key now if i want to actually be in charge of that what i can absolutely do is well i can have a key in my key vault i could bring that in i could generate it and now i'm saying actually i want customer managed key so i have now a cmk so the key is now living in my key vault now this is blob and files only and that's where i can actually leverage that functionality the nice thing here is obviously i still have to rotate that key but for storage accounts they can actually be configured that when i rotate the key i still have to do that rotation create a new version but when i do that the actual storage account will see that and start using that new key for me automatically so if we now jump over and look at a different storage account so if we go back over to our storage accounts and this time i'm looking for my bring my own key so i've got bring your own key encrypt i look at my encryption well this time you can see i'm using a customer managed key and you can see it's pointing to a particular key and i've got automated key rotation so as i generate a new version of the key the storage account will automatically actually go and get it so if i was to look at my key vault that i'm actually using here and i look at my keys we can actually see the the various keys i have and as i create new versions it will actually go and get that automatically now an important thing here is you need this kind of soft delete functionality because if you were just to go ahead and delete the key well now i've lost the access to the data so we need that softly enabled to make sure we don't do something silly and essentially delete our key and we lose access to our data there's nothing microsoft can do if you deleted the key and you contacted microsoft the best they can do is give you their condolences and they could empathize with you they can't do anything it's encrypted they can do nothing to get that back so in the key vault we make sure we have that kind of soft delete um so we protect the keys so we don't lose access so that's where hey i can now control that key instead of it being platform managed i'm saying i want it to be a customer managed key so this is all really at the service layer there is actually an ability to have double encryption and they call it kind of this infrastructure level encryption and what happens here is if i think about this is kind of that level what we can then also do is we take this and we basically encrypt it again with a different data encryption key and now that data encryption key is protected by a different key encryption key now this is always platform managed there is no customer managed key for this infrastructure level but if i needed kind of this double level encryption i can turn this on the idea of this is it uses a different algorithm so if in the worst case scenario somehow the encryption i have at one of the layers got compromised i still have another layer of encryption using a different algorithm to still actually go and encrypt that data so encrypting the data twice we have two different algorithms so here if i go and look at this and i'll look at a different storage account so i guess first thing is when i actually think about this is if i was actually i'm creating one of these accounts if i did kind of a new storage account what i can actually do in my advanced is we can see we have this option for infrastructure encryption now i have to go and enable this and the documentation walks through the process i basically on board to this infrastructure encryption and then when i actually create the storage account i can set this to enabled to now get this double level of encryption now in my case i have this storage account here and we can see yep so i've got this kind of microsoft managed key here but you'll notice i have the infrastructure encryption as well set to enabled so this data is essentially now double encrypted and i could it's up to me i could for kind of that layer have a customer manage key um or i could keep it as platform managed and then i can have this additional layer this double encryption so at this point this is at the storage account level everything i'm writing um the blobs um in this case and the files they're using um this custom managed key for example but it's everything what about if i'm a company and i have a storage account and actually i have different customers and i actually want to somehow be able to have different encryption for different blobs maybe different containers so we can do that so we have a construct called encryption scopes so now i could really think about this ability that i create these encryption scopes so for example i'm going to create two i'll have encryption scope one encryption scope two and then i could think about well i just have the default storage account if i don't specify an encryption scope and then i can write my data into these things so this is blob only so at the blob per blob or a container level and gets inherited by the blobs i put in it i can write to these encryption scopes now each of these encryption scopes they have their own data encryption key one data encryption key two data encryption key three and then obviously once again as we would expect each of those has their own key encryption key one two three and for all of these it could absolutely be key vault so i can use a customer managed key or i can use the microsoft store i.e platform managed key so for all of these i can pick any of those i want so i create these encryption scopes and then i essentially i have a marker container as hey you're in this encryption scope and an eight block i put into it will inherit or at a per blob level i can actually say hey um i want you to be in this encryption scope so now i have different encryption for different blobs different containers but these are all in the same storage account and that's kind of a key point now so if in this case we actually jump over again so here if i actually go and look at this and i think i've got this once in my bring your own key so in this case notice i have encryption scopes and you can see i've created two encryption scope a encryption scope b both for managers using microsoft managed if i add a new encryption scope you can see i could pick to use a customer manage key if i want so i have complete kind of flexibility there and then what we have if i go actually to my blob and we can see here i've got a container i've created which is container scope a if i actually look at this um and the details of it so the properties you can see its encryption scope down here is encryption scope a now anything i then uploaded to this would automatically go into that encryption scopes if i uploaded a blob to that container it would use that encryption scope now if i just go to a normal container that is not using an encryption scope you can see it's using the account encryption key down here at the bottom well now if i was to actually upload something so if i just pick some kind of random file i don't know what that is if i go to advanced you'll notice i actually have the option down here for encryption scope to use the default container scope which could either be the default storage account or it's been marked as a particular encryption scope or because it's using the d40 that container isn't using an encryption scope i could actually specify at a per blob level a particular encryption scope so now i can think about really breaking up how i really think about well how am i doing this encryption i have a huge amount of control on this okay so this is really all about kind of storage accounts and it's all about really this server-side encryption let's take it up a level because if we're dealing with virtual machines we don't really deal with storage accounts anymore what we're used to doing is if i think about my virtual machine so here's my kind of vm what we deal with these days are disks now if you go back through time in ye old days what actually happened is you would create a storage account and then you would basically create a page blob and then inside there you would have a vhd and that would then be used by the virtual machine the problem with that was you had to start worrying about well what's the um throughput and iops limits of the storage account how many disks do i have in that thing this wasn't a first party azure resource so it had zero clue about rbac and snapshots and images and so a managed disk basically takes all of that and just abstracts it it's still there i just don't care about it anymore i don't see those things that's kind of just done for me so for this disc so we're now thinking about here once again by default managed disks are encrypted and once again there is kind of that data encryption key that once again is protected by the key encryption key the once again here is is going into that microsoft keystore so it's that platform managed key that's the default if i do nothing else that's what i get and then obviously i attach discs the virtual machines and i use them but what about if i want to actually control the key i want a customer manage key so here what we do is we create this thing called a disk encryption set so we have a disk encryption set now for that disk encryption set what i actually do is once again it has its own data encryption key as you would expect and then that data encryption key is protected by a key encryption key as always that is going to go into your azure keyboard where that will be a key so now this is actually making this a customer managed key and at this point i put disks into the disk encryption set so i say hey i can either create a new disk into that disk encryption set i can move disks into it now to move a disk the disk has to either be detached from the virtual machine or the vm has to be deallocated it can't be in use at the time but essentially this is going to let me now hey i can put multiple disks in this disk encryption set they will be encrypted with my key and what you can actually do as well is you can turn on that double encryption so i absolutely can turn on that encryption where i have a double encryption where i have the key encryption key that goes to the microsoft keystore so i can turn on double encryption that that's totally available to me so if we look at this so let's jump over so in this case we're going to look at disks so if i jump up here and just look at my disks now this is completely transparent to the actual virtual machine like there is there is no knowledge of this now remember this the disk has to be in the same region as the disk encryption set um and the key vault so here if i just go and look at kind of random disks i might have i'll look at my windows 10 i think it's my data so a quick look so you can see here it's in a disk encryption set there we go oh it's messy in this des ussc that's all i do now notice i can't change it because that vm is running if i picked a different disk where the disk isn't running well now i'd be allowed to change it i could change it to an encryption at rest with a customer managed key and i'll see my disk encryption sets i could change it to a disk encryption set where i had turned on double encryption which i don't have any of if i actually go and look at my disk encryption sets you can see i created one again it's in the same region and here you can see the key i'm using and there's that current key so once again it's going in using a key actually in my key vault so that's where i can really think about hey uh i can use that once again i could move this if i create a brand new disk i get exactly the same um set of options available to me here i can kind of pick what i want to do so that's what i can do with disks so with a disk hey i can have the microsoft manage the key or i can use my own via this disk encryption set but it's some kind of interesting about disks more often than not nearly all the time we connect them to some kind of compute a virtual machine a work node in aks something well that vm it has an operating system and those disks basically surface as disks inside of that operating system and so if we think about it operating systems also have encryption capabilities so if i think about it well windows has bitlocker and linux has that kind of dm crypt so they have native encryption options and as you would kind of expect yes this disk kind of surfaces here that the content i can encrypt it within the operating system as well so that is using that bitlocker or the encrypt of its linux and this is known as azure disk encryption ade and what's happening here is we actually have there's kind of an extension that plugs onto this and the key that's used for this encryption as you would expect goes to your key vault as a key now in this case i have to turn on a flag in my key vault to say hey i want to use azure disk encryption i have to actually enable that capability and i can do this for the os disk and i can do it for the data disk i can even do it via flag so it's the temp disks as well that temp disk i can do that as well and then it will go and actually encrypt all the different things so if we look at this one and jump over so in this case if i go and look at my virtual machine and i've turned it on on this vm and this time if i look at my disk you'll actually see it says well sse platform manage key plus ade azure disk encryption so i have enabled it for that now the opposite of moving a disk into a disk encryption set to enable the azure disk encryption i have to do that while the virtual machine is running so if i picked a different vm that's running and i go to my additional settings this is where i can actually go to say hey i want to turn on the azure disk encryption for the os or the os and the data disks so that's for those additional settings but it has to be running because it's going to go and talk to the operating system and say hey use bitlocker or use dmcrypt if i actually go and look at this virtual machine this is that ds01 you can see inside it it's got bitlocker is on it's going through azure disk encryption is now using the encryption within the os to encrypt that data so that's kind of a big difference really between them one is kind of at the infrastructure level one of them is is actually going through and it's at the os level so it's kind of always encrypted it's kind of an important point if i look at this this encryption here we can go back to kind of our original picture it's encrypted at rest in the storage account or the disk is encrypted at rest but while it's running on whatever host is doing that it's decrypted before it's sent over to the host so any cache files here is not encrypted now the data in transit is not encrypted generally we don't care this is an azure data center we don't care about that stuff but if i'm actually using ade it is encrypted the cache is encrypted on the machine and that disk is encrypted kind of in the transit as well so there's a difference between them now one thing you might notice i kind of i'm talking about this connection you cannot mix a d e and disk encryption sets it's one or the other because and then the question comes up which should i use so i obviously i have that platform managed encryption and then i can encrypt inside the os or i use the disk encryption set where i control the key and i can do that double encryption here there are restrictions on where i can use azure disk encryption like custom linux images is not going to work in in many scenarios so that there are requirements it is doing work inside the os but you get that benefit that the cache is encrypted on the host it's encrypted in that transit whether you care or not and i can turn on kind of the temp disk encryption as well as part of this configuration whereas with the disk encryption set it's encrypted at rest but it's not it's decrypted as it sent over kind of that internal azure network to the host running the vm the cache is not encrypted the temp disk is not encrypted but this azure disk encryption is definitely um this disk encryption sorry the disk encryption set there's it is the way forward that's where things are going to go and i think we'll see less and less of using that azure disk encryption but what about that gap what about if i really want this channel kind of encrypted between the machine and i want the local caches so if i think about i have the host remember and that host has the vm and i can think about also on that host i have really kind of the temp disk for that vm and i also have if it's got caching turned on the cache disk for that vm and then obviously there's kind of the the disk over here where it actually sits and that gets connected through to the host maybe i care about those other levels of encryption i i actually want that and so what we can now do is there's this capability coming um i don't think my subscription i'm going to try it again but it wasn't there you have something called host based encryption so i have host based encryption now with the host based encryption now all of these other things are encrypted as well now whatever key is used to encrypt the disk so if this for example in this case this could remember be a cmk or a platform managed key could be encrypting this disk whichever one of those is encrypting the disk will also be used to now encrypt it as it goes over the wire to the host and it will be used to encrypt the cache disk the temp disk is also encrypted but that is always going to be a platform managed key so that's kind of the distinction between them but i can now turn on this host-based encryption again it's kind of in a limited capacity today i'll see if i have it i don't think i do but that really solves the problem so in this world once we have this host-based encryption there really would be no reason i would use this azure disk encryption within the os anymore i would use hey the disk encryption set if i want my customer managed key for the here and it will make sure any local storage as well on the host is encrypted with my key if it's the cache of the disk or platform managed key if it's temp and kind of over the wire so that that's really where everything kind of comes together for this so we'll have a look and i'll show you where it would be but last time i looked i still didn't actually have access to it so if i actually went to virtual machines over here and i said hey i want a new virtual machine it's only in certain regions but i could say for example well south central i'll pick windows server 2019 i would i won't do that at this point don't need that um there's certain sizes are supported at this point but if i did i'll just do it i think b2 works so what i would actually see under my disks yeah so i i still don't have it available on mine but i'll open up the documentation so we can see what this should like look like actually let me try one thing super super quick so there's actually a special portal you have to use i just realized i didn't do that but i don't think i have access to it but here i would go into the vm um once again i would pick the region pick the os pick the skew and then under disks yeah i don't have it but basically so let's actually see what this should look like so what you would get if you were worthy which i am not and once your subscriptions you have to apply for this and it's taking its time but what you would see is scroll the way down you have this encryption at host option so once you're on boarded for this now with that capability i would actually have that additional encryption at that layer as well and i forgot to show it just super super quick when i think about that key vault remember if you are using the azure disk encryption you need to make sure in the access policy you have just hit that button to say azure disk encryption for volume encryption has been enabled so that's kind of a key point there so that's really trying to bring all of these things together so for regular kind of azure storage hey we have these different options encryption scopes i can bring my own key i i can actually have that double kind of encryption capability with that infrastructure level for disks i use a disk encryption set if i want to use my own key or it's an or i can use azure disk encryption um as disencryption that gives me a few extra things because it does encrypt kind of that local caching as well and kind of because it's encrypted at the disk is encrypted over the wire as well and i can't mix them but when i now in the future can combine that when this is a broadly available with the host-based encryption i get the best of both worlds um it's not an overhead on the vm it will work with anything because it's transparent to the vm i can use a custom managed or platform managed key it will be encrypted over the wire to the host its cache will be encrypted with the same key as the disk the temp drive will be encrypted with a platform managed key final thing if you are doing a custom managed key means you're responsible for key rotation there's not a right or wrong answer many companies will go for kind of a six month rotation but it depends on the industry and depends on the regulations remember storage accounts when you rotate the key will automatically detect it and use it today that is not the case for a disk encryption set if i rotate the key i need to go and update the disk encryption set to use that new key there are things like azure policies actually if you go and search for kind of key vault there are policies specific to key durations and expiry dates so you could actually go and audit and check the right things are happening around your keys so that's it i hope that kind of explained what the different layers are i think there's been some confusion about well hey i've got encryption on my disk but this ad is showing it's not encrypted they're two different things there's disk at the encryption at rest level and then there's encryption within the os using bitlocker or dmcrypt and again i can't mix my own custom managed key using a disk encryption set um with azure disk encryption and i really think once we get this kind of host-based encryption we won't use that azure disk encryption anymore it won't be required we'll use the disk encryption sets with that host-based encryption so uh as always i hope that was useful if it was uh kind of a like subscribe comment and share is definitely appreciated um but until next time you take care of yourselves you

Channel: John Savill's Technical Training

Views: 7,254

Keywords: azure, azure cloud, azure storage encryption, azure disk encryption, customer managed key, disk encryption sets, host-based encryption, platform managed key

Id: EOXgzTqceok

Length: 34min 54sec (2094 seconds)

Published: Thu Feb 04 2021