New Boot SSD for my PROXMOX System

so kind of minor update for you guys today this is meal laab I'm sure you've seen this in plenty of my videos it's my testing proxmox system whenever I make a video there's a good chance that I'm testing it out on this one first so megalab has a problem and that problem is SSD wear out so I've got a sat m.2 in this thing I bought it for an old project a long time ago where I had to have SATA it's wearing out it's only 500 gigs so today I'm going to upgrade it fwat offered to send over an SSD to me as long as I tested it out so here it is this is the fwat FN 955 it's a 2 terb dless nvme SSD one I already had in there was also dless so hopefully this will be a little bit faster this is TLC so TR layer flash memory and this guy is going in there I mean it's bigger so with the amount of stuff I'm storing it'll have four times as long to wear out at least it's also brand new so it's not worn yet and so yeah come on along and I'm going to migrate my existing ZFS pool my boot pool from the old SSD to the new SSD show you how to set that up get that expanded come along in this adventure so it came with an itty bitty little screwdriver and oh there was a screw in the box that already lose it sh oh there it is okay so got the SSD itself it's got a little label on both sides it's going in that slot there it came with a tiny screwdriver which is nice and a tiny screw which is also nice but I'm missing the standoff and I think I took the standoff to hold down the other SSD I think this is the right size bigger so I'm going to take the standoff off and replace it with tape put the standoff back over there where it belongs screw down the new fwad SSD this one I'm just going to hold in with tape because it's coming out soon if you doing this yourself hopefully your motherboard already has a standoff installed get in there come on don't cross thread this is like double-sided duct tape but uh it's what I have on my bench today so it's what I'm going to use so they put a sticker over the screw which is nice so we don't lose it so here we go going in the slot this board is kind of odd it has one m.2 nvme nvme only and one that SATA only so I know that's how they did it on this platform the nvme goes straight to the CPU so I'm not going through the chipset it is still only gen 3 with this CPU but that's okay still a lot faster than the SATA I was doing before so now we Power It Up and the way we powered up is by shorting buttons are they these two to here we go so now let's go into proxmox and see how we migrate this drive over so before I put the new Drive in service I kicked off a benchmarking run I'm using the Linux tool fio the flexible IO tester I also pulled a script that someone else wrote off of GitHub which will batch automate these tests so we'll have data on sequential reads and wrs random reads and writes through a bunch of different IO depths be nice now of course these test take while so got a snack we're going to talk about dram lists and host memory buffer ssds little history lesson from appap alard long ago in the days of spinning rust we would have to address where our data was on the rust so we' use a term called CHS cylinder head sector that's a huge pain in the ass for software So eventually we decided that the way to address fixed medium was logical block addressing LBA this was the index into the block so the hard drive controller be responsible for dealing with how big the drive was we would just say I want block number 100 traditionally blocks were 512 bytes modern devices actually use 496 it's how you get a shift of 12 and ZFS 2 to the 12 is 496 some drives still tell you they're 512 because I don't know Windows is stupid or something but uh in reality they're pretty much all 496 now then came flash memory and flash memory had a Quirk we weren't prepared for the flash memory wears out when you use it I mean hard drives wear out too but they wear out from like spinning and from turning on and off and things like that flash memory wears out when you erase it so every time you program it you have to then erase it to use it again and that physically wears out the flash memory cells so life is primarily based on how many times you've programmed and erased them which is how many times you've written to the dis so to prevent some operating system from having a bunch of files that it never touches that it just reads that sit there forever pristine and having some other files like log files that it reads and writes to continuously those particular cells on disk on flash would end up getting destroyed more quickly so to De with this flash fry controllers implemented something called wear leveling basically every time the operating system wres to a block The Flash FR controller will put it somewhere that doesn't have as much wear on it and over time it'll keep moving blocks around so it's always right to the end of the disc coming back to the beginning if the blocks haven't moved in a while it'll move them anyway just so it can reuse that block and level the wear across all the different cells so as part of wear leveling the flash memory controller has a table and it says the operating system gave me block 100 I put it over in Block physical block 420 and then if it ever moves that block it updates the table if the operating system reads block 100 it knows to go to physical block 420 turn the data so that means that every time we do an IO operation we have to either read from the indirection table or allocate a new entry in the indirection table and that kind of add some latency to our steps unlike a spinning drive we can't just go seek to that address we have to read the indirection table walk it find where a physical block is and then we have to go there so this thing I've been waving around is a really old pcie flash drive it's an IO memory what is it Fusion IO drive so these guys predate nvme and they relied on a kernel driver in Linux to do the IND Direction so when the Linux kernel wanted to access a block it would keep a copy of that indirection table in system memory it would walk the table and then it would tell the hardware what physical block it needed that was much more efficient because the physical Hardware just has to be flash memory and in this case it's an fpga because they didn't have flash controller A6 back then they did pcie but it also means that that indirection table is stored in the Linux kernel in system memory so when the industry moved to a more generic interface either reusing SATA or eventually the nvme standard they needed a way to emulate that old LBA address adding that we had back in the day of spinning rust and do that at a hardware level not relying on a kernel driver so now in modern ssds the controller chip which in this guy under the big heat sink that will do this indirection lookup and it will store the inter table in its memory so now higher end SSD controllers include a dam chip either on the controller itself or as a separate chip that stores all of the indirection tables and all the other information the wear leveling algorithm needs so when the controller does wear leveling goes to its fast Dam uses that as a cache of all of the indirection all of the wear leveling information so the reads and writes go right on through to the flash pretty quickly lower-end controllers don't have a cache so they have to go back to the flash memory read stuff write stuff from flash memory every time they do an operation and this is slower now with modern nvme ssds there's an intermediate option that is called the host memory buffer basically the nvme device asks for some number of megabytes of memory from The Host system to use for its own purposes and so this way the flash controller can access host memory over PCI Express it's not quite as fast as having your direct Dam and it's also not as big but it can cach some of its wear leveling information and do stuff relatively quickly without the cost of an entire Dam chip on the controller so this particular SSD that I have is that variety it uses host memory buffer I can go into Linux I can verify it's using host memory buffer in this case it's using 40 megabytes of host memory so we don't know what it's using that for but it's using it for its something involved in its we leveling routines so this is the path that most consumer ssds are going down now they're getting rid of dram on board and they're using host memory buffer to fill in the Gap oh wow look at that fio finished now we have data so this is a lot more data than something like you get from Crystal dis Mark basically on the xais we have the size of each request kilobytes each line is a IO depth so the number of IO happening at once a number of parallel requests outstanding and then the y- axis is throughput in megabytes per second so over here for random reads we're hitting 3500 megabytes per second pretty consistently at any IO depth greater than one that just so happens to be very close to the maximum bandwidth of PCI Express gen 3 by four lanes which is what I have this hooked up as so we're saturating the PCI bus random right show a bit of a different story for large requests rable to get to almost 3,000 so not quite saturating PCI Express on the random right side sequential reads again very similar to random reads so for hard drives in particular sequential and random often has a big difference because the hard drive has to physically seek its alarm back and forth ssds don't have to do that if they have the whole indirection table in memory they shouldn't have any time difference between random and sequential reads and looks like this is pretty much the case now sequential rights are a bit different in this case it's also able to mostly saturate the PCI Express Bus roughly um in right mode so given that I'm satisfied with the performance of the drive let's get our boot Drive copied over get ZFS happy grub happy the EFI loader all that good stuff so in prox boox there are four things we need to copy to create a new boot disc so like any dis the first thing we need is the partition table so on virtually mod systems this is a GPT style table this is basically a list of all of the different partitions on the drive where they are how big they are what format they are what their uu ideas so we need to create a new partition table on the drive that fits the new sizes of the new drive but also has all the boot partitions and stuff at the beginning then the first partition is almost a megabyte that's the grub loader so if you're using Legacy bio style booting you'll need a grub loader second partition is the EFI system partition if you're using EFI you need the this partition but also this is where all the Linux images and in RDS are stored so whether or not using EFI or the gr loader you always need the EFI partition at least with proxmox and third partition this is where the bulk of our data goes in this case this is ZFS so four things to copy partition table grub loader EFI system partition ZFS partition let's get on that so I come here and I can Alis block on my two devices SDA and nvme we can see we have our three partitions here so that's the GR loader little under a Meg that's the EFI partition half gig and this is ZFS so that's the rest of the drive and then the envme drive is unpartitioned so we're going to use a tool called SF disk we're going to dump Dev SDA so it looks like this so this is all the information about the the partition table on dev SDA so it tells us how big the drive is ETC and then we have three partitions and how big they are so we're going to save this to a file call parts so this file contains list of all the partitions but it's also the commands back to SF disk to create a partition table because it dumps in the same format that it receives so now we're going to edit this file we created and tweak it a bit to match our new partition table so Nano parts so first thing I'm going to do is I'm delete last LBA so this is where the end of the drive is and since the new Drive is bigger that can go away next up we have three partitions and these are all fine but they need to not have a uuu ID so when a partition is created a randomly generated uu ID is used and a lot of things in Linux will Mount by uu ID and it's really bad if we have duplicates so if we delete them that'll tell s disk that it needs to recreate new ones when it generates the new partition table so we're just going to backspace the uid but we're going to leave the type and then for the last partition the data partition I'm going to replace the size with a single Dash this tells SF disc to fill the entire dis so this last partition the ZFS partition will just fill up the whole Drive however big it is save that so now we just have to tell SF disk to run these commands so we're going to cat this file back into SF disk Dev nvme N1 parts so now it's going to tell us the partition table has been altered so that means it's done it's also going to reload the partition table so if we're on LS block again we see we have our partitions on the old drive and we have our partitions on the new drive and this fills out the whole Drive 1.8 terabytes so next up we need to copy these three partitions and the strategy I'm going to take varies a little bit based on what's on the partition so first partition is the grub loader we're just going to copy with DD so that's for my source o is nvme0 N1 P1 go there we go done next up for the EFI partition proxmox boot tool will copy all the data to it but it needs a valid file system and one of the easy ways to do that is just to copy the other file Sy with DD again so we'll run DD and this time we'll do partition two take a little bit and then it'll be copied then we're going to run poxx boot tool in [Applause] it and we're going to say grub if if you're using EFI boot you don't say grub you just leave the grub off so now for ZFS since the system is running we can tell ZFS to add this as a mirror so I currently have Dev sda3 is my ZFS partition I'm going to turn that into a mirror with sda3 and nvme O N1 P3 so if we look at zpool [Applause] status so our pool is named our pool we see it did a scrub and it's pretty happy and it has this one drive so now we're going to do zo pool attach pool name is our pool so then we need to copy this guy here so I'm just going to paste him in and then our new Drive which is Dev nvme 0 N1 P3 now if we status you can see they are both there and there is a res silver in progress so ZFS is now going to read every block off the disc copy it to the new dis then we can detach it in ZFS once this res silver is done then I can reboot the system to remove the old drive so it has now been one week since I did that migration despite the fact that I'm wearing the same shirt it's from Veronica explains by the way the system was running just fine with a ZFS mirror so I did a ZFS detach that's where you take a mirror and you split out one of the drives from the mirror so it becomes a single drive after that I shut down the system pulled out the drive rebooted it booted up fine into the new drive so no issues with the grub loader EFI loader all of that was just fine it found the new rpool or I guess it's the same rpool on a different Drive booted into it system is running great if you're interested in buying the drive in this video I have a link to Amazon down below for that if You' like to support me I have a Kofi as well if you like to chat with me Discord server link down below for all these things and as always I'll see you guys in the next adventure

Channel: apalrd's adventures

Views: 15,757

Keywords:

Id: m4zwJvzQ65s

Length: 17min 6sec (1026 seconds)

Published: Tue Feb 20 2024