Proxmox 7.1 Guide: From blank system to Hypervisor

today i'm going to be going over my guide for proxmox virtualization server and just kind of some of my tips and tricks i've learned over my years of using it for quite a bit of uses on multiple systems i've used in many different use cases ran my own tests on it and just kind of have a view of what kind of works best in my opinion and i'm going to be going over a full setup taking a system from no os to full proxmox and going over all the little tips and stuff i've learned along the way so this is probably going to be a pretty long video with all the little tips i'm going to try to put in here hopefully i'm going to put little thumbnails in the bottom if you want to skip to a certain part like installing the os hardware recommendations once it's in the os what tape steps i take to get it working correctly but before i get into that let's talk a little bit about proxmox and why i typically choose proxmox so proxmox is a hypervisor which is a term for an os that basically runs other os's some of the other competitors on the market are like microsoft's hyper-v server vmware esxi and some of the other ones based off kind of the linux stack now proxmox is also based off the linux stack using linux's kvm hypervisor and is really just a nice gui and management protocols for that but i think the nice gui really does make it a better os and gives you a lot of easy to control functionality that it's a lot more of a pain to do if you ought to write it yourself and that's why i like using it because it makes a lot of things that you can do in kvm with the command line but proxmox makes it really nice gives you a good web ui makes it easy to add like other nodes adds a lot of good storage stuff manages updates and just generally makes it a good os and the other thing is for someone like me who does know linux pretty well under the hood they let you play with stuff a lot of it is relatively open to view relatively accessible and follows normal linux practices unlike other os's which really try to hide that stuff under it and just say here's how it should work they're like if those issues with config files there is the config file you can edit it and because of this i've been able to find answers to most of the issues i've ran into online and also been able to fix almost every issue i've ran into when it comes to clustering networking config files lock pro file problems and it's been a pretty darn stable reliable os in my experience i don't really have any huge complaints for it i wouldn't say it's perfect and i've definitely seen some ui i would have done it differently but as an overall os it works pretty well and the other thing that makes it really nice for home lab use and home use is the full version is free and what you pay for is support which i'd recommend if you're a business user something where it's critical for the operation of a money producing enterprise but for home use where it's not really needed it's pretty good the free version and compared to other things like esxi which is relatively limited on the free version you can set up a full cluster on proxmox completely for free and have the full functionality of the os and be able to fully use very high-end hardware with the fully free version now that i've gone over some of the upsides i see in proxmox there's some other downsides though the first thing is it's a relatively small player especially of commercial software so if you want to use a lot of like commercial pre-made vm templates or backup software stuff they often aren't available for proxmox or you're going to have to do some of the work to make it work and also because it's smaller it's less used in corporate environments so saying it on your resume might not be as useful but it is a lot of the same concepts and stuff to what the larger players like vmware esxi and hyper-v use so it's pretty easy at least for me to translate my knowledge from proxmox to another hypervisor as a lot of things like memory allocation and stuff are pretty similar a few different details in something like esxi but it's the same concepts and all the hypervisors now that i've gone over some of the pros and cons of proxmox let's go a little bit into hardware and what type of hardware i'm picking today and what type of hardware i generally recommend to use of proxmox so proxmox on the website actually has a hardware requirements page which i have pulled up right here and looking at this it is super minimalistic like it says two gigs for the os a x86 cpu if the needed instruction set and network storage and some sort of storage device it's pretty darn minimalistic and while proxmox will run on that type of hardware you'd probably want to run it on a lot more and my general rule of thumb for the amount of hardware you'd want to throw at proxmox is take kind of the requirements of proxmoxio so maybe about two gigs of ram a basic cpu and a little bit of storage and then add what your vms need so depending on if you're switching from physical hardware you can look in task manager and say hey that vm's using eight gigs of ram 120 gigs of storage and maybe the cpu is running at 40 on average or something and then kind of add all those vms up if you haven't ran these storage systems before you can often estimate from system requirements what a basic install would need and then i'd say add a 50 buffer on top of that because software on the future is only going to need more resources storage requirements just only go up memory requirements only go up two so add it but let's get a little bit more into specifics about hardware so first thing cpu i generally recommend a reasonable amount of cores and probably still try to get pretty good clock speed too i think clock speed's a little bit overlooked in a lot of these servers because while a lot of stuff's pretty well multi-threaded a lot of things aren't and things like windows updates and app updates and a lot of little scripts are pretty dependent on single threaded performance and if you can get that a bit faster it works better but also it spreads very well amongst core count because you can essentially just take the vms and run them on different cores or sockets if you have a multi-socket system so some of the cpus i'd kind of look into if you want kind of a desktop type build something like an i5 12 400 now as it has an igp so you don't need another graphics card very good single threaded performance pretty good multi-threaded performance and relatively inexpensive if you want cheaper there's the older generations the ryzen chip is also great and they also have ecc support which is great too and also if you want something use something like a xeon e5 v2 v3 v4 generations are pretty nice systems too and they're fairly easy to find relatively cheaply used if you want even cheaper and lower power you can kind of get the older i5s with like the i5 4000 generations which are a bit slower but still works fine and can run lighter vms just fine memory wise for evaluation hardware they say one gig of ram but they recommend two gigs i'd say you should probably run proxmox on a lot more than two gigs of memory my kind of general starting point is 16 gigs but i'd probably say aim for 32 or 64 for a home lab and potentially more as memory in my experience is the very first limiting thing with hypervisors it goes to first and it's the most needed you normally run out of memory before you run out of cpu or disk i o first so one thing i would look for in proxmox is a system that can handle a lot of memory as a lot of desktop boards are relatively limited in their memory capacity but the server boards can often handle terabyte or half a terabyte of memory relatively easily so if you want to grow it might be nice to have a system that's capable of expanding to a lot of memory in the future the next thing to look at is disks and storage so there's a lot of different ways to set up proxmox storage but there's a few things you'll need first of all you need somewhere to store the boot device and boot data i like to generally keep it separate but you don't have to i find it's just simpler to manage if you have a device that only stores the boot data i'd say aim for probably about 32 gigs at least and you want something reasonably reliable something like an ssd or a pair of them in raid 1 works pretty well in my experience but if you want cheaper usb sticks work but i wouldn't say are recommended mechanical hard drives are fine if you have extra bass for them too it'll boot a little bit slow and update quite a bit slower though if you're on a mechanical drive i'd try to get vm storage on solid state if at all possible today as i find hard drives are just kind of too slow for reasonably fast vm storage i'd say storage performance and i o performance is the next most important thing after ram for having snappy vms because when you have a lot of vms kind of hitting with little bits of random i o the same thing you want to make sure those i o requests can be answered quickly and ssds will do that much better than any mechanical hard drive solution network storage and clustered storage also works on proxmox but isn't really an issue with a single system like this you can set it up to store data on like a nas if you already have one and it supports things like smb or nfs storage networking in proxmox has a few different uses the first thing is proxmox management so this is where you access like the web ui sshn proxmox gets updates from this doesn't need much performance at all and probably the best case scenario would be to have it on a fully separate network so it's more secure but in a single system or home labs type system having it on the same network is going to be fine the next thing is storage if you're using network storage you want to make sure that that's a very high performance thing because as i said before storage is very important to have done fast and then the next thing is vm data so as vms access the internet and stuff they need to have some sort of network adapter and depending on your setup this vm networks can be on vlan so they can be separated or they can be all together and there's a lot of different ways to set up your vm networking if you're doing things like you're having less trusted vms it's probably best to have it on a separate vlan or physical port just so they can't access your management or storage networks but this is probably a little bit too much for a single system here for a single system here for most home lab use a gigabit network card is probably going to be fine unless you want to do high speed network storage and then i'd aim for 10 gigabit proxmox generally works with any network card that works fine in linux which i found as be almost all of them now there's a few systems that have network cards that don't work out of the box and might require drivers to get working or another card after install in that case i'd probably just get one of these little usb to network adapters these in my experience generally work to get the proxmox installer happy so you can do it the proxmox installer actually needs to have a network card to actually continue so this will fulfill that requirement and let you just get the install completed to install your driver later for general home lab use i'd let those three different types of traffic all go on the same port as that's not really an issue but as you expand you might want to have a separate storage or management network and maybe set up vlans for your different vms and then some of the other less important things in the system are things like gpus by default proxmox won't use gpu power at all for anything you can use it for pcie passive if you want a vm to have gpu horsepower to do something but proxmox itself won't use anything so your gpu all it needs to do is be fast enough to just put the installer on the screen which basically any compatible gpu will fulfill that requirement and then the other thing i'd say is just try to get high quality hardware that doesn't have issues having issues of hardware on your hypervisor is not fun to deal with leads to a lot of weird problems and just i generally say for high quality hardware business grade systems are generally better and server grade ones are kind of the best so things like used servers i'd say are generally better than kind of a low end desktop super low end parts also really aren't great but most consumer grade stuff these days is actually pretty good and i can see them run for years without any crashes or issues now that i've gone over all the different hardware parts let's talk about kind of some sample systems that i think would make sense so the first thing is kind of a desktop-like hardware so something like with a core i5 in it 32 or 64 gigs of ram maybe something like a b650 motherboard a few ssds maybe one for boot and one for storage and this i think makes a lot of sense if power consumption and noise are a big thing as desktop parts these days are built to be pretty power efficient and quiet especially under low loads this isn't often the cheapest solution but these can idle at like 20 or 30 watts where power consumption is a major issue that's going to be great the next solution is kind of the used server hardware type of thing and this is kind of the dell r720s r730s the hp dl380s like the gen8s the gen 9s and these make a lot of sense if you want to work with the server hardware and want to be able to grow in terms of ram and expansion so these get you the full server hardware of things like dual psus on off band management cards up to like a terabyte of memory expansion if you want depending on the exact model lots of cores of dual socket support often lots of drive bays lots of pcie slots the problem with these is they normally idle using a good amount of power often over 100 watts sometimes over 200 watts and kind of the low to medium loads on these systems and they can be very loud as they're designed for racks and stuff where noise doesn't matter the last option i'd look at is kind of the used business desktop so this is kind of like the dell optiplexes and the lenovo think centers these can often be found very cheaply as businesses kind of just dump them as they don't need them after their upgrade cycles so not the fastest systems out there and often can't hold that much ram or expansion but it's great for getting your feet rat they often have the very low power consumption that the desktop parts have and if you use the little nodes like the tiny mini micro style ones they fit almost anywhere it's easy to play with clustering with those and no very low power and often near silent under most operating circumstances but these are just some of my hardware uses there's a lot more different ways to use hardware for proxmox as almost any x86 system can work now that i've gone over some of the hardware needs of proxmox let's get it installing on my test system today i have a little usb stick has to be about four gigs but in this case i have a 32 gig which is more than enough space and then today i'm gonna be putting on a little optiplex i have a little optiplex 990 that i'm going to be putting in it on today it's a relatively low end system with like a little i5 22 gigs of ram an ssd for boot and a hard drive for vm storage but it's going to be more than plenty for playing around with running a couple vms and doing some testing on now in order to get it started i'm going to need two things first of all i'm going to need a copy of proxmox so i'm just going to go on their site go to proxmox virtual environment and just say download and i'm going to make sure i get the newest version they often have older versions on their site if you are running like an older branch so in this case that's 7.1 but it might be newer by the time you come along all this stuff in the video i'm going to be basing off 7.1 and what features it has and then i'm going to need a program to put it on a usb stick currently my favorite is rufus so i'm going to go download this program and this is going to let me take my iso i downloaded put it on my usb drive and boot from it and now that i've finished downloading both of the programs i'm going to open up rufus and i'm going to say i don't need it to install updates i'm going to go select my usb drive select an iso image in case in this case it's that proxmox one i just downloaded and i can leave everything else in the defaults in my use case you might want to change it to like the uefi mode if you have a uefi system but now it's ready to go so it's going to take a few minutes to copy all the files to my little usb drive and then i'm ready to install so now that i've plugged in my usb drive i just want to check a few things before the initial setup i have power plugged in i have my network card it makes it easier to set the initial ip keyboard and mouse it's nice to have a mouse for the proxmox installer though i think it's possible with just the keyboard video output and one other thing i like to do is to unplug any drives that i'm not installing the os onto because every once in a while you get confused about what drive you want the os onto and which one's a data drive and if you can unplug it or remove it it's just easier not to have to think about it and now i have the boot menu on my optiplex i'm gonna select the uefi option for my usb stick as the boot option and it's gonna give me the welcome to grub message and i'm gonna just say install proxmox ve and within a minute i'm going to be ready and in the installer so here i am in the install i'm going to agree to the license i'm going to select my drive since it's the only one is the drive now it gives me several options of how i want to set this up i get ext4 so just a simple ext4 this is within lvm xfs and lvm zfs options and btrfs probably stay away from the vtrfs because it's kind of beta right now zfs is my favorite because it allows for easy snapshotting easy i'm adding caches or multiple drives later on i'm just a fan of all the big zfs features and also just the data reliability because it's checksum and i also just like their command line interface so i'm going to set it up as a raid 0 which is the way to set up a single drive if you want one xfs and ext4 i'd say are generally best if speed is your highest priority as they are the fastest ways to do it but i like using zfs for the features as for my use the features are more important than a bit extra speed advanced options i like to leave all at default all reasonable for what i need here and then i can click next i'm going to set in my time zone and location and then i'm going to put a simple password in and then email this email is where it's going to send em notifications i've yet to get any issues of like um marketing emails or anything so that's really nice so i'm going to just put my email in off camera now the other thing of this email is the way it seems to work is it just reaches out with smtp by default and with most home internet connections like i have here it won't be able to send any emails but if you have an internet connection and like a data center a business grade one where you can send out on port 25 i believe it will let you send those emails out so most home internet connections you won't get it by default but a business grade you might so the next thing is it's setting up network address so this guy only has one network card it looks like it grabs a dhcp address and then sets that as static by default so it looks like i got 106 which i'm fine with it also grabs the dhcp dns server and the gateway and sets those statically too change it if you need and then you can set it to install it has this nice little summary and it's going to reboot when it's finished one cool thing of proxmox is installed super fast so it's probably going to only take like a minute or two to do the whole install and also once this install is done you should be able to access it using that web ip address or sshn and you shouldn't need to touch this system ever again for normal use assuming it's all operating correctly so i finished installing proxmox and it should be on the network and ready to be accessed by another system so let's take a look on this computer and see how that can be done so the first thing is it gives you a web url on the actual screen so i'm going to just use that and do the https192.168.1.106 and then the port 8006 and try to load it since this is the first time i've loaded it my browser is going to say there's an issue and there's a unknown certificate for a home lab environment this is likely fine to use a self-signed certificate but if you're in a production environment you probably want to use your actual certificates for your actual website and i have servers running like that and they just work correctly with https in any normal browser but in this case i'm going to just accept the risk and let it continue and then i'm going to get a login by default it is root as the username and then whatever you set your password to as a login one cool thing firefox does that i'm not sure about the chromium-based browsers is it lets you save it which i like to do and it's going to say there's no valid subscription if i pay for a subscription that message will go away and it's a pretty blank looking proxmox instance while we're here let's go over one let's go over the ui quickly so on the left hand bar you get kind of the different views of how it is if you have a single server it's going to be pretty simple here but if you have multiple servers it might be nice to see all the vms together from your multiple systems so that can change the view here but by default you have your data center which is everything your node so in this case i have one node and then it'll show you the storage devices by these little kind of disks icons and then virtual machines and containers will also be shown here and then on the middle shows you what you've selected and information about that thing you've selected so in this case of storage i can set like see what's on it see what permissions it has and then at the very bottom i can see the tasks that i've done so if i open like a creative vm if i restart a vm that's all going to appear under the tasks and i can double click it to view more info in a status bar so to show me more info this can be useful for running tasks because it'll show me if something's running in the background like a backup and what the status of that is and then let's actually look a little bit more at the system settings right now so there's two ways to get it system settings here and those data center which is kind of the data center wide settings and then pve which is your single system so now let's do a little bit of first time setup things before i put vms on this system so the first thing is repositories and updates you can do this both in the gui and in the terminal if you want but i'm going to just do it in the gui for this example so you get the debian gui so this is just the debian version that does all the like the os stuff and a lot of the little packages and then you get the proxmox enterprise by default this is enabled but it will throw an error if you don't have a subscription i'm going to go disable this guy because i don't need it and then i'm going to add a one so in this case i'm going to add the no subscription one and this is gonna basically let me access all the proxmox updates so i can update proxmox without a subscription it's gonna say it's not production ready and it's not perfect i have used it a lot and it generally works fine so we have that and then under the updates we can do refresh it keeps wanting to bug me refresh is apt to update so basically it's going to grab a new list of what possible updates there is and then kind of look through it to see what can be updated and then the other thing i can do is do an apt upgrade so it's going to say there's like 12 debian items and 7 proxmox items to upgrade and i'm going to do that so right now it's going to open a little terminal and let me download and install all the latest updates and then reboot the system i'd highly suggest you do this update before you do anything else because sometimes it breaks stuff and also just always nice to have the newest version whenever you're tweaking with anything else though i will mention proxmox in my experience has been very good about updates not breaking things i've had extremely few issues with updates breaking things and i'd say much less than like hyper-v in my experience so now it says my system is up to date and just to be sure because sometimes it can update things like the kernel and other things that you need to reboot i'm just going to do a reboot you can do that by selecting the server in this case pve select reboot near the top right and reboot node and if i look at my screen here that the actual system is plugged into i can see that it's rebooting and going through all the bios reboots and everything so it'll be probably up in a couple seconds so proxmox has finished rebooting now so i've reloaded the page and i'm ready to start playing with a few more things and getting it ready for my use one great thing with proxmox is it gives you a lot of different options when it comes to storage and how you set up storage but because this is only a single node system there's less options that really make sense here so the options that make the most sense here would either be kind of basic file storage which stores it on any file system lvm storage where it makes like an lvm drive and puts it in that lvm virtual disk and then zfs i'm kind of a zfs fanboy so that's normally how i go but i've seen the use for directory and lvm storage as they're generally a bit faster as there's less kind of processing and overhead involved compared to zfs but i really like the zfs features like its ability to do raid its ability for using caching the check summing feature zfs send and a lot of other goodies it has that i like to use with my own scripts and processes so i'm going to keep doing that so the first thing if i want to add another drive of zfs is go under the server and the proxmox ui and then go under disks and in this case i see i have another ssd in it and i need to set that up so i'm going to initialize the disk with gpt it's going to take a second and it's basically just making an empty file partition table that you can now put your partitions in and then i'm going to create create zfs i'm going to call it donut today just why not and then i get raid level as a single disk here compression is on with zfs it almost always makes sense to do compression but in this case i'm going to set it to zstd as it's generally the best compression in my use and then that's it so i'm going to click create and now i'm going to have another zfs pool so now i have my donut zfs who i can start putting storage in so the way that proxmox works with zfs is it only lets you store the containers images and vm images on zfs directly if you want to store stuff on zfs like your iso files you have to um you have to make a directory on there and store it on that directory but in this case what i think makes a lot of sense is just put iso files on the boot drive as they're generally not very big it's set up already for that use case and speed and reliability don't really matter for the iso files as you generally only need them for the initial install not for later on use and luckily they already set that up with you with with a little directory that's made for that use case so let's take a look at now adding the zfs pool so i can use it because in the server all it lets you do is make the zfs and settings it doesn't actually let you mount it all the mounting is done under the data center tab under storage and actually if you do it this way it makes it and sets it up for you already and then i can see the other two options which is local and local zfs so local is a directory and it's designed for those other types of files i talked about that zfs can't store and then local zfs is just lets you make more zfs files and the way that proxmox uses zfs is it makes a z volt and then it stores your vm image in that z vol so that's what it'll be doing when i start adding vms to it so now it looks like that's set up if you want more functionality the way to do it in the gui is pretty limited so if you want to put it in a nested so i'd suggest doing that in the command line and then you can do zfs create and then add it so if i wanted to do a little bit more i can say add zfs and then i hit say like test and then zfs pool i can see all of my different ones and if i create them in the terminal and use it one other thing i want to talk about with storage for like zfs and lvm is thin provisioning so when you set up storage there's kind of two ways to do it the first is if you make 100 gig virtual disk for example is to fully allocate those 100 gigs and say they're dedicated for this vm no one else can touch that space even if it's not using any of it the other way to do it is to thin provision and say if you make a new vm and it uses 20 gigs it only uses 20 gigs on the disk the advantage of thick provisioning is you often get better performance because then it doesn't have to allocate new sectors and stuff on the way and also it guarantees you never run out of disk assuming you aren't doing any snapshots or things like that because the space is already fully allocated thin provisioning lets you use much more space and in my experience generally vm disks are like half full a lot of the times almost empty so it can give you a lot more free space i almost always enable thin provisioning so i can use less space and then just kind of keep an eye on disk space and try to make sure you always have a reasonable percentage of free space on the system and you won't run into issues but it really depends on your exact use case and if you expect your vms to suddenly need much more storage it's good to thick provision it but i'm in a home lab use so i want to kind of get the best usage of my resources so thin provisioning makes a lot of sense for me let's talk about users and permissions now so by default proxmox only has the root user but you probably want to create more especially if you have multiple users to delegate permissions and also better keep track of who's doing what this also lets you limit how much each user can do so then worst case if someone else gets their permissions they can't do too many bad things so taking a look at proxmox in the data center tab we can start seeing the permissions one the first thing we want to do is add a user and we'll actually be given the option if we want the pam linux authentication which basically means a normal linux user or the proxmox ve i'm going to use the proxmox ve because it lets us set the password here and i'm going to call this one test1 i'm going to give it a simple password in this test use canary and i'm going to add it and now that that user is created it can't do anything i can change the password here and i can click permissions to view permissions i can't change anything with that permissions what i probably want to do is click on permissions at the top and click add permissions and i'm going to click user permissions and then i can say what do i want to give them permissions to so for example i can say like different nodes different storage pools different vms or i can do slash to give them access to everything i can select the user i want and i can select the amount of access they have a list on the website about what exactly each of these do so you can do things like have users that can add um things with data space but can't change them there's a lot of like being able to view settings but not change them and a lot of different uses but i'm just going to give this user full admin access and click ok so now i can see that this user now if i go into users test 1 permissions has full permission to everything in here so they can do whatever they want the other thing i could do is i could set up groups so i could set up like an admin group and give permissions to the admins group like i did here so then that way if the users in the admins group they can do something so that's great if you have a lot of users that should have a similar level of permissions you can do that and then the biggest thing i'd say security-wise is two-factor so you can set up totp which is basically using something like google authenticator scan the code type in the little number every time and you can use two factor it's pretty simple to use i'd highly recommend you use something like that just to significantly increase the security of accounts because then just the password alone won't get you access to anything and then if you want under realms and stuff you can set up a lot more so you can add things like an active directory or ldap and add users and passwords from that and there but i think just setting up a few users for all the different people who might use the server and then the minimal amount of permissions they need is probably a good start so let's talk firewall for proxmox a little bit so by default proxmox has no firewall enabled but it has a reasonable amount of feature sets when it comes to setting up a firewall it has its own firewall system which is separate than the more traditional like ip tables a firewall d that you use in a lot of other linux distros and you can access it manage it via the web gui here or via the command line but i'm going to be doing it in the web gui here as i'm mostly using the web gui for this video so there's two control parts the first is in the data center and the next thing is specific to each one so there's kind of the general firewall settings in the data center things like if it's on by default and the default policies groups of policies you can easily apply a lot of policies for your application for example and then it's specific for each item so in this case it's server and then it's also on the level of the virtual machine so you can have a firewall on each vm so if you don't want vms to talk to each other this is one way to do it or you can have it so your vms can't access on certain parts or you can keep kind of a rogue vm from doing certain things if you want it also keeps your hosts from doing it by default it is all off but if you want to turn it on you can go on your system say firewall turn it on and if you just turn it on the rules are allows ssh and the web portal here so it will continue to work as expected but if you want to allow more applications you can go under the top firewall and add one in this case i was doing some netcat testing and made a little rule that let me do netcat on port one two three four five i'd probably suggest turn it on for the host if you want to play around i normally leave it off of vms i don't see a big reason to as your vms normally have a host but it really depends on what your vms are doing and if you trust your vms in my case i generally trust my vms not to do anything weird because i'm managing them and i have full control of them but if you're not trusting your vms you probably want to do a lot more firewalling and network segmenting of them than you normally would let's talk about networking and proxmox before i get too much into vms and setting up vms so networking by default in proxmox sets everything up as a layer 2 switch where everything is connected so think of it as you have a box where the proxmox system is and then inside that box you have a layer 2 switch so think normal cheap switch and then you have the actual proxmox system so unlike the normal linux system where the system actually connects straight to the network there's actually a switch in the middle so and that switch is the one that gives it connection and then that switch can also connect to other devices like the other vms and containers using and because proxmox is connected to the switch instead of the actual network adapter that means the proxmox host can actually talk to all the vms a lot of the times if you use something like virtualbox on your desktop the host and the vms can't talk so that solves this problem here but then we have the options if we want to do more switches so we can set up another switch and just keep adding them and if we want use more network ports and essentially have fully isolated vms that have their own network ports let's take a look at creating one of those now so i can set a linux bridge right here think basic layer 2 switch i don't have to give it any ieps i don't have to give it any ports and now i essentially just made an empty switch so now i have another switch with nothing connected i can connect vms to it and i have a switch that's only connected to vms and not the outside now the other thing i can do and i've often done is set it up so i have more virtual nics and the way i do that is using vlan on my main nik in this case en01 and i have to do that in the command line but i can edit it and say like en01.20 and that means that if it sees traffic coming in on vlan 20 it'll reroute it to that dot 20 and then that dot 20 is connected to my second switch and then i can set up vlans that way so multiple systems can connect via vlans so i have a trunk port coming straight out of the host here and that's how i typically use vlans here you can also use multiple physical ports if you have multiple network cards and it really depends on how exactly you want to set your network up because it's pretty individual based off your use case here i'm not going to touch it because i think for a basic home lab this setup is actually pretty good the basic switch because a normal home network is essentially just one layer two switch that connects everything so this is just kind of an add on that but if you're doing multiple things you can get very in-depth here and i'm just touching the surface of networking and proxmox so the first thing before i set up a virtual machine is i need an image to install on in this case i'm going to take an iso file so i'm going to find that by clicking on the little arrow so i can see my thing go to local.zfs as because this is the only one i can store iso files on currently actually it's local local zfs is the one for the images on the boot drive go into iso images and then i can do upload since i already have one on my computer i'm going to just go to my downloads folder and put a ubuntu image i have on here and it's just going to upload it it's relatively quick because it's on a local network i also have the option of downloading from a url so in a lot of linux distros i can get that url and just paste it in here and it will download which is great if i don't want to upload it over a slower connection for example and now that my vm and now that my iso is done uploading i can start setting up my vm to use that so i'm going to go click on create vm in the top right hand corner um i can select the node by default it's the only one on here vmid is a number that starts at 100 and just listed on the left and is accessible in things like config files doesn't really matter what it is typically but it's just the internal name the friendly name is going to be something that you set that isn't really used internally so in this case ubuntu test resource pool is good if you're using multiple resource pools i have some systems set up that way for like permissions and stuff but by default it's blank and then this is the ucd dvd image so in this case i'm going to be using the ubuntu one you can also use a physical cd drive if your system has that or do not use any media which would probably be a way of doing like a network install or just a blank one i'm guest os type you can put some info on here from everything i've seen all this does is change some of the default devices it comes with because for better compatibility typically i leave it on whatever it is recommended but i will change some of those devices later on based off what i need i'm going to be doing a linux one and linux generally supports this kvm virtualization a lot better out of the box i'm going to do a windows install a little bit because it's slightly different graphics card i leave by default mostly um by default it selects the most compatible um the default bios you can select between c bios and ovmf ovms supports uefi boot which some os is like now and want uh they're probably going to switch default someday because a lot of os isn't everything switching to uefi boot but in this case i'm just going to use the bios boot and then that's it and you also have options for like tpms and stuff because some os is like windows 11 are getting quite picky now so now for storage devices for this vm on the left we see all the different storage devices just like a real computer you can add as many hard drives or virtual disks here as you'd like and then let's talk about the options for each drive so the first thing is bus slash device and this is kind of the virtual driver and type of disk it presents because just like in the real world where there's multiple different types of disks there also is in the vms so there's ide sata io block and scuzzy and you should almost always use vio block because it gives you the best performance and just generally works the best if i o block doesn't work scuzzy is almost as fast when it comes to performance and then sata and then ide the only times i'd use things like ide or seder is if i'm running an old legacy vm that doesn't have the modern i o driver so something like xp or 2000 or even older or an old linux distro but i want io block here um storage is what device you want to be stored on so in this case i want to store it on my donut pool um disk size is the size you can only go up really you can't really go down easily so i can only start small so 32 gigs for linux vm is plenty and then cache so i actually just did a reasonable amount more testing with caching and zfs enabled because a lot of what seems to apply for caching that they list on their page like here doesn't really apply with zfs because a lot of this depends on if the host page cache is enabled or not and zfs kind of bypasses that in linux and forces the arc to be used and so when you use modes that say they disable the page cache it doesn't really disable the page cache because of zfs as using arc instead so essentially i found those two caching types when you have zfs and essentially the right cache is not allowed so it allows it to dump rights and ram for a little bit before it puts it on disk and then write cache isn't allowed so basically right through and direct sync will not use that right cache and make sure it goes directly on the disk before it says the write is completed otherwise it will dump it into ram say the right's done or let the vm choose depending on the application that's used and then it'll keep writing into ram a little bit before it finally goes onto disk um generally they say the preferred is none um and for my looks at it for performance leave it at none it'll dump it to disk relatively quickly so it's fairly safe but if you need to guarantee that it is enabled either set out your vms here or set sync equals enabled for zfs and you probably want to add a log device too as it significantly increases performance if you're forcing sync to be used and then the next checkmark is discard and this basically presents the os with the trim command and says it supports it and the advantage of this is then your os will kind of see this as an ssd and run the trim command every once in a while in the background and because it does that it uses significantly less space on the host so it saves file space and to my knowledge there really isn't a disadvantage of doing this so i almost always leave it on and most modern os's will work just fine with that and then there's bandwidth which lets you limit read and write speed and the burst speed i almost never do this in a home lab scenario as i don't have problems with vms and i can add more disks here if i want something to like be a file share or another disk cpus you have sockets and cores the reason i've seen them separated is because some os's are picky about socket count so for example things like windows server is picky when it comes to how many sockets you want because extra sockets cost extra money generally i leave sockets at one there can be a little bit of tuning when it comes to things like numa and if you want your vm to span multiple pneuma nodes but i'm not going to get into that today typically with a generic vm i'd give it at least two cores here so that would be up to half the performance the next thing is the type so this essentially means how many features are presented to the vm that it can use because there's the base levels of features of a cpu of all its instructions and then they keep adding them every generation basically and you can go and put in a specific generation of chips and generally with either intel or amd you can go back a generation just fine but you can't present it as a newer generation and there's often issues if you have an amd chip adding it to like a cluster of vms so what you might want to do is if you have a cluster and they're all running let's say nahallium as like a base or newer set them all to the helium and then that way you can easily transfer between the vms live if you aren't running a cluster basically always run it at host and it'll present the full features that the host has which just guarantees it'll work fine memory so by default it just gives you a level of memory that you can use so two gigs that seems fine for a basic vm and then the model of network card is either i o para virtualized this is a vertio it's the highest performance it works the best the other option is intel e1000 the main reason i'd use this is compatibility so for example the e1000 works out of the box and windows whereas the pair virtualized one doesn't and then the firewall is enabled or disabled so that would do i'm going to turn it off here just so it has full access and then confirm will show you all of the types of settings you said i'm going to click start after created which for me that automatically starts it there also is an advanced checkbox that gives you a few more settings but i'm not going to get into those today and when i said that my vm 100 is now starting up i'm going to take a look at the console and this is essentially like sitting in front of an actual system so i can go through the install here and i'm just going to do this quickly and it installs like any other physical real system would now let's talk a little bit more about setting up a windows vm because there's a few differences so i've already loaded this system with an iso of a windows 10 21 h2 but we're going to want another iso to get the most out of this vm and that's the io iso so i can just google vio iso and i should find it online i think debian's the i think the fedora people typically host it so i can see the github page for it and then i want the stable iso and the reason i want to get this as an iso file is because i want to add it as a cd-rom so then when the vm's booting it's going to see two cd drives one is the windows drive and one is this i o drive because i'm using the vio disk for boot i need to install those i o drivers before i install windows so i can't just download a driver once i'm done with it i also need these drivers for the network cards so that's another step too so this is almost done i'm going to upload the iso to proxmox now so now it's time for me to set up my windows vm so i'm going to call it windows test and i'm going to leave that i also have the advanced check now so just so i can see that i'm going to select my windows disk right now and under yes device i'm going to just select windows so 10 and it's going to show me the same options here for disks it's going to pick ide by default i want the io block for the best performance i'm going to keep the discard on cpu i'm going to give it two threads again i'm going to give it my host cpu for the best performance kind of same two gigs of memory and this is where i'm going to select e1000 because it's plug and play but i want i o so what i'm doing now is i'm setting it up with the best performance options if you leave it by default it'll just plug and play and work fine but you'll get a reasonably amount reduced performance because the ide device and the e1000 are really not optimized for the best performance in a windows vm so now i've seen it's created here and i can go under hardware and i'm going to add a cd-rom drive and i'm going to set it up with my local disk with the um i o drivers so i'm going to create that and then i can start it and then once i log into the console it should start with the um yeah windows 10 setup and it's going to take a minute to get me into the setup menu for windows 10. and i can click through all these menus as usual and when it's going to need my attention to start installing drivers is when i start getting the select disks on how i want to do that so now i'm going to select custom and i don't see any disks and that's because i haven't created i haven't loaded the drivers yet so i'm going to go browse vert i o and then it's under ios store at the very bottom windows 10 amd 64 okay and it sees that driver it's going to take a second to install the driver on the system and allow it to use the disk and then once it's finished installing that driver i'm going to see the 32 gig disk it's ready to install on right now and i can hit next and it's going to just start installing windows on there now that my windows vm has finished installing i'm at the windows desktop but i don't have access to the internet or anything else because of missing drivers so let's go take a look at that and fix that now so i'm going to run dev management which is the device manager on windows and i notice those two devices that are missing so what i do to fix those is just go update driver browse my computer includes subfolders makes it super easy so i have to actually find the actual driver point it to the cd-rom and hit next to anything that has that yellow exclamation mark and once it's found it it says it's updated my drivers and i'm going to do that with that other pcie device browse it include subfolders and in a second or two it says oh it's the i o balloon driver which is what allows it to dynamically change memory so now i have both of those devices installed and i have network access on the system here and i can access the internet on here so now i've installed all the drivers for my windows vm and it's ready to be used like any other system and since it has the i o drivers it's going to be running in its highest performance mode it can another thing to take a look at is how proxmox shows you data about what's going on internally so if you take a look at the summary tab for vm or the host it's going to show you quite a few little cool graphs and stuff of what's going on in the usage so i can see cpu usage i o delay ksm sharing swap usage that and let's go over a little bit of what this means right now so cpu usage is your simple how many coils are being used when load average is the linux um 1 5 10 1 5 15 minute load averages and this takes a look into things like i o weight general rule of thumb is you want this number to be less than the number of threads your cpu has so in this case with the four core i want it to be less than four so since i have vms going it's pretty high but if this is a consistent like 4.1 i'd probably want to think about upgrading my system i o delays are taking a look at how much i o is causing basically weight and how much time the cpu is spinning waiting for i o if this is really high you're probably i o limited and want to think about either optimizing your i o or upgrading your disks ksm sharing is something that proxmox uses to save memory usage on linux vms by kind of merging all the kernel memory together basically it says how much ram it's saving using that ram space usage it shows i'm using 93 of the ram the fun part about this with zfs is i'm not really using 93 of the ram used a lot of it's being used by the zfs cache which counts as used and it'll free it up as needed you have hard drive space which looks like it's on the boot drive and is almost full you can also view hard drive space on the left here and i can see donut has some usage everything else is almost empty and then swap usage which i don't have swap and swap isn't enabled by default i've looked into using proxmox with swap i believe in the past and it can be pretty useful for like a home lab it can also hurt performance so it really depends on do you need the best performance just make sure you have enough real ram if you want to be able to save money on ram or just can't add enough ram adding a high speed swap device can be a good solution and then you get graphs of a lot of these things so you can see like my server load is shot up once i started installing oss same with cpu usage and i can also see my memory usage was idling super high which is that zfs can use up to half the memory thing i also have a graph of network traffic and then clicking on each of the vms under the summary tab i get a very similar view just with a bit less information as it is showing the actual system here not the full one so i can see memory usage of the vm cpu usage network traffic disk io and then i can change in the top for how many hours days this vm hasn't been out very long so it doesn't even show up on a day but it'll just show me info for each system another thing i wanted to talk about a little bit is snapshots so snapshots is a feature where you can take simply take a vm and pause it in its time and place and just keep it there for later so in this case i'm going to be using this debian disk test vm i created and just create a snapshot of it so i'm going to take a snapshot and we'll just call it test1 include ram which means i have the full running state of the vm and it takes it now this is going to be super fast and work pretty well because it's using zfs for snapshots so it's using the ram snapshot which takes a little bit longer to store all the ram on the disk essentially but now it's created a snapshot so if i want to go back in time i can now select my snapshot hit roll back and now that vm is going to get rolled back to that snapshot i made a couple seconds ago and every change that happened since then which essentially is none is going to get erased and i can make a whole chain of these snapshots and zfs is super nice also if i want to save space i can hit remove and it's going to delete that snapshot so it's no longer on the disk this can be super handy if you're doing things like an update and you're like maybe it's going to break something if you're doing like a windows 10 to windows 11 update or something like that well you think there's gonna be a lot of issues take a snapshot first let it do its update if everything's good you can delete the snapshot if there's issues you can roll back to the snapshot and kind of work through those issues now and you have a super easy fallback plan if there's an issue of doing something that's risky now let's talk about backups which is kind of your more continual there's probably a risk but i want to make sure it always has another copy proxmox has had the old way of doing backups for a while and in my opinion it's really not great and you should probably using pbs pbs does a lot better when it comes to backups and uses proxmox backup server to do it all and you can actually install it on this host or even in a vm on the host and have it store all your backups somewhere and it does things like incremental backups super well this does not do incremental backups it only does full ones so it takes a lot of space it always takes a while because it has to read the whole disk to do it but i will show you the simple part basically you can say i want to do a local which is my my disk i can store it on i want to do a snapshot so you can either stop suspend or snapshot the vm compression you should almost always be using zstd now and hit backup and it's essentially going to take all the vm data squish it into proxmox's format they actually have a way of reading it so it's not a super proprietary format and then when you want it back you can say hey restore from that file can be useful if you want to like back everything up move them to a new server and restore it but generally use pbs to do your backups you can also automate your backups in the data center tab on here but really use pbs to automate your backups one other tab for vms i wanted to look at was the options tab so the options tab goes to a few more of the advanced settings you don't typically change and some of them that i think are useful to look at is start at boot so if you want to make it so that this vm starts when the system turns on you can turn this on and that'll auto start services so i'd highly recommend doing that if you want if you have services you need once it's immediately powered on and it makes it easy so if there's a power outage everything stays going also if there's a reboot it also turn it on then too start shutdown order is the priority of how if it has to run after something else um boot order there's just like a boot menu and order normally these you might want to play with use tablet for pointer this actually uses a reasonable amount of cpu so if you have a system that doesn't have a gui you probably want to turn this off and i've seen people lower sometimes in the double digit cpu percentage normally in like the medium single percentage if they have a lot of linux vms with no gui basically this is what allows you to have a virtual tablet that puts the mouse pointer through that makes it work correctly but in this case i don't need it because it's a linux vm that doesn't have a gui so i could now that i've gone over a good amount with virtual machines let's take a look at containers so containers are a little bit different than virtual machines because they're a lot more integrated with the system and they have a lot less of their separate parts so since the vm is a full like virtual computer there's a lot of stuff like drivers and controllers and other hardware that essentially gets duplicated it needs to be stored in ram it takes a few cpu cycles and has some overhead that you really wouldn't need and want to have at all times so containers get rid of a lot of that they're not technically running a separate kernel so it's more like a separated way to run linux programs in its own full environment and the advantage of this is it gives you a lot less overhead so you the basis of a container is much less storage and much less memory usage and performance should be a bit better too so it's a lot closer to running it like a program on the host versus a virtual machine but you still get a lot of benefits of running a program in a virtual machine like being able to have its own ip address having a full separate you can't see the host processes for example so it's still quite separated and it's still a level separation that if a user had access to it they couldn't really break out into the main system assuming it's all set up correctly now the disadvantages is it has to use that same kernel version so it limits you to being able to use that same kernel version of other linux versions you get different linux distros where they do their own updating and different things but they're still all running that same kernel under the hood and also because it's the same kernel and because it doesn't have full access a lot of things that require stuff like kernel modules won't work so some services and servers won't work if they need very tight integration of the kernel what i typically like to do if i'm running a one-off thing is just see if it works in container and if it does that's great because it's lower overhead and if it doesn't fire up a vm because it's going to certainly work there because i can do whatever i want in that vm the other thing is stuff like windows open bsd freebsd mac os other types of vms and os's won't work in a container because it's a different os type and all so it won't even work and now with that out of the way about the basics of containers let's look into creating one so i'm going to go under the local tab and state look at the cm templates so just kind of like i have an iso image that makes a vm you have a container template that gets turned into a container so what i can actually do is they include all of these for me so these are just different linux distros so i could download let's say like debian 11 standard and start downloading it and it'll download it from the proxmox repositories pull it onto my system and i will have access to that once it's finished downloading and also in addition to all like the standard linux stressors they have stuff like these turnkey containers which is a basic linux distro and then a lot of pre-made stuff so sometimes if i want to do a basic file share i can use like the turnkey smb share one container which has a little web gui that makes it super easy for me to make a smb network share out of a container one thing i have seen in the past is sometimes you don't see all these containers it seems to be some sort of automatic updating thing if you don't see it i say just leave the system running for a day and normally i've always seen these come up after about a day of running and checking at stuff and now the container template's finished downloading so i can go and create that container now so i can click the little crate ct button and i'm going to get a few more options than i would for vm because it's a bit more tightly integrated with the main system so i can do things like setting the password and ip address when i create it so i'm going to set a very basic password i'm going to say it like contest and then we have the option for the unprivileged container and this you want to leave on by default because what it does is it lets it run as a non-admin user in case you broke out of the container somehow you're a non-admin user on that host system and that means you really can't do as much damage on that host now um and you really can't access as much you can't break other vms you can still do some nasty stuff but not as much as if you unclicked it game made it a privileged container and then now you're a root admin user and you can do whatever you want to the rest of the vms if you happen to break out it's not a super likely issue and hopefully it won't happen but generally you don't want to check it now you can't always have it checked because it's not it can cause some issues and sometimes vms don't like it when it's unprivileged but you have to try to see it if you can do it without so i'm going to pick my template i downloaded disks are the virtual disks you want to create it's a little bit simpler we're going to make a little eight gig one the great thing of containers is they're super tiny i'm going to stick with one cpu memory i can do something super small network is where it gives me the option so i'm going to actually just do dhcp right now and let it grab a dhcp address i'm going to use host dns settings and then i'm going to confirm so this is a super simple vm that's going to be created and i can use it now one thing to note with a lot of these distros is you're going to actually have to log in via the proxmox ui and you can't just ssh in directly because it's only created the root user and a lot of linux systems don't let you ssh in via root so that's one thing you have to set up and look at but my container test is running so if i log in here i can see i have a little container so i can log in with root and my password i created and it looks like another system now the thing i'm going to notice is this is debian 11 debian 11 does not normally use kernel 5.13 pve that's the host kernel it's using and if i run like lsb ok it's a bit weird because i can see all my block devices but i can't really access them so if i do like df dash h i only see my actual ones and if i cd like slash donut or something where it would be i can't access it so it's kind of this weird mix of like all the hostile hardware and a little bit of the virtual stuff but it runs stuff super fast and it does it pretty well so for example i wanted to install apache so i could do apt install like apache 2 i get the full apache server this is great for something like a web server a little database a basic like smb file server something like that this type of container would work really well for and i'd recommend using container if you can use it because it's lower overhead less resource utilization and just i'd say easier to set up and use because it's managed more by the host and you can also do things like ssh keys through the host instead of having to manage them per vm so now that the container has been created let's take a look at some of the options proxmox gives us for this container so the first thing is i get a summary tab and this looks very similar to the vm ones with a little bit different options and one of the huge advantages of containers i can see right here which is their memory usage in this case i'm only using 23 megabytes of ram for an idle debian container which is much less than a typical vm would use which might be in a couple hundred meg range i'm also using much less storage of only 340 megs used and also cpu usage is essentially a rounding error at zero so that's a super low resource utilization versus a vm which is nice here because it's almost just only the resources your actual application needs in the container the container's super low overhead i'm taking a look a few more of the tabs you get the console which is just a shell so i can just run like lsb okay or whatever commands i want in here resources are essentially like the virtual devices it gets i can add a mount point which is essentially a virtual disk but instead of having a virtual disk where it has to make like a virtual block device and then put your files in that with the vm it is just a file share essentially that the container can access directly so a bit less overhead works well um and then i can access like network settings so you actually have full network settings here and like a vm where you just give it a virtual network card um options there's a few things you can change like start at boo and the order that they start up but most of these things like architecture os type unprivileged container and others are set during creation or permanently part of it you have to make a new container if you want to change those um you can back it up just like how you would back up a vm i'd still suggest using pbs here but it'll essentially make a little tar file that has your vm in it and some of the config files and stuff so you can recreate it on another proxmox host and i just made a backup job super small like 250 megs for this backup here i can do snapshots the same way so i can easily take a snapshot of it call it like test for example and make a snapshot of the system i have my same firewall options and then permissions of who can access it so containers are pretty simple they work in a lot of ways very similar to vms but there's a few kind of weird oddities i'd say if you're trying an application try it on a container sometimes you run into weird arrows if you google them people are like oh it's a container so may just make a vm for that use case but i like to run containers with things like basic web servers because it's just less overhead thanks for watching my little proxmox ve tips tricks and user guide this is based off my experience of proxmox and while i tried to cover a good amount in this long video so now that so now that the container has been created let's take a look at some of the options proxmox gives us for this container so the first thing is i get a summary tab and this looks very similar to the vm ones with a little bit different options and one of the huge advantages of containers i can see right here which is their memory usage in this case i'm only using 23 megabytes of ram for an idle debian container which is much less than a typical vm would use which might be in a couple hundred meg range i'm also using much less storage for only 340 megs used and also cpu usage is essentially a rounding error at zero so that's a super low resource utilization versus a vm which is nice here because it's almost just only the resources your actual application needs in the container the containers super low overhead i'm taking a look a few more of the tabs you get the console which is just a shell so i can just run like lsb okay or whatever commands i want in here resources are essentially like the virtual devices it gets i can add a mount point which is essentially a virtual disk but instead of having a virtual disk where it has to make like a virtual block device and then put your files in that with the vm it is just a file share essentially that the container can access directly so a bit less overhead works well um and then i can access like network settings so you actually have full network settings here and like the vm where you just give it a virtual network card um options there's a few things you can change like start at boot and um the order that they start up but most of these things like architecture os type unprivileged container others are set during creation or permanently part of it you have to make a new container if you want to change those you can back it up just like how you would back up a vm i'd still suggest using pbs here but it'll essentially make a little tar file that has your vm in it and some of the config files and stuff so you can recreate it on another proxmox host and also and i just made a backup job super small like 250 megs for this backup here i can do snapshots the same way so i can easily take a snapshot of it call it like test for example and make a snapshot of the system i have my same firewall options and then permissions of who can access it so containers are pretty simple they work in a lot of ways very similar to vms but there's a few kind of weird oddities i'd say if you're trying an application try it on a container sometimes you run into weird arrows if you google them people are like yeah that's containers so may just make a vm for that use case but i like to run containers with things like basic web servers because it's just less overhead you

Channel: ElectronicsWizardry

Views: 92,751

Keywords:

Id: Ce0uwBxbVRQ

Length: 66min 36sec (3996 seconds)

Published: Sat Jan 29 2022