10Gbe Network Setup for Storage & Virtualization using UnIFI US-16-XG / Chelsio / FreeNAS / XCP-NG

so I did my RAC toward the other day and people that asked the butch cars I was using for all my 10 gigabit connectivity because I'm using in both the XenServer and FreeNAS so these are Chelsea Chelsea oh I'm not sure exactly how to say that Chelsea's 10-gig e320 cards this is an older model card and it works great I've actually bought a handful of these off of ebay one in each of my servers now I bought dual sfp+ cards now SFP and SFSP plus are very similar as in they physically look the same SFP being the one gig SFP module standard and SFP+ is a 10 gig standard there is some interoperability because you can slow this down to one gig on the connection as opposed to 10 gig but all of the equipment we have including the unify 16 G 10 gig switch is going to be all connected at 10 I will know because someone had asked about our unified 24-port switch and that only has SFP not sfp+ on it the one that we have in particular I think all the unified 24 is you don't get sfp+ on the unify 24 switches but you do get it on the 48 so which is so I figure to cover that real quick now we've chose deck pools as an option for connecting these these are direct attach copper twinax they make cable management fairly easy they're kind of designed for either in rack or you can go if you have an adjacent rack they're running them over there are different than the fibre I mean the fibre is popular if you want to do long runs and you can get a fiber transceiver that fits into SFP and will work and I have tested this generic one here and unify sells their own fiber in tacacs so if you need to do a longer run fibers a little bit harder to work with it doesn't quite Bend as well it does Bend really well actually it just will break easier and it can be a little bit more troublesome to try to set all this up for cable management so we're gonna stick with the deck now buying duel cards the reason I buy duel cards is if you have in case of Zen servers of e/m if you have your storage network which we do plugged in and you then want it a virtual interface that also has a 10 gig connection you could assign the storage here and the virtual interface for your virtual machines over here that way you don't have any conflicts with the Sorge but in that example would be if you're pulling a heavy i/o on your storage and then you would like to serve that over SMB a 10 gig via the virtual machine let's say a Windows server that's running on here you'd want to have dedicated NIC if not they would share the bandwidth now for our use case we only have a single one in use I don't really have a need right now for my VMs themselves to have a 10 gig interface but it's there and these cards like I said are reasonably priced now the direct attached copper is pretty easy to install whoops it's just like USB you'll get it wrong the first time it snaps in and clicks mmm now a couple options here if you are setting this up in your home lab and you would say you know I just want to have two servers talking to each other no problem go ahead and snap two together you can have two cards direct attached to each other and it works perfectly fine you two statically assign the addresses and they can talk to each other without having a switch in between this is a really convenient way to do it especially when you're doing testing and when I build my test stations here on the desk I do usually set them up this way it's just really convenient and it saves you the expense of a switch but in our case because we have two free now servers and two Zen servers and we have a storage network we built which I'm going to show you here in a second we like to keep them all plugged into a switch because that gives us really fast transfers between anything that we want to do across that network on and isolated all right so let's take a look at the physical layer now on our network and then we'll get into the software okay we'll start here at the front so here's our unify 16 port 10 gig switch and here is the SFP cables that we have connected there and these go to the back of the server now this is the SFP ports on here but please note there's no Plus next to them so we've connected the rj45 style from port 19 to port 13 and that's only going to connect to at one gigabit which is why show the camera that's green so that only shows we have a one gig connection and all the white lines so we have a ten gig connection here those lines all come back this way and down and now they're plugged in over here so here is the 10 gig interconnects that we have on each of these and like I said we have the duel cards those Chelsie oe3 20s and it carries on down to each server and place it we have a spare if we need to so pretty straightforward and you can see the active light when they're plugged in and know this would be the light for this site over here now the nice thing about these and we'll get a wider view of my rack here for cable management they do have a decent Bend radius so when the door closes it touches these a little bit but it doesn't exceed the venn radius on these it would be a worry if I had a fiber module and a fiber sticking out and then a bend down all right we're going to start here with in the unify part so with unify I love the way that their software makes the layout look exactly like the switch it's just convenient and pretty so we can see that all these are connected at 10 gig via SFP just like we seen when we're looking at the rack here is the air rectangle quiche wondering what this one is that is also an rj45 10 gig and here is that one purple cable that connects from the unified 24 port to here for the uplink that can bridges this into the rest of the network and it's only at gigabit because well the 24 ports only eight gigabit switch now these being the storage network they're on their own VLAN and by doing that that means we have done this any locked down a setup essentially so if we look at it and let's actually go jump over here to the settings networks move this over have our storage VLAN only and it's set at VLAN ID 20 and then we go over here to any port that we edit and we've all chose this as a profile to be 20 now this network is not defined in my router and a reason why is because I want to keep the storage network isolated from the rest of the network that way anything in there is locked down to only those servers there's no other interference there's no routing going back and forth or in and out of that network it is essentially a flat network now by doing that and not having even a DHCP server in there and it does mean we have to statically assign ease which is pretty easy to do now the one config change that I did make in the switch itself is under services enable jumbo frames jumbo frames you're not familiar jumbo frames you can look them up and get way more detail that I'll give you right now but the short of it is the Ethernet frames are set to 1500 by default a jumbo frame carries 9000 bytes of payload but you do have to have a switch that supports it and it does add some speed to there because for each packet you're able to get more data squeezed into it so if you look at the frame type here as a standard at 1500 here's nine and overall you can see the payload side total transmitted efficiency is a little bit more in real-world performance yes you get a little bit of gain on here and it varies by protocol but generally speaking if you're on a network that supports it it will give you a little bit of enhancement on there there's plenty of debate that I'm not going to get into about how much enhancement in the real world testing for it it's easy to do it doesn't have a cost to it there's not really a downside as long as your switch supports it so we have any enable jumbo frames it's the only real change that we've done on this switch outside of defaults so once that's enabled now we're gonna go to how we have these all statically assigned now by the way because these are assigned only that VLAN I don't have to tag anything within the network interface itself so you don't when you're configuring these in FreeNAS or in Zen server you don't have to make any changes or add any VLAN tags to these because it understand you do that together so those are all focused only on that VLAN no other VLANs information is passed across so we have here 1 9 2 1 6 8 10 10 on this FreeNAS server and let's go ahead and look at some of the interface exactly and we're gonna go ahead and edit one 92168 1010 I set them to be slash 24 I've covered this before and someone says well you're wasting IP space and why I don't know I shall say the things they say anyways you can set it to whatever you want the only thing on here are these four servers so you could make it the net mass smaller habit is slash 24 in case I had a couple more devices to the storage network it's easy enough to add them to you just got to remember I'm statically assigning all of these then MTU 9000 this is how you pass that option on to get the jumbo frames so I set this to m2u 9000 inside of FreeNAS and we'll look at the new freenas same thing 10-gig we're going to go here and edit here's a static assignment / 24 they do all have to match the net mass or whatever Minette mass you go with just make them all the same across all the machines are statically assigning and MTU 9000 now Xen orchestra is a way to do this when you're looking at the XenServer is just a couple options here so we're gonna look at Network and I've got it passed over as storage network m2 9000 I'm gonna show you a different way because the way I find it a little bit easier I love Xen orchestra when it comes to networking if you're not clear on how to use it there's another way you can assign these so if you open up the XC PNG Center you can see all your networks and then you go down here to configure and this is how you can configure your storage network as an extra network so it's nice and separated here if you're doing it in the XC BNC and want to go back to the darkish in a second show you how do you do that but if you notice what I'm assigning is this is 10.15 but has no gateway because like I said we don't want any routing going in there now on our side note when you're doing this once you've done this if you want to and we're gonna head and properties of this you notice how the MTU is grayed out the MTU is grayed out because I have it assigned down here this question has come up before you want to set the em to you before it because once this has been assigned to the storage network and is using it as a storage Network then it doesn't allow you to edit the MTU just kind of a side note but if you're wondering about that that's something that's important to note and the same thing we're going down here is enough for two networking same thing storage Network this one's 10.20 so 10.15 10.20 and it allows you then to do the same things configure no gateway and all these of course can now see each other and when you're connecting to things like the freenas storage you do it by IP you just type into one 92168 10 dot and then go from there now back over to the Zen orchestra so if you go over to your hosts and go to network you see all the networks and you can add a network but where you want to go when you're setting up the networks is you go to pools full network this is where you can create like the bonded networks and you'll see the options are very slightly different this is what it allow this is where you can set the Mt use and configure some of those things so you can add this you can kind of get the idea that they're a little bit different I guess then senator seems to be a bit easier to read when you're building a storage dedicated network first is in here because there's two spots you can go for networking just want to leave that note in there so you have two options and whichever way you choose to configure it you can easily change it in either one after the fact inside of here but only when the storage networks disconnected can you change it his terms of the MTU like I said easy enough to pass the MTU setting now one of the cool things is once it's on here and I grab this little demo and we have the disk is dosser right here and that is our FreeNAS this one here over here to the pools to show you what it looks like storage pools these are just a bunch of spinning drives a bunch of older - 2 terabytes ass tries but they're reasonably fast and I'm bringing this up and showing you how they're connected so here it is and VMC's are connected via NFS and the Tang gig network so what kind of speed to get out of this I'll just run this quick speed test the most accurate I'm not getting real in depth here but over the 10 gig network it's reasonably fast I mean we're getting a right speed of about 586 Meg's now what command that I would do to run this speed tests SSH it's this right here it's time SH to see rates a bunch of zeroes test file 1.9 gigs count 1000 then it syncs the file system and then it removes the file and that gives us like I said a speed of 586 I've done some of the benchmarks on this it's it's reasonably fast it's not quite as fast as a Samsung Evo SSD but the speeds are pretty reasonable on it over the 10 gig network it's it's not bad of course it goes faster if I ever get some money to throw at a storage array that is all flash it would of course increase that much because we do we have plenty of speed going between these one other thing that's kind of neat is when you do a migration using Zen Orchestra so an orchestra is aware so we're going to choose it's on Zen affirm whereupon is enter for to the other one I want to send it over to local storage it sees not only the pool wide area eath one so it says hey these servers talk to each other and eath one it realizes they also can talk across a link that I just named a 10 gig storage so it's the storage network but instead of storage this because the sensors are on that network they can talk to each other and this will allow me to migrate VMs it only takes a couple of minutes to migrate even a live one because they're talking to each other at 10 gigabit through the switch so that's not an advantage you get it's not only that you have fast access to the storage repositories that are stored on a 10 gig network the 10 gig interconnects between the Zen servers allow me to easily pass them right over at full speed now last thing I'll show you is you know people want to know is the speed test between the servers and we'll cover that real quick here so we'll go ahead and login to one of the FreeNAS boxes we're gonna run iperf - ask for server - B capital B one 92168 10.8 and what this does is binds it to that IP address so the servers do talk on other networks we ought to make sure they're only talking across the 10 gig network this is T MUX for those wondering it's how I split the screen now we logged in as ennifer and we're going to do iperf three client one 92168 10.8 and we can see that they're able to transfer it quite fast at nine point nine gigs a second so we're saturating the network between these two and each send server can talk to the others end server at this speed and each sent server can talk to the free-dance at this speed or each of the free Nasus because we have to free announces and the to Oz answer is under so I'm able to move data around quite fast through this switch at 10 gig and of course if I because these are all duel cards if I ever wanted to have the VMS have an extra network interface go back over here to hosts networking network I labeled them all is not in use but this is the eath for historic 10g that's not in use is actually the not plugged in right now other network interface that is available on there so if I ever wanted to have the VMS assigned to it it's pretty easy to do so hopefully it clears up or gives some insight into how the 10 gigabit network is connected for my network and how to set it up it guess it's pretty straightforward these stacked cables are reasonably priced I'll leave links where you can buy these from you can get these cards out I'll put the model the card in there that way it stays relevant I can't really link directly to an ebay link because well they're only relevant for short periods of time but as long as you know the model of the card I've had no compatibility problems with bringing these in free nafs or XenServer and that includes in the freenas 11 one and in freenas 11 2 they're all the same card in all of these now you can mix and match cards and I have tested it with a couple other cards that I have and I didn't see any difference in speed or performance on there when I was doing kind of some lab setup these cards happen to be both low profile and full height so you can get them at either one with different brackets so please note when you're getting them get what matches your server but the nice thing is the card itself does come in different formats thanks for watching if you liked this video go ahead and click the thumbs up leave us some feedback below to let us know any details what you liked and didn't like as well because we love hearing a feedback or if you just want to say thanks leave a comment if you wanted to be notified of new videos as they come out go ahead and subscribe and the bell icon outlets you tube know that you're interested in notifications hopefully they send them as we've learned with YouTube anyways if you want to contract us for consulting services you go ahead and hit Lauren systems comm and you can reach out to us for all the projects that we can do and help you we work with a lot of small businesses IT companies even some large companies and you can farm different work out to us or just hire us as a consultant to help design your network also if you want to help the channel other ways we have a patreon we have affiliate links you'll find them in a description you'll also find recommendations to other affiliate links and things you can sign up for on Laurens systems comm once again thanks for watching and I'll see you in the next video

Channel: Lawrence Systems

Views: 75,990

Keywords: Chelsio N320E, chelsio n320es, chelsio 10gbe, chelsio 10gben, chelsio, 10gbe, nas, 10gb, ethernet, xenserver, xenserver tutorial, xenserver 10gbe, xen server, server, Freenas 10gbe, freenas 10gbe setup, freenas 10gbe nic, freenas 10gbe performance, freenas 10gbe card, storage 10gbe, nas server, install 10gbe, 10gbe setup, upgrade to 10gbe, virtualization 10gbe, switch

Id: RWf3klyvPts

Length: 18min 20sec (1100 seconds)

Published: Tue Nov 13 2018