Mellanox ConnectX-4 100Gb NIC, Firmware Update and Ethernet Mode

hey guys I have here a melanox connectex 4 100 gigabit network interface card a hundred gigabits this is model number mcx455a you can see there it's got a single Q sfp28 Port it's a pci3 x16 bus size now this is a dual purpose card it has two modes you can either use it for infiniband or standard ethernet we'll be using this for standard ethernet in some upcoming videos in today's video I'm going to show you how to update the firmware on this controller I'm also going to show you how to switch it from infiniband to ethernet mode all right so I've got a session open to my server here and there are three pages here on the Nvidia website we'll be referencing throughout this video and video does own a melanox for those of you who aren't aware the first page is a getting started which provides most of the commands we'll be using the second page is the firmware tools download and the third page is the firmware download I will link to all three of these Pages down in the video description as I realize the font on these particular websites is very tiny and probably difficult to read in the video so the first thing we'll do is install a few supporting packages at minimum you'll want Colonel devel you'll also want PCI utils you shouldn't need USB utils however it will prevent a warning message from being displayed when you run some of the commands and I didn't want to see that warning and then lib IB verbs is for the infiniband options as you see I have all of these packages installed already so next we'll verify that our card is indeed installed and detected we'll do an lspci grep melanox helps if you spell it correctly and you see it's labeled as an infiniband controller but it says the connect x-4 so we're going to head over to the web browser in our firmware tools the mft and download that application and you see the latest version here is 4.22.1 it is an LTS or a long-term support release so we're on Linux we're on RPM this is Oracle Linux or Red Hat Linux x64 I'm going to copy that link address we're going to do a wget and post that URL next we can do tar xvf to extract it mft okay CD into that directory so you'll see there is an install.sh script all we have to do is run that dot slash install it's going to go through a number of checks and we'll also let you know if you're missing any supporting packages it does install a kernel module as well so once you are completed with your configuration if you don't want that kernel module you see there is an install.sh script you can run as well get out of that we're going to go back to our web browser here and now we are on the firmware downloads page so you'll see the latest version is 12.28.2006. this card is the mcx455a dash ecat if you're not sure that part number should be printed on your card directly on the back of the PCB so right click copy link again wget and we'll extract that the same way that's a zip file so we can use unzip so now if we do an LL you can see we have our firmware file and we have our firmware tools so first we're going to update the firmware and then we'll worry about changing it from infiniband to ethernet mode so to begin we'll run MST start it's going to detect your cards and then we can run MST status to show what it's found and you can see we have one card loaded here so first we're going to check the version of the firmware that's already loaded and that's done using the Flint command Dash D and you'll give it the path to the device as discovered above and then query and you can see we have version 12.21.2017 and that was released on November 27th of 2017. so the version we have now is quite a bit newer so next we're going to do the firmware update we'll use the Flint command again Dash D give it the path name Dash I and then your firmware file and then the word burn is the word that's going to write it to the card and you can see again the current version is 12 21 2010 and the new version we're flashing is 12 28 2006. these are firmware numbers they are not dates even though they do look like dates so keep that in mind so once that firmware is complete it does say it wants a reboot so we'll just do a quick reboot all right you can see our server is back online here so we'll make sure that firmware did save correctly we'll do an MST start again MST status and we'll do a flint Dash D device name and query and now you can see we are running 12.28.2006 which was released on May 15 2020. so the next step is you want to change this from infiniband mode to ethernet mode so if I do an ipaddr you'll see we have eno one through Eno 4 which are the onboard one gig and then we have the two enp4s interfaces which are from the 10 gig card we do not have an interface listed here for the 100 gig card and that is done using the mlx config command so Dash D device name and then query so these are all of the configuration options we can change on that card we're only interested in one in particular we want the one that says link underscore type so we'll do a grep on that and you'll see link type P1 is set to IB or infiniband P1 means Port one if you have one of the cards with two ports you'll have a P1 and a P2 and you can configure those both to be the same or you can have one be ethernet and one be infiniband it's pretty much up to you so in order to make the change we'll do mlx config Dash D device name again and we'll do a set link type P1 equals and you'll see this says IB or infiniband and then it's got a one in parentheses so mode one is infiniband and mode two is ethernet so we'll do link type P1 equals two and it will ask you to confirm you're setting link type P1 to the new value of eth or two which is ethernet apply the configuration change yes please and then we'll reboot once again all right and our server has begun responding again so now we can reconnect so we'll verify it saved that change correctly with MST start MST status MST config Dash D device name again and query grep link underscore type oops mlx config what am I doing mlx config and it says link type P1 is eth2 or ethernet so we can do clear and we should be able to do ipaddr and now we see our new network interface here enp130 so if we do an eth tool and we can see all of these supported link modes this card has it goes all the way up to 100 gigabits so it looks like the modes that are supported are 110 40 25 50 and 100 gigabit modes that is quite a lot of bandwidth so and you see it is not actually connected that is a separate topic for another day but we have our firmware updated and we have it successfully running in Ethernet mode all right so that was actually pretty easy like I said earlier I'm going to leave links to all three of those resource Pages down in the video description and that is the getting started guide the firmware update tools and the firmware file itself if you have any questions or comments please leave those down below hit that like button before you go and thanks for watching

Channel: Home SysAdmin

Views: 12,542

Keywords: mellanox, connectx-4, infiniband, ethernet mode, mellanox firmware, connectx-4 firmware, firmware, connectx-4 vs connectx-5, 100gbe

Id: D1qN7Qg3bSg

Length: 7min 30sec (450 seconds)

Published: Fri Dec 30 2022