Getting Past Incubation: DPDK for Windows Platform Goes Mainstream

good morning everyone hello and welcome to this session i am harini ramakrishnan i am a program manager at microsoft driving network data plane initiatives and particularly dpdk for windows support um now this session today is a join session uh from among the multiple partners that are working to bring windows platform support to dpdk we have uh i'm representing team microsoft and we have partners from uh mellanox or nvidia and intel joining us this morning and behind the scenes we have tons of other other contributors that we are representing here today as well so we can get started momentarily so the agenda today is we're going to talk about briefly the motivation and the journey that we've had so far uh and where we are with respect to windows platform support um and then we'll go through some demos and you know talk about our challenges roadmap and we will stop with q a uh now let's take a moment uh for to talk about the motivation now windows developers have expressed a need for a high throughput a low latency data path for their workloads now here in orange is a quote from a customer which kind of sums up the sentiment that we have received over the past couple of years which is to say that they've tried about uh all inbox inbox solutions that come on windows including rio sockets window sockets etc and they spend a fair amount of time uh you know going through the technology and uh you know familiarizing themselves and benchmarking it and the net results seems to be that you kind of don't get um dpdk-like performance which seems desirable from the community we have heard from our partners developers in the ecosystem about the need to have a kernel and a network stack bypass solution like dpdk not just that uh the windows networking stack uh you know has improved its cycles per byte uh you know optimizations right from windows server 2016 to 2019 and then from 2019 to 2022 but we are kind of reaching a point of diminishing returns for all the investments that we are making uh the performance improvements that we are seeing are not comparable to the network stack bypass technologies and that is one of the reasons and the motivations why uh we want to make dpdk a viable solution on uh windows platform um not just that uh the windows networking stack is at the cusp of multiple transformations at this time uh for future proofing architecture um and the crux at the crux of all of that is the dpdk support so you will start to see many solutions come up even first party solutions that come on top of dpdk and using dpdk so that's one of the reasons why we are uh wanting to make sure that dpdk support on windows is uh prime time and mainstream now um moving on uh so i want to quickly recap the journey that we've had so far uh back in 2017 we launched a dpdk for for windows support it was a it was a disconnected repository it had a separate development in building pipeline a trail behind upstream community uh maintained repository and uh was uh paint to maintain um and we talked about it multiple times at the 2017 2018 and the 2019 summit then in 2019 the upstreaming work for windows platform support into the main dpdk repository began uh and ever since then uh the contributions have come up and uh we are the upstream con upstream um support uh has accelerated uh microsoft uh us as a team joined both as a contributor and as a sponsor back in 2020 2020 and since then we have seen the community expand with uh not just intel who were who was our primary contributor early on now we had partners and our friends from nvidia we have uh you know as cisco marvel and multiple subject matter experts from the wealth of the dpdk community so far and independent software vendors such as maya data reader path and many more that have joined us in this effort to accelerate windows platform support upstream um so much so that we also have a storage platform development kit for windows that is based on dpdk uh forking off and available today um i will leave some slides in here for any of you that will be interested to see um the progress that has been done with spdk in the future um moving on um now i'll quickly take a moment to recap where we are as of 21.05 release that came out in may um now we have about 15 odd code libraries that are validated to run with windows uh and not just that we have the kernel mode drivers net uio and wordfest that support windows today and we have hardware pull mode drivers and software pull mode drivers uh including uh melanox five um melanox four and above um you know i40e ispmd and the software pmds that we have pcapp and vm net uh vmx net three um we have we have multiple hardware um you know vendors uh that support uh dpdk for windows today um and we have multiple sample applications that you can test out today notably l2 forward a subset of test pmd also works today um in terms of the tool chain we have uh mason plus ninja tool chain and the clan ming gw um compiler as well uh we are also a com we also have compilation gates at the unh lab um just with clang and mingw in terms of compilation uh we expect to add unit tests and dts tests soon that's on the roadmap um so now i'll pass it on to dimitri k for taking it forward from here dimitri yeah please follow karina yeah hello my name is luc i've been the windows community maintainer uh for dpdk from the past 2020. uh i'm going to talk about some challenges that we faced when importing the core dpdk libraries mainly el to windows there were there were many of them but i'll try to focus on those that most vividly show the nature of windows specifics the first one would be uh thread local storage variables defined in shared shared libraries the dlls on windows the problem with them is that a portable executable format used in on windows doesn't have a special section for such variables so basically you cannot use them without some support from the compiler and the difference will change solve this problem differently clank only works with such variables within single application within single binary and it doesn't work with dlls at all what's most embarrassing it compiles such apps but runs them with errors and mingw which is basically a gcc port for windows uh use simulation for local storage it comes with some performance penalty of course and then there are some issues with exporting those variables but at least it provides a working solution you can find more if you follow the link to the demo code if you want to learn the subject and i'm also glad to say that microsoft is working on a solution that will work for all tool chains and next slide harini the second interesting point was alarm support alarms are treated as a kind of software interrupts in dpdk the model is that we have an interrupt thread that waits for alarm to expire and executes a callback and seemingly windows does have all the facilities to implement that it has a function to set waitable timer it has functions to make an alertable wait for that when the timer expires it will call the callback so we even don't need to execute it manually but the catch is that these two functions the setting timer and waiting for it must be executed from the single thread that is from the dpdk interrupt thread so what we have to do we have to pack the job to call the set waitable timer to special kernel object to pass it to the interrupt thread via the system call and then in drop thread we'll set the alarm and wait for it in a usual loop this way we both satisfy the pdk model requirements and comply with what windows offers us and once for us when using this api but the most challenging part of the api was unmistakably the memory manager on the next slide and memory management in el consists of three broad layers the first one is our team allocate api then comes the mellow heap allocator and then the so-called memo log path with which users don't interact it allocates huge pages from operating system it can operate in legacy static mode when it uses static memory layout or dynamic mode when it allocates huge pages on the fly windows supports only the dynamic mode and our task was to implement the memo lock layer for windows without touching the upper layers so that windows part would benefit from every improvement in those layers so that we have no violation the challenge was that monologue works in the following way it allocates large portions of virtual memory area and then it maps each pages to predefined virtual addresses it does so for multi-process support windows has no multi-process support yet but we have to work this way in order to fit into the dbtk architecture if you move to the next slide you will see that windows have several several apis that can deal with mapping memory at fixed virtual address and also that can map huge pages but only the newest apis the virtual alloc2 that is available since server 2019 or windows 10 is really up to the task that we place for it in the pdk because not only do we need to resolve memory but we also need to split the big reserved part of virtual memory into chunks and then free one chunk and replace it with a mapped kit page is an inherent race in this process between the fin a chance and mapping it but for now we have to live with it and hope that it doesn't fire eventually after mapping a huge page we need to get its physical address on the next slide the rear outline the tasks to do that we need a kernel driver and this driver must be secure to be secure it has to make sure that physical address it handles to uh the user space process is never used otherwise we may introduce security vulnerability uh like data leakage so we need to paint all the memory that we give physical addresses for and we need to track allocated memory for each process which is also subject to configurable limits to do that we use a special driver work to fees on the next slide how it works when we allocate memory from the system we get a memory region with virtual pages and when uh application called dpdk api and this driver installed we recover the whole memory region unlock it as a whole check it if it's physically contiguous and give address back and at the same time we add this region to our tracking structures and enforce limits red to fuse is the first of the two kernel drivers uh provided for windows to make the ptk work and the next one is netflix so pallavi please uh yeah uh hello my name is pallavi kadam and uh i work at intel as a software engineer and today i'll be talking about intel pmd support on windows uh so as mentioned earlier as of dpdk version 21.05 we have uh i40e that is 40 gig and ice uh 100 gig pmd support on and we did uh experience quite a few challenges earlier when enabling uh the code on dpdk code on windows so one of the challenge and requirement was to keep the code as much same as with the existing dpdk code and uh the good part was that with the help of community and microsoft folks we were able to decide on the compiler of used for windows dpd clang lvm uh and today we also have mingw uh so it was then possible to uh support gcc extensions such as uh type of attribute specifiers and as well as to minimize if dev uh clutters uh in the common code and uh to deal with os specific apis we had to replace uh and add additional uh header files as well uh and uh yes intel pmds are os agnostic and uh they can be easily portable on windows so we just took the existing pmd code and compiled it on a windows platform and we did experience few uh warnings such as typecasting integer comparison uh which were easy to resolve and uh we also had to define new uh macros at one or two places uh as the linux specific apis were deprecated on windows uh so these are the next a few uh immediate uh tasks planned so as we have enabled i40e and ispmd on windows we have also uh verified that the basic rxtx and we will also make sure that other features such as rss receive site scaling flow classification etc works on windows um and and also that uh it is full parity with the linux performance uh the next is the iabf uh we are currently actually working on enabling ivf uh that is intel adaptive virtual function pmd on windows platform on windows dpdk and uh hopefully this will be available soon to try out this is mostly to execute the dpdk application in inside the vms uh using sr iov mode and uh the next we have also planned the automation uh using test cases uh to verify the performance and features uh for intel pmds next slide uh is the windows dpdk setup from our lab uh the right part or the table is showing the hardware details of our windows server and uh in the block diagram the right block is the device under test which has a windows server 2019 os installed and the architecture of windows dpdk is very similar to the linux dpdk as you can see uh in the user space we are running l2 forward application using dpdk dpdk stack and full mode driver uh and in kernel space we have a net ui kernel model and also as dimitri mentioned we have our two fees uh as well uh now and uh as you can see from the picture that the net ui is used for the initialization purpose and uh resource allocation and thereafter l2 forward application will uh directly talk with the hardware uh using dpdk stack and pole mode driver uh bypassing the uh kernel networking stack and uh left part left block is uh linux os as a link partner and we are using the pdk package and to send and receive the traffic and uh next slide we have a demo uh showcasing this l2 forward application app uh yeah uh it's i think it's playing so you can see the left part is linux packet uh generator and right is dpdk l2 forward application running on windows os and uh the application is starting and able to receive the uh packets on the uh left side on the linux packaging so yeah this will be available to you later as well and uh when the slides are available and yeah i would like to hand over to tal now uh from nvidia to cover his part uh thank you tal over to you hi uh thank you pallavi uh my name is uh schneiderman i'm from nvidia today i'll talk to you about the uh work we did to port the mlx 5 pmd to uh to walk on windows um so can you go back a slide before that i really yeah thanks okay so analytics 5 pmd to those of you who familiar with it is a bifurcated pmd which means that as you see in the diagram to your right it's using the kernel driver of the nick to create control pad objects the data path of course is directly through the hardware but it needs the um the kernel driver for a allow so this challenge this uh structure uh presented us with some challenges to port the pmd to windows and mainly because apis that the linux kernel driver exposes the users and don't exist on the windows kernel driver for example the verbs api the the api that does uh ex is exposed by the windows kernel driver which is called the defix api is is used currently in the in the before the port was done but it's used partially and needed on windows as well and the the pmp is also use the specific linux libraries linux code for example pthreads and the os system calls which of course didn't exist on windows needed to be resolved so and next slide please okay so the effort that we did on on a work to make mlx 5 uh to make mlx 5 os agnostic uh before catered pmd is of course reuse the maximum code in the control path everything that isn't os specific will be used by both oss and will not be changed any os specific code will be isolated to [Music] functions which will be implemented uh by uh once for windows and and once for linux you can see it on the right for example there is an open device function on linux it's implemented using verbs on windows it's in implementing using device so the common code calls the same function um depending on the os that's running a different implementation is called everything that was linux uh specific regarding uh library calls and the os calls were were replaced by uh or enhanced by uh eal and uh for um easy installation of the device package for dpdk developers to simply one click install it and run an dpdk compilation we added to the win of two which is the windows kernel driver we added the device installation package uh available for the dpdk developers and users so and next slide please okay so here is the end result on the left you see the previous uh uh version um depending on the os it's either using device only on windows or um all the apis that used and continue to be used on linux nothing has been changed the enhancement was done on the windows part and the pmd is fully bifurcated and both oss so our status and the roadmap as mentioned before sense dpdk 2102 windows dpd mlx5 is supported using winona f2 driver 2.60 it supports all of connectex 4 and higher nvidia cards it supports both native setups and installing it on a windows virtual machine using sriv and of course our future roadmap is roadmap is to continue and enhancing the pmd in every dpdk and in every those windows 2 release you can see in the dpdk 21.05 we enhance the more checksum offloads to a wino f 2.70 which was released a week ago um enabled pairport rss and promiscuous mode and those are for the future release of window f2.8 so now i'll show you a brief demo of our pmd running with the latest windows kernel which will use an l2 forward application uh but also using an rss uh and an rss course with it yes so okay so you can see uh on the right there is uh on the left you see l2 forward running it's running on eight cores each core each core uh receiving packets receiving and sending the total number is 80 to 83 million packets per second on 64 byte packets you can also see that the the way that windows is using those eight cores and very high performance using a single mlx 5 port okay thanks so much this is harini i think actually it's me so uh what i wanted uh to say that as you have seen now the pdk supports a large part of the dpdk and we more and more often hit situations when we contribute windows supporting patches to different parts of the dpdk and we need people to review them to test them maybe but most developers are really linux people honestly i'm one of them my radio system is linux so we need some tools that are free and that allow us to uh review the patches from familiar environment uh what we have here to for this task for first from the early 2020 we have cross compilation using mingw uh so that uh most of the development if it's not a canon driver happens on linux if you like uh then to test windows patches in kuyama there are free development virtual machines from microsoft and there are also tools to develop drivers in them and we also as windows maintainers try hard to provide command line versions of all the actions you need to set up dpk and windows and to tell the patches so that you can control it almost without mouse what i wanted to say to all the developers of the dbtk don't be afraid of this environment it's unfamiliar but it's new and that's what is amazing about it hey uh thanks everyone uh so uh quickly to wrap it up in the interest of time so uh this is what i think we at microsoft are looking at at this time so as summarized by both the intel and the nvidia teams here you can expect expanded hardware support and more feature full pmds um and we'll have more uh more hardware nick pmds coming on board as well uh we are also um uh working on an interface to allow sharing of the nic between the windows kernel networking stack and the dpdk app so that your app doesn't take over the entire neck so if you're interested in learning more about this please drop me a note offline and i can get you some details uh we're also going to be looking at adding unit tests and dts tests and you know supporting the pdk in a windows vm eventually um now we can do it with sri oe depending on the pmd that you use but uh you know we want uh we want it to be outside of the assad iv scope as well and then we also expect to add some compiler support uh in the future um so uh so how can you help so first up i think i wanted to thank the team behind windows platform effort so far uh our tech board and governing board members that have kind of seamlessly helped us integrate and our partners at nvidia intel and our multiple isvs and you know subject matter experts that have joined us from the community um if you want to learn more um you know please help us make it make this better provide us feedback go on to the getting started guide and for any questions drop a note to dpdk when at microsoft.com um thanks everybody thanks for the time okay yeah let's take it so is this effort also on windows 10 os yes it is windows 10 and the windows server 2019 uh then do you plan uh vert io net support yes it is planet there were rough patches for that it's just not yet upstreamed it will be upstreamed i think in the nearest future uh then i think the third question is for harini yeah i think at this time uh i think at this time it's it's on x86 but i think arm64 support i mean arm platform support is not ruled out it's just not at this time of focus for us um so i i can reach out to you offline on a plan we can sync on that and again there is the last question uh is it for server only or client versions of windows it is for both server and client as long as the nic can work uh with the os um you should be able to use it both on windows 10 and service queues all right for any other questions please drop us a note at dpdk win at microsoft.com or reach out to us at dev dpdk.org email list and we'll try to get back to you what is it for end users if dpdk on windows os i don't quite follow the question this um you should be able to download and install dpdk both on windows 10 skus and windows service queues that should be possible any other questions um rc uh i think i'm gonna bring you offline i think we are um we are out of time almost um so um i will take up uh the ai to follow up with you offline um thanks everyone thanks for the time today and thanks all my uh partners here uh for making the time bye

Channel: DPDK Project

Views: 186

Rating: 5 out of 5

Keywords:

Id: i3vw-n5wx6I

Length: 31min 49sec (1909 seconds)

Published: Fri Jul 16 2021