Integrating and using DPDK with Open vSwitch

it's always good to see new presentations about performance improvements and new features in obvious with DP D K so we're not going to do that we're gonna do something slightly different instead what we wanted to look at was just talked about you know how we felt the user experience I'm you know and the challenges that we've had involved in integrating DP Tiki DB TKO into OBS okay so I mean open V switch dates back from 2009 or maybe even further been can tell us later but DP DK wasn't integrated it into OBS until 2014 so a five-year DISA five-year old application and then it has to integrate DP TK and of course you know the obvious reasons why you'd want to integrate DP DK is the performance improvements but the challenge is that OBS was not built for DP DK from the ground up so it had a lot of infrastructure in place already so then it makes it even harder than to integrate something like DP Hey okay so just a quick kind of overview of challenges and if you kind of you know rewind to 2014 with DP D K so Bruce may live to regret that he said DP DK is greedy no he says no but you know obviously things like trading and it has very sad models and DP DK wants to own some data structures so what we mean by that is it has a very definite representation of what a packet is like the the RTM buff everything gets done at an initialization so back in 2014 there wasn't a hot plug so you know you had to have your devices bound before you started V switch um we found sometimes there's been inconsistencies in PMDs you know that said it's really nice to have a PMD interface and be able to wrap things like rings and V host and that saves a lot of code and in open V switch but sometimes the differences between NICs can leak out at PMD interface debugging well I think that's been a challenge long because I mean a typical OBS user just wants to use TCP dump long term support issue so the first LTS was 1611 so back in 2014 there wasn't the opportunity to have book fixes and I can remember one example where a pretty big book fix came into DP DK it was marriage into master pretty quickly but obvious had to wait maybe three months before it picked up that book fix okay so just a few things on that on that list some terms of trading so I mean obvious creates its own control treads and its own treads for control and data path functionality so it doesn't use the DP DK slave L cores but of course we still have to pass in the minus C argument if we want to initialize DP DK you know and that caused some problems but you know eventually we were able to find a way where we could use the one of the CPU set of the OBS control treads to initialize DP DK and then we'll just kind of go back to normal open V switch control treads that we can schedule with Linux across different cores so so we kept you know after a while we were able to get back to the OBS control tried functionality and then the results of additional treads added to for the data path so I guess the point is it you know if you are integrating into an application you don't necessarily need to stick to the DP DK trading model you know you may be able to integrate it with an existing model okay so packets so OES has its concept of a packet and that's a DP packet so obviously then there's a clash because DP DK has the RTM buff and potentially that could have been an issue but as happens locally DP DP packet was very well layered and I guess I'm saying locally but I'm probably a hundred percent sure that it was done intentionally and just well design code but that meant that we could have a build option for 4dp DK and we could we could back the obvious DP DK packets with the our tem buffs and so and it's very well confined in one file so there's just a different accessor functions depending on what the back to the DP packet is and it's probably one of the very few places in OBS that there is any hash to find about d PT k okay so initialization so DP DK initialization requires arguments and lots of them we know that MJ talked earlier on about the minus n for and I guess to a DP DK user you know you could see it the way MJ described it that DP DK users have a hard time to you have to be able to figure out that command line and especially things like how many memory channels do I have so then you take it one level above when you've got so you've got the user the users of DP D K so maybe some of the developers of open V switch with DP D K but then you've got the users of OVS so it's way too low-level but we still have to pass them in because EA l and it needs arguments so so so that was one problem another thing was they had to be done when the V switch D was initialized so Aaron actually submitted patches and made changes for that and we used OBS d OBS DB to capture those parameters so even though it might look similarly complex to a an OB s user this is a more standard way of putting parameters in the OBS DB and then they can run V switch D the way they would have done previously on a course 10 with defaults none of these things actually need to be set the only thing that needs to be static a equals true ok so we go back to a normal switch D command line okay so on configuration so again devices still need to be bound and you know people can use DP d hey dev bind or driver control was introduced as well so that there was no hot look back then so DP DK devices had to be bound before the switch D started and then hop ler came along so that was a very good addition and something that seems like a small thing but in terms of the actual name and the the syntax to add a port there was a very hard restriction on the naming so had to be DP D K and then the port number and there was you would see numerous males and the mailing list you know but people confused about the port number and you know when we were trying to get DP DK you know I suppose or removed the experimental tag on on DP D K in OVS this was a big stumbling block because it was felt that it was a bad user experience that people had to do this so there was patches submitted and that was changed to use the the PCI address so that that way we could have an arbitral reports just like you would normally in OVS and the the PCI could be used there's a few problems with that that we're trying to figure out because multiple devices can be behind one PCI so we need some more generic usable way but I have that's something I think we're looking at so yeah one of the issues that that we actually still struggle with so we say here PMDs are used for i/o with hardware that's good and we're able to interface with the PMD library no problem one of the ongoing issues that we kind of struggle with is there can be integration issues with new NICs so vendors will come with their PMD say hey look you know we can you can support our NIC and they'll just use test PMD to run their NIC we've seen actually like first bring ups of Nicks new NICs cause seg faults and in V switch D because other arguments weren't planned for or you know order of operations they they didn't expect or you know lots of different problems there and additionally the reports that come back from those are inconsistent meaning like what the user sees so user might see one nic reporting like some number of queues when it's when the queues are enabled disabled and another NIC might report well you know we have like a hundred queues and and so that's all we're gonna report and it's up to the software to change that config so there's some inconsistencies not just in the user experience but also in like the first bring up and so the the big point that we wanted to say was like we you can't assume that there will be zero integration effort just because something runs with test PMD it's not like gonna be golden so now we get to debug debug is very difficult to go through when DP DK site goes wrong there's not much available for debugging unless you actually do get a seg fault and then you have a core dump and you have lots of great information to sift through otherwise you have nothing there are sparse logs that DP DK provides and kind of what's worse is you have to recompile to enable the logs which means customers won't ever do that and people you know users won't ever do that and if if like as a red Hatter I have to provide a new rpm every single time I want to like get deeper in debugging that's really not friendly and it's difficult to support whenever new features come along applications need to actively enable them and that goes especially for debugging so you know if some new debug framework comes in or you know like pea dump for example the application has to adjust to enable that that's kind of a pain point for for integration and then we've seen failures that sometimes like they're difficult to reproduce because you have all these threads running in concert you have a lot of work use being drained by different parts of the system and it's difficult when that causes a problem to actually debug it to reproduce it you know to make it be reliable and you know we're speaking here from a developer standpoint so if it's difficult for us I mean just imagine how difficult it is for users it's like impossible tuning parameters figuring out how to get around you know maybe like a hardware issue firmware issue whatever requires a lot of specialized knowledge and there's not much documentation that that exists for that so but I didn't want to leave you with something that was all dark and gloomy like we use the LTS and Yunnan and and Luca doing really good job huge amounts of bug fixes back port it to sixteen dot eleven so that's really nice I I hope that continues all right so upgrades yeah we've had a lot of discussion about API ABI it prevents dynamic linking and that means that we need to carry actually multiple versions of D PDK every single package that that we want to provide like you know if we want to provide a moon gen package a TRX package and OVS package up they'll all have to have their own version of DP DK LinkedIn so plus we'll have to carry the standalone DP DK package for anyone that's doing app development and all that so you know OVS developers are very kind of clued into DP DK but that's not a guarantee that other apps will be and we're trying to say like they should be really kind of have to follow the development and I just wanted to point out that so there's a link to a mail on the mailing list from the very first discussions of integrating DP DK where it was brought up that API ABI are going to be issues and hopeful hopeful II they'll you know settle out and there'll be a stable ABI I guess in three years we haven't seen that yet but I'm hopeful maybe it's coming questions okay thanks Kevin and Aaron thank you [Applause]

Channel: DPDK Project

Views: 567

Rating: 5 out of 5

Keywords:

Id: bUZSDm809K8

Length: 13min 51sec (831 seconds)

Published: Wed Dec 06 2017