Reverse engineering vendor firmware drivers for little fun and no profit [linux.conf.au 2014]

hi thanks for coming on it's a pleasure seeing so many of you so I'm going to be talking about work I did on reverse engineering a piece of software provided by a vendor for configuring systems now I've previously written about this publicly and therefore you may be able to figure out which vendor is is I prefer you not to attach the name of the vendor to this presentation in an obvious manner for reasons that will become clear later on I've also been informed that I am not permitted to swear if I suddenly go very quiet then you can read into that as you will right then I'm I am NOT going to swear in French either yeah anyway deploying servers is a fairly tedious activity if you want to roll out say several thousand servers then not only do you have to physically take servers into a hosting facility physically plug cables into them you then potentially have to physically attach a keyboard and monitor in order to perform firmware configuration and make sure that they're configured to your requirements for instance it may be that you don't want hyper-threading enabled because you have performance concerns of your particular work so it doesn't deal well with hyper-threading it may be that you want to TPM to be turned on it may be that you need to use a CI rather than the legacy IDE or vice-versa that kind of thing we pay people to do this and I assume that they live fairly sad and unhappy lives so the obvious flip side to that is that automating server deployment is awesome it means that these people do not have to go and touch large numbers of individual servers and poke things into them instead you can just wreck the servers unfortunately so far we haven't found a way of automating that you have to plug the cables in you turn it on and then you buy some automate automate admit ISM rollout all the firmware configuration and that means that the people who formerly had to spend their days standing there configuring systems can instead be made redundant which is possibly a win depending on how you're looking at is so there's a few different mechanisms for doing this the most straightforward and traditional one is over serial console most server level firmware will allow for firmware serial redirection as well as just providing a operating system level serial console and that means that you can typically drop to some sort of configuration interface over serial this allows you to modify configuration settings however anyone who's ever tried to write a configuration script using expect something that sees inputs and then pokes output will know that it's painful not doing that when you're running over studio and doing it over serial instead is going to be absolutely miserable so it you could do it that way it's not secure the attractive vendors gradually started realizing that some sort of standardized mechanism for doing this would of course be completely inappropriate because that would remove vendor value add so they all came up with their different ways of doing it even when they are using the same standards they all have different schemas on top of that standards in order to ensure that it's almost impossible to write a piece of software that couldn't configure multiple different vendor machines so you end up locked in to anyway several systems now offer a web services interface that allows you to make API calls and get back the current configuration stage and pushing you configuration stations this and then there tend to be various vendor specific methods sometimes these run on the local system even so they make vendor specific calls to the firmware in some completely undocumented way which then permits you to modify configuration options and then when you reboot the system the new configuration options have been applied the vendor in question that supplied a tool that did this it is a small where it's not actually particularly small it's a 250 kilobyte binary you run it on the system that you wish to configure or on an example of the system that you want configure and it spits out an XML file and the XML file contains every configuration option the current value and the possible values you then modify the XML file and you set configuration values as you want you feed that back into the tool it does some magic and then when you reboot the system the firmware is now configured in the way that you request it so this means you can for instance turn on TPM change the hard drive settings change the boot priorities and then all the systems can boot if they configure some that boots they can let boot the first time you can feed them a small provisioning environment that runs this tool can fix them appropriately and then reboots them and they're then appropriately configures downside is that it's a 32 bit binary now for our specific case I should give you a little background nebula we make a box you push into a rack you plug a bunch of servers into our box you turn it all on and it automatically deploy stuff to them and you have an open stack cloud now some of our security work has focused on taking advantage of firmware level features like using TPM server vendors tend to ship systems with the TPM disabled because people still have privacy concerns over them so it's it's more convenient for us if we can tell customers yes these new features can automatically be used in your system without you having to physically change thermo options we can upgrade the cloud controller you reboot all nodes and suddenly they have a new phone configuration that's the ideal scenario we can turn on firmer options as we require them we don't ship any 32-bit libraries because why would we do that so having a 32-bit binary was not particularly useful to us I as a result decided that we should figure out what this binary did and then reimplemented and then we would be able to do this without having a 32-bit binary so the first thing you do when you have a userspace binary and you want to know what it does is you rose on the rest trace and s trace is if you haven't heard of s trace it's a tool that intercepts every system call that an application makes and then dumped the system call and the arguments to standard output and that means you can see every files an application opens you can see every read or write it performs to any open files you can see IAP CIL's it uses so kind of expectation I had was that this utility would open some kind of hardware interface and then do some rewrites IAP tolls to it and I could simply moles of those to rest raise and then reverse engineer its protocol so firstly I assumed that it would be speaking to the base pen management controller over IP mi IPM I know I'm not actually planning on swearing here I was trying to emboss IPMI actually stands for but I don't know anyway it's awful ipmi is a management in PMI sense for management interface it's a standardized mechanism for talking to devices then do you have something to say okay right so the question was does the IP mouthing provide an access method via the a network connection in this specific case there is the vendor provides a network action that you should obviously have connected to an entirely physically separate network this is not in any way connects to the outside world but they do not have an API for configuring systems over it so at this point this vendor only provide this is the only method for configuring these systems so I assumed that it was going to use IPM ID which is the standardized mechanism for speaking to devices like base pair management controllers the service processes that monitor your system they're typically actually small Linux computers inside your computer that should scare you but it wasn't accessing /dev /i PMI which is the kernel interface to ikmi so it wasn't sending any ipmi commands to the ikmi hardware so okay well there's no fundamental reason why they would use IBM i it would be the obvious way of doing it but maybe they're doing it in some cleaner manner the service processor also appears as a PCI device so I thought well okay maybe it's accessing PCI configuration registers no no it does not touch anything under too much they slash past-life PCI and it doesn't touch the legacy one on to /proc either so it's not doing PCI access by the kernel as a result of this I thought well okay it must be doing some sort of memory mapped access and the kernel has a feature called mm IO trace my o trace means when an application attempts to map a PCI memory area the kernel doesn't really make it when the process then tries to access this PCI device the kernel can catch this access dump it and then send it through to the hardware and that means you can see all the PCI accesses that a tool is performing it's not limited to PCI you could also do the same for any part of address space but typically applications doing this use PCI so this was used heavily for the Nvidia driver reverse engineering that were I used to produce the nouveau driver Mme Oh tres meant that they could see all the accesses that the binary and video driver was performing to the hardware and then they could reverse-engineer the mastering from there now I configure the MMI Oh trees it's part of the stock kernel now it sounds a debugger fess it's pretty brilliant but there's no access to any PCI space so overall at this point we know it doesn't use dev ipmi it does not use any of the standard PCI configuration space access methods and it does not just map a PCI register region directly and then reason right to it in fact this tool does not use any kernel provided Hardware access now this is a little bit strange and once I'd come to this realization there's only really one thing to do after some consideration I went back and looked at the esterase logs again because a user space binary even if running as root is by default not allows to form any hardware access but that's not entirely true it could use USB stuff I also checked it wasn't doing any USB accesses just in case but the device doesn't appear on USB so that was being very very strange a lot of the S Trace logs and I saw it was calling IO pl/i OPL allows you to set the privilege level for i/o operations and this means that you can then access from user space ports based IO which is something provided on PC hardware you have the memory bus and you also have the i/o bus the i/o bus is a straightforward method for accessing Hardware it's low performance it's low bandwidth but it's a single instruction to read something and it's a single instruction to write something so it allows you to design very simple hardware it allows you to write very simple applications it's a really bad sign to see this cold really really bad sign because at this point you know that the application is doing hardware access itself without asking the kernel for any assistance so the problem with an application that does in and out Pro that IO port writes is that those are not they're executing directly on the CPU they're not system calls and so you can't follow them using s trace there's no way of just tracking them and seeing that that's happened so my initial approach was to instead use gdb and I ran nm on the binary which gave me a list of the symbols in the binary and I saw it has its own in-and-out functions so I set gdb breakpoints on those and then I wrote a small gdb script that meant that every time it hit one of these breakpoints it would print the arguments to the function print the function name and then continue and that members I could see every in and outs that it was performing through these function calls and that meant that I could see all the hardware accesses it was performing so excellent I can now actually see what this application is doing and then I realized I really wished that I was not able to see what this application is doing because it's really really really awful uh so this time I am failing to swear now a small digression PCI devices have two main ways of accessing features you have register space that is typically memory mapped or can even appear in IO space and there you have PCI configuration registers and you need PCI configuration registers to do things like map the memory regions into the memory map you need to tell the PCI device where it can actually fit itself in address space and make sure it doesn't conflict with anything else you often also have some low-level hardware configuration that's performed via PCI configuration space now the firmware needs to do the initial piece of a configuration and the firmware at this point is running very early in the boot process the firmware does not have a great deal of intelligence available to it so the firmware needed a very easy way to configure PCI devices and way you do that is you take the device address and the register you want to access and you write about all of those you pack all of those into a 32-bit value and then you write that 32-bit value to oxc f8 which is an i/o port oxc FC at that point will then be mapped to the register you requested on the device you requested and you can read or write from alexey FC so you read the value you want and then you write it back the kernel still does this for the majority of PCI devices PCIe provided a better way of doing this and the other thing that limits this is that with this mechanism you can't have more than 256 PCI configuration registers PCIe as it more so it needed this other way of handling them but that's kind of out of scope here now looking at this there's kind of an obvious problem you write 2 CF 8 and then you read and write to CFC now in between you writing to CF 8 and you touching CFC what happens if something else touches CF 8 and the answer is that you read or write to the incorrect register which is probably upsetting so there obviously are mechanisms to prevent this from happening and unfortunately all of those mechanisms are implemented in the kernel and if you're not using the kernel nothing prevents this from happening if you ran two copies of this utility at once for instance you would end up with incorrect values being written but even worse if the kernel itself decides to touch a PCI configuration register while you're running this tool which the kernels allowed to do because it's the kernel then again you'll just write over arbitrary well not auditory well defined you will not write the stuff you expect it to write and potentially you will in the best case scenario you will end up writing to a different device and you will end up just putting most configuration registers get ignored worst case is the hardware sunday page weirdly and the kernel crashes or the bus crashes and your system falls over the absolute worst case though is you write to the kernel rights to cf8 you write to c f8 and then the kernel writes something into the register that you were just about to write something into and then suddenly you've just written a completely invalid configuration value into your system firmware and it's likely that your vendor didn't actually test the system to make sure that this wouldn't cause problems so you could perhaps now end up with a system that decides to try to program its memory clock to a couple of terahertz and that is going to cause you problems later on so that was pretty bad I was a little bit upset at this point but it wasn't just touching PCI configuration space some of the values changed PCI configuration space but some of the values performs access to the system CMOS region and the CMOS region is where traditional firmware configuration values were it's part of the real-time clock cards where on the PC and there's two i/o ports to access it there's an address port and a data port you write the address and then you read or write the data and you can see that maybe this has the same kind of problems we just talked about because again the kernel will tend to access the CMOS in order to do things like read the time from the clock and yeah Ben suggests that you should not run this configuration utility while you're running ntpd so at this point I am clearly already scarred for life I was before I started this process as well but things have just got worse some options don't trigger any of my GDD breakpoints so I thought they're still changing the hardware somehow but they're not calling in bu or out P from the they're not calling these functions so what are they doing so I started stepping through instructions in gdb I broke down the laws the vendor was nice enough to leave the debug symbols in the tool ernie's the the function names were still there I didn't have foldable ability but you know anyway it was nice of them to do that this would have been more miserable otherwise so I was able to step through in gdb and I was able to discover that when I stepped past a certain point the next instruction was an address that was not part of my text region it was not part of the applications executive or code but it was clearly code if I disassembled it I saw code and this was when I realized why it was only available as a 32-bit binary it was opening slash dev slash mem which provides access to the system's physical memory the kernel won't let you map most of that but because obvious security disasters but it does provide access to the firmware and it provides access to certain PCI devices so it was opening dev mem and then it was M mapping a chunk of physical memory into its own address space and when I looked at the address it was mapping it was in fact part of the firmware so this system was in fact mapping the BIOS into its own address space and then executing BIOS code I really deserve a medal for this performance so gdb wasn't going to help me here because ggb can't set a breakpoint on an arbitrary instruction these another approach an LD preload is a wonderful thing LD preload allows you to intercept certain calls that an application makes and execute your own code instead so what you can do is provide an LD preload library that takes IO PL and then when an application calls IO PL you don't actually do anything so the application believes that it now has the ability to read and write to system i/o ports but it actually doesn't when it's attempt to it will take a segmentation violation which was in normal circumstances caused the application to crash but because we are brilliant people who I'm not sure I was going because we're awesome we don't allow that to happen we insert a signal handler that catches the segmentation violation and then when we get into that signal handler we decode the instructions around the instruction pointer because the kernel hands us a stack frame that tells us where the Koger excuse me was so we can read that out of memory look at those instructions and then we take the x86 instruction documentation we work out every sequence of bytes that can perform and in or outs to system i/o and then we decode them set IO PL to 3 which means that we can then do the port i/o perform the port i/o said I OPL back to zero so the next one will still trap increment the instruction pointer and then return and then the application continues it gets the values is asked for but we're also able to print what it did ray this was not my idea in fact I use some code written by someone else to do this so I take no prints at all for this concept it's a great way of doing things it's the only way that you can do certain types of debugging in the reasonable way which is a terrifying thing so what was the tool actually doing and that was still an open question I could now see every single port i/o it is not just the ones that was doing through his own functions and it was accessing some IO ports that were directly on the ikmi controller so I if you looked if you look in LS PCI you can see that some PCI devices have IO ports as well as memory regions and that is typically so the firmware can do access to this configuration space early in the boot process now the access patterns it was performing were quite familiar looking and then again this was really quite upsetting once more because after doing all of this after laborious ly finding out how to get a full trace of what this tool it I discovered that the mysterious work it was doing was actually identical to the PCI configuration space access the PCI config registers that it was touching were mirrored in IO port space the only expansion I have for this is that some of the values are required by the firmware before it's actually done any PCI configuration and so it does the port IO access in all to get them instead but anyway ah that was a little bit upsetting at this point I actually had to borrow my neighbor's recycling bin anyway now that I had all of that I was able to write a utility that could set all the firmware values I knew how to program every single type of them I could reliably set values if I reboot the system I saw that those values had in fact being changed I could turn the TPM on I could turn the TDM off if I wanted to I can now write a small hard code utility that I knew would work on this system the question was what if I wasn't running us on that system most of I was running is on a different system from the same vendor that have the same programming model where was the XML file that it generated coming from so I ran strings over the binary strings as a small tool from binutils it basically prints out any sequence of printable ASCII characters that's longer than a configurable number so it'll print any string and also a bunch of garbage but any string will appear and none of the XML data was in the strings output so it wasn't creating the table itself that information was coming from somewhere else but there was one thing I saw and so this final line here is anybody able to guess what that is right it's a list of command-line options it's an argument to get off so I had run the tool with - - help and it most of these options were documented options but that one at the end the cattle D lowercase D that was not documented ha was in fact an undocumented debug flag and in fact when you set that the program printed out every single thing it did then now follows a short period of silence so and one of the nice things I saw there was it hunted for a specific EMI table DMI is a specification that allows the firmware to provide typically static tables of data to the operating system some of these our stands died some of them invented specific so for instance there's a standard one that gives you the vendor the system vendor name the device model a list of pieces of hardware attached to the device so that's mostly straightforward and then been specific tables are obviously been specific and obviously the one that was looking for was vendor specific because what else would it be doing this turned out to be fairly uninteresting the DMI table was way too short to contain anything particularly interesting but it did contain a value that was clearly a physical address and then okay you map best address and then you get another table and this table contains the entire list of strings and values that have been used to create the XML table XML file so success I now knew how to finds the set of options I knew how to obtain a the set of values that each option could be set to and I knew how to access the address of each option so I knew whether to go through PCI space where to go through I owe port space where to go through CMOS and which registers right there that was mostly just tedious it was a matter of staring at a dump of the table writing some pausing tools to dump it out to make sure that I understood enough the table I didn't end up decoding all of it there's a small number of bits that I didn't figure out but they didn't seem to matter that much I don't think I've killed any systems with this so they can't have mattered that much anyway so here is a summary of the behavior of this tool it's accessed PC on configuration space in a racy manner ray it's access CMOS space in a racy manner array and it's executed BIOS code from user space alright anyway I took what I'd learned and I reinvented this 250 kilobyte user space binary which was full of race conditions that could potentially destroy your hardware and I realized it as an approximately 1,000 line kernel driver all right and normally that would be the ends the story I would have mailed this to LK ml and we was merged it and everybody would have been happy that's not actually what happens things as you can tell because this driver has no emotions the kernel but you would think that this meant that something had gone per if eclis long in fact things got quite a bit better and I got an email from someone at the vendor in question who was interested in ensuring that future generations of the hardware didn't behave the same way and wanted to make sure that we that it would be implemented in a way that was compatible with Linux so there's ongoing work to make sure that that happens so that was actually wonderful and that's a genuine hooray at which point I felt that I deserve to drink so I think at this point we've got about 10 minutes for questions so please if you have any questions ideally questions that relates to this presentation though I will potentially be able to answer other questions you have about life mark I okay the question was in my communication with the vendor was I able to identify the programmer who created this I did not feel that actually tracking this person down would do anything to improve my life were I to somehow sink into some sort of haze of red mist it's difficult to write a significant amount of free software from inside a jail cell where they don't allow you to see lights let alone computer how long did this take as I ended up doing this in most of this work got done is I've June or July last year which in Boston is brutally hot and incredibly humid and therefore I was completely incapable of sleeping so most of it got done then and it took about three or four days of pretty much full-time and where is he pretty much full-time looking at about 16 hours rather than eight the rest of the time as I said I couldn't sleep so was probably drinking but it was under a week for the most part and then the most tedious part of actually writing the kernel driver was all the string parsing that I had to do which having reconsidered this and having actually communicated with the vendor I'm now thinking that doing string parsing in the kernel is a bad idea for several reasons and instead we'll probably just dump the data route and then parse that in user space and we'll just have a driver that provides a mechanism for providing free access to the hardware rather than one that actually implements all of the code locally and can't you just do the context p3 sister pairs yes but that's not sufficient you need to also go through i/o port you need the i/o port access to the PCI device things and you also need the CMOS access which you couldn't theory do three slash dev slash NVRAM I suspect what actually happened was that this tool was a port of the windows code and that probably has a stub driver to grant the same sort of levels of access or alternatively it was written before no we've always had sir we've always had PCI context base access haven't we yeah so I I don't know why they didn't do it in a more reasonable way you could do this entirely in user space in a safe way potentially but it would be really miserable yeah and there's some downsides to doing it in kernel you could change these in some of this in user space but there are some benefits in doing at least a hybrid implementation where the hardware access is in there there's already a driver in the kernel for driving did different bits of functionality on this card so as you get in there seem like a reasonable thing to do too but you also said the xml strings okay so where did the XML come from robben vines the XML where said yeah I smell string sorry I meant the this interesting strings that were contained within the XML feed binary did produce the XML but it did so using these strings that it had read out of the table reference from DMI table so that answer your question why the hell even we I extend how we allow user space programs to write to PCI why are we letting user space write to PC our configuration registers and to i/o ports and X 2 files code that kind of thing the answer is mostly X 3 kms we did graphics drivers in user space because that sounds like a brilliant idea well part of it was that xorg was fairly kernel agnostic so it needed to have methods for handling direct access to hardware when the hardware could be the same over different operating systems so basically yeah now that you've got kms this is actually functionality that you can completely disable and have a completely working Linux system in fedora when we use UEFI secure boot we in fact disable all of these accesses because their horrific lis insecure so yeah we you don't need them anymore bare they're mostly as a result of historical accident is anything I learned here generally useful for other devices or other systems potentially there are some most other vendors do provide a web mechanism for accessing these configuration things which is actually closer to what we wanted from the systems but some of these vendors as well as providing the web access also provides user space tools that can be used to configure from the host side state provide both mechanisms and I would hope that none of them are quite this bad but maybe they are I I haven't needed to look because it didn't directly in hi what I was doing but you would probably use the similar techniques in order to figure out what they were doing except ideally you would find out where they have a secret debug flag that tells you everything they do first there's only try content from the vendor at the start of the process in also alpha documentation then I didn't because my experience of doing that has guarantee not been great and often it would probably have involved filling out a sufficient amount of paperwork that it was faster to reverse-engineer it than it was to just actually fill in forms that also the problem is if he speaks with the vendor first then the vendor may say don't reverse engineer this or we'll sue you if they haven't said that to you they might still sue you but it's it's a little less straightforward for them to do so then I'm a big believer in this kind of instance of asking for forgiveness rather than permission anyone else are there any good reasons for developers we've written it this way I ain't parts of it the ACM code from the BIOS was presumably because that is the specified way of doing it from the vendor technically my implementation is probably violating the spec by doing what the BIOS does rather than excusing BIOS code but actually executing BIOS codes in the kernel would probably be frowned upon so being charitable that aspect of it is potentially legitimate in a sense the rest of it okay Ben suggests that it may have been a das tool and that's actually quite plausible it may well have been a originally written for DOS and then this is a part of it that was explain why it doesn't use any operating system functionality other than a very limited set obviously endorsed you can do this and they're asking races because dos doesn't do anything yeah was this a lot more or less fun than fruit flies so context here I used to be a fruit fly geneticist fruit flies are undocumented there's a certain level of similarity between the two topics however fruit flies the codes that fruit flies execute is not written by any one competent at all and if I did clearly wasn't in fact written by anyone it works as a result of multiple side effects it basically exploits buffer overflows and all kinds of things sometimes you read straight off the end of some code into some other codes and miraculously it carries on working anyway and we don't have the faintest idea what it's doing really we've managed to figure out that well this bit does that and this bit does that and somehow you then end up with a fly but beyond that it's the other ones went wrong doing with fruit flies though is that code stays still while you're trying to work on it and fruit flies don't in fact fruit flies will do things like die and then you need to wait for more of them to be born than they you need to wait them to grow up and then you need to separate the male ones from the female ones and then you need to make sure their carries specific mutations this is way better did I look for a debug flag and the fruit flies no if anyone does find an undocumented debug flag and fruit flies that when set causes fruit flies to tell you what they're doing that would be excellent so now's the time I said thanks for coming I'll be around for rest a week and I'm giving a keynote mark can I uh can I get one more round of applause from a figure

Channel: Konstantin Bläsi

Views: 41,059

Rating: 4.9547172 out of 5

Keywords:

Id: j5NciKpHZzs

Length: 43min 32sec (2612 seconds)

Published: Mon Jan 13 2014

Reddit Comments

[u/__foo__]

I haven't seen this posted, but I think it's pretty interesting(and humorous).

This seems to be the blog post about this topic mentioned in the video: http://mjg59.dreamwidth.org/25686.html