Console Hacking 2016 (33c3)

[Music] you [Music] you have been here on stage before you successfully tempered with the Wii you successfully temper with the ps3 and get some legal challenges over there some unfounded legal challenges yes and then you an excuse my French over here by the way that is number 8 0 to 1 to get the run selection on the DECT phone so you with the wheat you as well and well console hacking 2016 yes here we go so I'm a lazy guy so I haven't turned on my computer yet for the slide so let me do that hopefully this will work so my computer is a little bit special it runs a lot of open-source software it runs FreeBSD it runs even has things like open SSL in there and nginx and Cairo I think and WebKit it runs a lot of interesting open-source software but we all know that BSD is dying so we can make it run something a little bit more interesting and hopefully give a presentation about it let's see if this works it's a good start black screen you know like it's you know it's thinking to disk and file system shutting down there we go and yes I run Gentoo Linux this is the does Wi-Fi work moment hopefully and TP ya know NTP failed well that's a bit annoying but it's the worst hello yeah it takes a bit to boot it doesn't run systemd you know so it's it's it's saying it's a tiny bit slower but it's saying there we go this is the just my controller work moments due to than the install one okay it does alright so let's get started so this is console hacking 2016 ps4 PC master-race I apologise for the horrible Nazi joke in the subtitle but it's a reddit thing so yeah PC master-race why well ps4 is it a PC is it not a PC but before we get started I would like to dedicate this talk to my good friend bin buyer we all knew was pushing unfortunately he passed away in February of this year and he was a great hacker he came to multiple congresses one of the nicest people I've ever met I'm sure some of you have met him with a agree with that and if it weren't for him I wouldn't be here so thank all right so the ps4 is it a PC is that not a PC well it's a little bit different from previous consoles it has x86 it's an x86 CPU it runs FreeBSD runs WebKit it doesn't have a hypervisor unfortunately then again the ps3 had a hypervisor and it was useless so there you go and so this is different from the ps3 but it's not really different it does have a security processor that you can just ignore cuz it doesn't really secure anything so that's good alright so how to own a ps4 well you write a WebKit exploit and you write a FreeBSD exploit duh right and everything runs WebKit and FreeBSD is not exactly the most secure OS in the world especially not with Sony customizations so this is you know this is completely boring stuff like what's the point of talking about WebKit and FreeBSD exploits instead this talk is going to be about something a little bit different first of all after you run an exploit well you know step three some things that for-profit what is this about and not only that though before you write an exploit you you don't want to have the code you're trying to exploit and with Witkin and freebsd you kind of do but not the build they use is customized and it's annoying to write an exploit if you don't have you know access to the binary so how do you get the binary in the first place I'll you dump the code that's an interesting step so let's get started with step zero blackbolt code extraction the fun way a long time ago in a hacker space far far away failure will fall got together after three one three three and we looked at the ps4 motherboard and this is what we saw so there's an alias Southbridge that's a code name by the way then there's a Liverpool APU which is the main processor it's a GPU and the CPU which is done by AMD it has some RAM and then you know the South widget connects to a bunch of random crap like the USB ports a hard disk which is USB for something explicable reason the internal disk and the ps4 is USB like it's SATA to USB and then to USB on the Southbridge even though it has SATA like what the blu-ray drive is SATA the Wi-Fi bluetooth as as the i/o and Ethernet is GMI I ok how do we attack this well Dedede are what to stop oh I have a screen saver apparently that's great I thought I killed that let me kill all that screen saver real quick something had to fail it always does I mean of course I can SSH into my ps4 right so there you go okay goodness or I fix that anyway so yeah which one of these interfaces do you attack well you know USB SATA is the i/o GMI I that's the you know the raw Ethernet interface by the way all these are CPU control the CPU issues commands and the devices replied the devices can't really do anything so you can't write to memory or anything like that you can exploit USB if you find a bug in the USB driver but we're back to the node code issue so ddr5 that'd be great we could just write to our memory and basically own the entire thing but it's a very high speed bus it's definitely exploitable if you're making a secure system don't assume we can't own ddr5 because we will but it's not the path of least resistance so we're not going to do that however there's a thing called PCI Express in the middle there mmm that's interesting PCI is a very fun for hacking even though it might seem intimidating because it's bus mastering that means you can DMA to memory it's complicated and complicated things are hard to implement properly its robust people think that PCI is this voodoo high speed no it's not it's high speed but you don't need like matched races to make it work it will run over wet string like you can Hotwire PCIe with pieces of wire and it will work at these are short distances anyway like believe me it's not as bad as we think it's delay tolerant so you can take your time to reply and the drivers are full of failed because nobody writes a PCIe driver assuming the device is evil even though of course everybody should because devices can and will be evil but nobody does that so what can we do well we have a PCIe link let's cut the lines and plug in the Southbridge to the motherboard to a PC motherboard that we stick on the side so now the Southbridge is a PCI card for us and we connect the apu to an fpga board which then can pretend to be a PCIe device so we can man in the middle this pci bus and it's now x one with instead of x four because you know it's easier that way but it'll negotiate that's fine so how do we connect the motherboard on the FPGA this of course many ways of doing this but how many of you have done any hardware hacking even Arduino or anything like that raise your hand I think that's about a third to a half or something like that at least and when you hack some hardware you melt some hardware after you blink an LED what is the first interface you use to talk to your hardware serial port so we run PCIe over rs-232 at 115 Keela baud which fixes PCIe I said it was delights all right so it makes this PCIe point zero zero zero to X and eventually there was a Gigabit Ethernet port on the FPGA so I upgraded to that but I only got around 200 to doing it in one direction so now it's PCIe point zero zero zero two in one direction and point point in the other direction which has to make this one of the most asymmetric buses in the world but it works like believe me this is hilarious you can run PCIe over serial also we were asking coding so have to bend with it was fine it's fine it's fine so PCI u 101 it's a reliable packet switch Network it uses thin culture and slash transaction layer packets which are basically just packets you send it can be you know a memory read a memory write I or read I or right configuration read configuration right it can be a message signaled interrupts which is a way of saying hey listen to me by writing to an address in memory because we can write the things so why not write four interrupts it has legacy interrupts which are basically the emulating the old set this in a wire low for interrupts and high for no interrupt thing you can tunnel that over VC ie and it has completions which are basically the replies so if you read a value from memory the completion is what you get back with the value you try to read okay so it's PCIe right so we just go wide with DMA we can just read all memory dump the kernel hey it's awesome right except there's an iommu in the apu but of course the IMU will you know protect the devices it will only let you access what memory is mapped to your device so that you know that host has to allow you to read and write the memory but just because there's an iommu doesn't mean that Sony uses it properly so here's some pseudocode and you know it has a buffer on the stack it says please read from flash to this buffer with the correct length can anyone see the problem with this well it Maps the buffer and that reads and maps the buffer but I of amuse don't just map bite food to bite bar they Baptists and pages are 64k on the ps4 so Sony is just mapped 64k of its stack to the device so we can just DM a straight into the stack basically a whole stack and take over so now we get code execution FreeBSD kernel dump in WebKit and iOS lips dump just from mapping the flash okay that's step zero so we have the code but this is not you know that's not the ps4 that we did this on it's not you know it's a giant mess of wires someone here knows about that you know flying over the face but we're gonna make a nice exploit we've done that because as I said what get freebsd whatever what comes after that okay we want to do something of course we're gonna run Linux and how do you go from FreeBSD to Linux it's not a trivial process but you use something that we call ps4 KX ik so how does this work it's you know bit simple right you just want to run the next just jump to Linux right well kind of you need to load the neck since you can take its physical RAM set up boot parameters shutdown FreeBSD cleanly halt secondary CPUs make you page tables they say bogey bit better a lot of random things I'm not going to bore you with all this crap because you can read the code but there's a lot of like iteration in getting this to work let's assume that you do all this magical clean up and you get Linux into a nice state and you can you know jump Linux okay now we jump Linux right it's cool yeah okay you can technically jump to Linux and it will technically run for a little bit yeah I didn't stop and you're not getting cereal or any video or anything what's going on here okay let's talk about hardware what is x86 x86 is a mediocre instruction set architecture by Intel it's it's okay I guess you know it's not great so yeah ps4 is definitely x86 it's x86 64 what is a PC aa PC is a horrible horrible thing built upon piles and piles of legacy crap baked thing back to 1981 the ps4 is definitely not a PC then again that's practically Sony level hardware fails so it could be but it's not okay so what's going on well PC a legacy PC basically has a 8259 programmable interrupt controller 253 program of interval timer a UART at i/o 3 f8 which is the standard address for a serial port it has a ps2 keyboard controller lady 42 as an RTC a real-time clock with the CMOS or everyone knows the CMOS right MC 1468 18 is the chip number for that and is a boss even if you think you don't have an ISI bus your computer has an ISI bus inside the selvedge somewhere and it has VGA the ps4 doesn't have any of these things so what do we do well okay let's look a little bit about how the PC works and how a ps4 works this is a general simple PC system there's an APU or an intel core cpu with the Southbridge you know until close at PCH AMD fch there's an interface that is basically PCIe though until costed DMI and AMD calls it um i whatever ddr3 ram and a bunch of peripherals and SATA whatever the ps4 kind of looks like that right so you think this can't be that yet what's so hard about this you notice all the crap I mentioned earlier is in the South Region a PC right so you know feature has a South Bridge right right right so the South Bridge the AMD standard fch implements into legacy from 1981 the marvel aiolia marvel is the maker of the ps4 South Ridge implements Intel legacy from 2002 what does that mean ah that's our South Bridge that's a marvel armada SOC so it's not actually a South Bridge it was never a South which it's an arm system-on-a-chip CPU with everything it's a descendant from Intel strongarm or XScale it has a bunch of peripherals and what they did is they stuck a PCI bridge on the side and said hey x86 you can now use all my arm so it exposes off its own peripherals to the x86 they added some stuff they really needed for pcs and surpass its own Ram why do they do this well it also runs 3 B is the on the arm in standby mode and that's how they do the whole download updates in the background get content mean update whatever all that crap is because they have a separate OS on a separate chip running in standby mode ok that's great but it's also batshit insane yeah so quick recap this is what a PCI bus number looks like it has a revised number so it has a bus number which is a bits a device number which is 5 bits and the function number which is 3 bits so you foresee this an LS PCI if you ever not this is what a regular South which looks like so it has a USB controller a PCI you know is a bridges SATA whatever and it has a bunch of devices so one South which pretends to be multiple devices because you only have three bits for a function number so you can only have up to eight functions in one device so the you know intel size which is as well M divides 1416 1a1 beyond just a bunch of devices and you can talk to all of them if you SPC I on roughly unpatched Linux kernel on the ps4 you get something like this so the aiolia first of all clones itself into every PCIe device because they were too lazy to do if device equals my number then reply otherwise don't reply now they just said oh just reply to every single PCI device that my query so that exceeds the Southbridge you know like 31 different times which is kind of annoying because it gets rican fuse when it sees 31 clones of the same Southbridge and then it has 8 functions h CPI ethernet SATA is the MC pci express ok 8 functions so all three bits turns out 8 functions are not enough for everybody function number for PCI Express glue has a bridge config MSI interrupt controller ICC we'll talk about that later HP timers flash controller RTC timers 2 serial ports I squared see all this smashed into one single PCI device so then X has a minimum system requirement to run on anything you need a timer in it interrupts and you need some kind of console the theater has no plate and opaque in the standard serial so another standard PC stuff is going to work here the board has test points for any 250 standard serial in a different place so we want the message over that okay fine Linux has early con which we can point to a serial port and say please send all your D message here very early because I really want to see what's going on doesn't need I accuse you said control equals D or a 250 that the type the address the speed and you see it says 3200 instead of hundred and 15 kilowatt that's because their clock is different so you set 3200 but it really means a hundred and fifteen K and I can see the message that actually gets you know the next booting on compressing whatever that's that's pretty good ok we eat a timer because it spies everything explodes lyric supports the TSC which is a built-in CPU to timer which is super nice and super fun the ps4 has that but the next starts to calibrate it against the legacy timer which on the ps4 doesn't exist so that's fail so again the piece for really is not a PC so what we need to do here is defining use of architecture because Lake supports this concept say is this not a PC this is a ps4 the bootloader tells annex a this is a ps4 and then the next is okay I'm not going to do the old times time calibration I'm going to do it for the ps4 which has a special code that we wrote that calibrates against the ps4 timer and it disables the legacy crap okay so now this is not PC this is officially not a PC anymore okay now we can talk about a CPI you might know a CPI for all its horribleness and all its evilness and always Microsoft dienes but a CPI most people associated with suspense and suspended hibernate it's not just power it has other stuff too so we need a CPI for PCI config for the iommu for the CPU frequency the ps4 of course has broken acme tables because of course it would be so we fix them in ps4 KX deck okay now interrupts we have timers we have serial we fix some stuff the ps4 does message signaled interrupts which is what I said the non legacy the nice new thing where you just write a value and what you do is you tell the device when you want interrupt please write this value to this address the device does that and the CPU into our controller sees that right and says oh this is an interrupt and then just fires off that interrupts into the CPU that's great it's super you know super fast and very efficient and the value directly tells the CPU that's the interrupt vector you have to go to ok let's see that's a standard M aside waylay that your computer does M aside that way this is how the ps4 does MSI the aiolia ignores the msi config registers in the standard location instead it has its own MSI controller all stuffed into function for which is that glue device yeah glue each function gets a shared address in memory - right - and the top 27 bits of data and every sub function because you can't a lot of things into one place only gets the different 5 bits and all MSI's originated from functions for so like this device has to fire interrupts then it goes to here and then that device for is an interrupt like what this is all about you know what the hell is going on like this seriously this is really up and the eyes are missing in the front there but yeah so yeah driver hell now the devices are interdependent and the RQ vector location is not sequential so that's not going to work and need to modify all the drivers and like this is really painful to develop for so what we ended up doing is there's a core driver that implements an intercultural for this thing and then we have to make sure that loads first before the device drivers so that Linux has a mechanism for that we have to patch drivers some drivers we patch to use these interrupts and drivers we've wrapped around to use these interrupts unfortunately because of the top bit thing everything has to share one interrupts within a function thankfully we can fix that with the iommu because it can redirect interrupts so you can say Oh interrupts numbered zero goes to here one goes to here too so here so the that's great this is consecutive right zero one two three four five is obviously going to have the same top bits but we have to fix the Acme table for that because it's broken but this does work so this gets us interrupts that function and their individual so just look at the checklist we have interrupts timers early serial late cereal with interrupts we can get some user space we can you know stash some user space and binaries into the kernel and it'll boot and you can get a console but you get a console and you type try writing commands and it's sometimes it hangs like okay what's going on there so it turns out that freebsd masks interrupts with an AMD proprietary register set we have to clean that up too and that fixes serial and all the other interrupts this took ages to find it's like why interrupts on CPU zero sometimes donut right yeah I entered dumping register sets and I saw this ffffff here and not ffff what's that hmm but yet like backtracking through the stack to find this was really annoying all right so we have the basics we have like a core platform we can you know run Linux on even though I don't won't do anything interesting at drivers so we have USB X HDI which has three controllers in one device again just you know let's make it insane we have SD HDI that's the SDIO for the Wi-Fi and the Bluetooth needs a non-standard config and each quirks Ethernet needs more hacks it's still partially broken only runs at gigabit speeds if you plug in 100 megabits which it just doesn't send any data not sure why and then all of this works fine in the next 4.4 and then just three days ago I think I tried to rebase on 4.9 and so we have the latest in the greatest and everything failed and DMA didn't work and all the drivers were just throwing their hands up in the air and what's going on here hey olya strikes back so that's what you know the you know the olya looks like normally so you have its again it's an arm sock it's really not a device it's like its own little system but it Maps it slow to gigabytes of the address space to memory on the PC and then the PC has a window into its registers that it can use to control those devices so the PC can kind of play with the devices and the DMA so the same address and that works great because it's mapped in the same place and then has its own DRAM you know its own address space this works fine but now we had an iommu because we needed it for the interrupts and the IMU inserts its own address space in between and says ok you can map anything to anything you want it's great you know it's a page table you can say this address goes to that address Linux 4.4 did this it would find some addresses at the bottom of the eye of MU address space say you know page 1 goes to this page 2 goes to that page 3 goes to that and say device you can now write to these pages and they go to this space in the x86 that works fine it turns out the next 4.9 or somewhere between 4 point 4 and 4 point 9 it started doing this it would map pages from the top of the iommu address space and that's fine for the iommu but it's not in the window in the earlier so now that you know you say Ethernet DMA to address F II something something something and instead of DMA into the RAM on the PC at DMA so the RAM on the aiolia which is not gonna work so yeah effectively the aiolia implements 31 bit DMA not 32 bit DMA because only the bottom half is usable it's like wine it this is all really up guys like seriously and this is littered all over the code in Linux so there's needed more patches and it works but yeah painful ok devices they all devices work now for something completely different who can tell me who this character is starship from space battleship yamato and apparently that's the codename for the ps4 graphics chip or at least that's one of the code names because they don't seem to be able to agree on like what the code names are like it's got Liverpool in some places and starts trying other places and Phoebe J in other places and we think Sony calls it starsha and AMD calls it Liverpool but we're not sure we're calling it Liverpool everywhere just to avoid confusion but yeah okay what's this GPU about well it's an AMD see Ireland's generation GPU which is spelled CI instead of Si because s was taken it's similar to other chips in the generation so you know at least that's not a batshit crazy new thing but it does have quirks and customizations and oddities and things that don't work or it is we took Vaughan air which is another GPU that is already supported by Linux in that generation and just kind of added a new chip and said okay do all the poner stuff then change things and hopefully adapt it to the ps4 so hacking on AMD drivers okay well they're open source but AMD does not publish register Docs they publish 3d shader and command Q documentation so you get like all the user space 3d rendering commands that's documented but they don't publish all the kernel hardware register documentation that's what you really want for hacking on drivers so that's annoying and you're thinking the code is the documentation right just read the Linux drivers that's that's great well yeah but they're incomplete and they have magic numbers and it's you know you don't know if you need to write a new register that's not there and it's really sucks to try to write a GPU driver by reading other GPU drivers with no docks so what do we do our hackers right we Google every time you need information hopefully Google will find it because Google knows everything and any tip but you can find in any you know forum or code dump somewhere else great one of the things we found is we googled this little string our 8xx GPU in quotes and you get nine results and the second result is this place it's a silicon kit token okay it's an XML file and if we look at that it looks like it's an axiom XML file that contains a dump of the Bonaire GPU register documentation but it's like broken XML and it's incomplete that stops at one point but like what's what's this doing here and why like what what is this where there's from right so let's take a little deeper okay Google what do you know about this website well there's some Brandon things like what the hell note that txt and what the hell yes dot txt and some Excel files sorry excel like XML Sal sheets and then there's a thing in the bottom they are called Rai grammar dot 4 dot txt mm I wonder what that is and it looks like it's a grammar you know the end of the notation description for a syntax of some kind of register documentation file mmm it just looks like a name the internal format but it's on this website okay so we have these two URLs slash pragmatics lesbo-ner that XML and slash Rai slash REI grammar dot 460 let's try something how about maybe pragmatics such Bonaire dot Rai Nats a 404 okay it's pragmatic / REI slash Bonaire REI ah bingo [Applause] so this is a full Bonaire or almost full Bonaire registered documentation with like full registered field descriptions breakdowns all the addresses it's not 100% but executive vast majority this seems to be a indie internal stuff and I looked this guy up and they're finicky worked at AMD at some point so but yeah this is really really helpful because now you know what everything beans and debug registers and yeah so I wrote a working parser for this format not the XML see this it was friendly writing a XML parser sound like convert this thing to XML but it was all broken oh he was wearing leading PHP by the way that there you go so I wrote a working run in Python and you can dump it and then you can see you know what each register means and it'll tell you all the options you can take a register dump and map it to the you know basically documented you can diff dumps you can generate defines it's very useful for AMD GPUs and this grocery speaking applies to a lot of AMD GPUs like they share a lot of registers so this is useful for anyone hacking on AMD GPU stuff over 4000 registers are documented in the just in the main GPU address space alone so that's great okay so we have some Doc's how do we get to a frame buffer so if you you know the install HDMI that's easy right the GPU has HDMI and if you query the GPU information you actually get that it has an HDMI port and a display port port okay maybe it's unconnected that's fine right ah but if you actually ask the GPU it tells you HDMI is not connected DisplayPort is connected okay yeah they have an external HDMI encoder from DisplayPort to HDMI because just putting a wire from A to B is too difficult because this is Sony so let's put a chip that converts from protocol a to protocol B yeah yeah yeah yeah yeah and okay it's a it's yeah it's a Panasonic DisplayPort to HDMI bridge not documented by the way requires config to work that's why it doesn't just work even though some bridges do and you'd think okay it soaked up to the GPU I squared C bus because GPUs have in the past use these bridges and not this one particularly but other ND cards have had various ships that they stuck in front and the code has support for talking to them through the GPU I squared C interface right right that's easy yeah you wish this is Sony enter ICC so remember the ICC thing in the earlier it's an RPC protocol you used to send commands to an MCU that is somewhere else on the motherboard it's a message box system so you write some message to a memory place and then you tell it hey read this message and it writes a message back and it tells you that's the reply you access it via a o via not via the GPU use it for things like power button the LED is turning the power on and off and also the HDMI encoder R squared C D so now we have a dependency from the GPU driver to the yo-yo driver and two different PCI devices in two different thank you yeah and okay again ICC but this I squared C you know I squared C it's a simple protocol you read a register you write a register that's all you need it's super simple right right now let's make a bytecode scripting engine to issue I squared C commands and delays and bit masking and everything and why so only why like why would you do this well because Isis is so slow that if you actually try to do one read and one write at a time it takes two seconds to bring up HDMI yeah like yeah I don't even know at this point I have no idea okay and by the way this thing has commands where you can send scripts in a script to be run when certain events happen so your dawg i heard you like scripts I put scripts in your script so you can I squared C Y you r squared C like this just go even deeper at this point right cuz yeah yeah yeah yeah okay we brought some code for this you need more hacks it needs all DisplayPort mains up the next trace to downscale doesn't work memory ground with calculation is broken mouse cursor sizes from the previous GPU generation for some reason I guess they forgot to update that so wait all this crap you get a frame buffer but X won't start ah well it turns out that ps4 uses a unified memory architecture so it has a single memory pool that is shared between the x86 and the GPU and games are split a texture in memory and say hey GPU render this and that works great and this makes a lot of sense and their driver uses this to the fullest extent so there's a VRAM you know the legacy GPUs had a separate vram and all these integrated chipsets can emulate vram using a chunk of system memory and you can usually configure that in the BIOS if you have a PC that does this and the vs or sets it to 60 16 megabytes which is actually the lowest possible setting and yeah 16 Meg's is not enough to like have more than one full HD frame buffer so obviously that's going to explode in Linux pretty badly so what we do is we actually reconfigure the memory controller in the system to give 1 gigabyte of RAM to the vram and we did that as the K X X so it's basically doing like by OC things were reconfiguring the Northbridge at this point to make this work but it works and with this we can get extra stored because they can allocate its frame buffer but ok it's 3d time right yeah GPU acceleration doesn't quite work yet so we got the least you know X but let's talk a bit about the radeon GPU for for a second so when you want to draw something on a GPU you send it a command and you do this by putting it into a ring which is really just a structure in memory that this is a list of cards and it goes or it wraps around right so that way you can queue things to be done on the GPU and then it does it on its own and we can go and do other things that's a graphics ring for drawing a compute ring for GPGPU and the DMA ring for copying things around the commands are processed by the GPU command processor which is really a a bunch of different CPUs inside the GPU that are called f/32 and they run a proprietary AMD microcode so this is a custom architecture also the rings can call out to ring back to I bees which are indirect buffers so you can say basically call this piece of memory do this stuff there return back to the ring and that's actually how the user space thing does things so you know this is draw this stuff and it tells the kernel hey draw this stuff and the kernel tells the GPU jump that stuff read it come back keep doing stuff this is basically how much CPUs work but Radeon specifically works like you know but this s32 stuff okay the driver complains ring zero test failed take ready test them so at least you know it has nice diagnostic and how does the test work it's real easy it writes a register with a value and then it tells the GPU with a command please write this other value to the register runs it and then checks to see if the register was actually written with a new value so the write doesn't happen it never it's there thankfully thanks to that rai file earlier we found some debug registers that tell you exactly what's going on inside the GPU and it shows the command processor is stuck waiting for data in the ring so it needs more data after an op command yeah not this hard let's go stalling so packet headers and this GPU thing has a size that is size - to whoever thought that was a good idea so a two word packet has a size of 0 then AMD implemented a one word packet with the size of minus 1 and old firmware doesn't support that and thinks oh it's three ffff so I'm just going to wait for a shitload of code in the buffer right it turns out that Hawaii which is another GPU and the same gen has the same problem with old firmware so they use a different not packet so there was an exception in the driver for this and we have to add hours to that but again getting to this point many many many hours of head-banging yeah ok we fix that now it says ring 3 test failed that's the SDM a ring that's for copying things in memory and it works in the same way it puts a value in RAM tells the SDM a engine hey write a different value and checks this time we see the right happens but it right zero instead of the dead be for whatever okay so I tried this I put two right commands in the ring saying write to one place what to a different place and this time if I saw what it did is it wrote one to the first destination and zero to the second destination I thinking ok it's supposed to write dead beef which is what you see there it says you know that beef is that word with the value it writes one well there's a one there it wasn't there before there's a zero because I was padding right so yeah turns out they have it off by for error in the SDM a command parser and it reads from four boards later than it should again this took many hours of pet dying and it was like randomly try two commands oh one one one yeah so it reads two or four words too late but only in ring buffers indirect buffers works fine that's good because those come from user space so we'd have to muck with those we can work around this because it's only used in two places in the kernel by using a fill command instead of a write command that works fine again how do they even make these mistakes okay but still the GPU doesn't work the ring test pass but if you try to draw you get a bunch of page folds and it turns out what happens is that on the ps4 you can't write the page table registers from actual commands in the GPU itself you can write to them from the cpu directly you can say just write memory memory register right and then I all right but you can't tell the GPU please write to the page table register this so the page tables don't work the GPU can't see any memory so everything is broken Linux uses this FreeBSD doesn't it uses direct writes and we think this is maybe a firewall somewhere in the Liverpool some kind of security thing they added we can directly write from the CPU but it's like breaks the regular like it's not a synchronous anymore so this could break things that's a really hacky solution I would really like to fix this and I'm thinking maybe the firewall is in the firmware right but it's proprietary and undocumented firmware so let's look at that firmware it's a thing it needs microcode right a CP thing it's undocumented but we take the blobs out of FreeBSD and that's great because we don't have to ship them let's dig deeper into those blobs so how do you reverse engineer an unknown CPU architecture that's real easy you run an instruction and see what it did and then just keep doing that thankfully we can upload custom firmware so it's actually really easy to just have like a two instruction firmware that does something and then writes a register to a memory location and that's actually real easy to find if you just that first like write the memory instruction is real easy to find in the binary because you see like GPU register offsets that stand out a bit in the in one column so long story short we wrote f32 dis which is a disassembler for the proprietary AMD f/32 microcode I shamelessly stole the instruction syntax from arm so you may recognize that if you were used to arm assembly and this is not complete but it can disassemble every single instruction in all the frame we're in Liverpool for pspme CMAC and RLC which are 5 different blocks in the GPU as far as I know this has never been done before of the firmware was like you know voodoo black magic thing that's being shipped not even the non AMD kernel developers know anything about this so and you can disassemble the you know desktop GPU stuff too so this could be good for debugging strange debut shenanigans and in non ps4 stuff alright alas it's not in the firmware it seems to be blocked in hardware I found a debug register that actually says there was an access violation in the bus when you try to write this thing and I tried a bunch of workarounds and I even bought an AMD APU system desktop dumped all the registers dip them against the one I had on Linux and tried like setting every single value from the other GPU and hoping I find X of magic bits somewhere but no they probably have a setting for this somewhere but you know it's a sea of ones and zeros good luck finding it it does work with the CPU right workaround though so at least we get 3d and it's actually pretty stable so if there's a race condition I'm not really seeing it so check list what works what doesn't work we have interrupts and timers the core thing you need to run any OS we have a serial port we can shut down the system and reboot and you think that's funny but actually it goes to ICC so again need some interesting code there I actually just implemented that what four hours ago because you're pulling the plug was getting old the power button works USB works there's a funny story with USB as it used not to work and we you know said fix it later there seemed to be a special code missing and then someone pulled the repo from the USB not working branch and tested it inside it's working mmm seems we fix it by accidents by changing something else the hard disk work switches via USB blu-ray works I wrote the driver for that also four hours ago three hours ago now yeah something like that and it spent 20 minutes looking for someone in the hack Center that had a DVD I could stick in to try it apparently I'm from the past if I ask for DVDs so yeah but it does work so that's good Wi-Fi in Bluetooth works Bluetooth Ethernet works except only at gigabit speeds frame buffer works HDMI works it's currently hard-coded to 1080p so yeah it does work we can fix that by improving the encoder implementation 3d works with the ugly you know register right hack and speed if audio works so that's good HDMI audio doesn't work mostly because I only got audio grossly working in general recently and I haven't had a chance to program the encoder to support the audio stuff yet yes again or you know more annoying hacks there and the real-time clock doesn't work and if you think that's simple well the cook like device is simple but ever since the PlayStation 2 the way Sony has implemented real-time clocks is that instead of reading and writing you know the time on the clock which is what you think is the normal thing to do they never write the time on the clock instead they store an offset from the clock to the real time in like some kind of storage location and there's a giant mess of you know registry it's called in the the ps4 and I don't even know where it's stored it might be on the hard drive it might be encrypted so basically getting the real-time clock to actually show the right time involves a pile of nonsense that I'd haven't had a chance to look at yet but we haven't TP right so it's good enough all right oh and we have linking lights important you know the power LED does some interesting things if you run the next so that's good so the code we can get the ps4 kay exit code on our github page that has the KX ik and the hardware configuration and the bootloader Linux stuff you can get the fuse for the next branch which is the war fork of the kernel rig based on 4.9 which is the latest public version I think you can get our Radeon patches which are three I think really tiny patches for user space libraries just to support this new chip really simple stuff the not thing and a couple commands and the REI and F 32 this thing I mentioned you can get Radeon tools at that github repo I just push that right before this talk so if you're interested there you go and if you going at the RAI file well you know you want to pro you want to run before the guys at that website realized we should take that down but I'm sure the internet wayback machine has it somewhere so yeah okay well that's everything for the story of how we got Linux running on the ps4 and you know you can reach us at that website or failure phone on Twitter so I hope that wasn't too fast sorry I have to rush through my like 89 slides a little bit because I really wanted to do a demo I thinking this kind of is the demo right but we can try something else so maybe I can shut this so I can if I can aim with my controller and this is really not meant as a mouse that's not right button come on uh yeah I think you just closed closed maybe yes so we have this little icon here I wonder what happens if it works we have internet access hopefully Wi-Fi works actually you just check real quick is it this could work really bad if we don't then ping 8 update 8 right yeah we have internet access ok Wi-Fi works ok wonder what happens we click that it takes a while to do this this is not optimized for [Applause] so the CPUs on this thing are a little bit slow but hey you know hey it works and now it's a real game console this is a there we go okay so yeah I think we can probably take some Q&A because this is a little bit slow to load but we can try a game maybe well if you are for Q&A I think there will be some questions so shall we start with one from the internet testing testing okay hey the internet wants to know if well most of your research will be published or if stuff's going to stay private well all of this I mean like the publishing is basically the code which and you know the explanation I just gave as I said everything's on github so all the drivers we wrote all the you know I mean in that case I guess also the the spec is the code if you really want to I could write some wiki pages on this but roughly speaking you know what's in the drivers is what we found out the really interesting bit I think is that there's 32 stuff from the from the AMD GPU stuff and that we have a repo for but absolutely if you have any you know general questions on any particular device or any details authority to ask I don't know you know again it would be nice if we wrote a bunch of Docs and everything but it's not really a matter of not wanting to write them it's you know lazy engineers not running to write documentation but the code is at least you know the things we have on github is fairly clean so ok so someone is piling up on for guys if you have questions you see the the microphones over here just pile up over there and I'm gonna con for please just the small question how likely is it that you upstream some of that stuff because I mean so there's two sides to that one side is that we need to actually get together an upstream at the code some of it has horrible hacks some of it isn't too bad so yeah we want to upstream it we have to sit down and actually do it I think most of the like custom x86 space machine stuff and the kernel is doable the drivers are probably doable some people might scream at the interrupt axe but it's probably not not terrible and if they have a better way of doing it I'm all yours the other kernel devs the Radeon stuff is right fishy because of the encoder thing that is like me non-standard and also understandably MD GPU driver developers that work for AMD may want to have nothing to do with this and in fact I know for a fact that at least one of them doesn't but we can I mean they they can't really stop us us from observing things and things into the next kernel right so I think as long as you know we get to come to a state where it's doable it's fine but most likely I think I think most likely the non GPU stuff will go in first if we have a chance to do that and of course if you want to try of streaming it you know go ahead it's open source right so over to microphone one please hi first I I think I should implore you to try and find trouble Hudson and control him into using your freebsd k exact implementation in heads instead of having to run all of linux in it as a joke but my real question is if the reason you used gen 2 was because system D was yet another hurdle and getting this to run where runs into on my main machine at run gen to on most of the machines I care about I do run an arch and a few of the others and then I'd live with system D but the reason why I run gen 2 is first it's what I like and use and second it's super easy to use patches on gen 2 you get those things we pull out of github which are just patch files it's not really a repo because they're so easy it's not worth cloning everything just get those patch files sticking monnet support touch patches you have a little hook to patch in that's all you need so it's really easy to patch packages in Gen 2 that's one of the main reasons number three please and will there be new exploits new way to boot Linux on ps3 with modern film ways because finding one with humour 176 is really rare now that was four or five ah but again our goal is to focus on the I just hope you the story of the pre exploit thing because I think that's good like hacker story a good knowledge to try new platforms and the next thing we're working on the reason why we don't want to publish the exploit or really get involved in the whole exploit scene is that there's a lot of drama it's not it's not rocket science in that it's like super custom code this is what can freebsd it's actually not that hard and we know for a fact that several people have reproduced this on various firmwares so like there's no need for us to be the exploit provider and we don't want to get into that because it's a giant drama fest as we all know anyway so please like you know DIY at this time okay thanks and what is the internet saying testing okay but the internet wants to know if you ever had fun with the BSD on the second processor oh that's a very good question and ie myself haven't I don't know if anyone else has looked at it briefly one of the commands for rebooting will boot that CPU into FreeBSD and there's probably fun to be had there but we haven't really looked into it and over to five please I was wondering if any of that stuff was applicable to the ps4 we are additional efforts called the new one sorry you ever tested sorry today again the Sony brought upon you peace bore the probably mean the ps4 Pro yes yeah so Linux boots on the pro we got that far GPU is broken so we would like to get this ported to the pro and also working it's basically an incremental update so it's not that hard but the GPU needs a new job definition you chip all that stuff yeah I get a lot of fancy frames here yeah but yeah as you can see the o3d works and listen you have to look up and down in this game mandatory physical and mental wellness exercise yes well then number three please sure I want to ask you if you want to put this right here iam patches to the new AMD GPU CPU a GPU driver because aim the now supports at the southern island a keep use yes it's a very good question actually the first attempt we made it writing this driver was with AMD GPU and at the time it wasn't working at all and I was a bit concerned about its freshness at the time and it was experimentally supporting this DP degeneration I'm told it's but it should work so I would like to port this you know move to a DQ DP you noting that we have a you know working implementation and we got the clean up code much better we know where all the nits are I want to try again with AMD GPU and see if that works that's a very good question because the you know newer gen might require the driver maybe so yeah well then I'm going to guess we asked the internet again okay the internet asks states that a year about a year ago you argued with someone on Twitter that the ps4 wasn't a PC and now you're saying it kind of is something and what's about that so again it's the my reason for saying it's not a PC is that it's not it's not a piece it's not an IBM personal computer compatible device it's an exceeding six device that happens to you know be structured roughly like a current PC but if you look at the details so many things are completely different it really isn't a PC like on Linux I had to define you know sub arch ps4 it's an x86 but it's not a PC and that's actually a very important distinction because there's a lot of you know things you've never heard of that are x86 but not pcs like for example there's a high chance your monitor at home has an 80186 CPU in it so yeah so nobody's piling at the microphones anymore is there one last question from the internet yes there is and um the question is what to do if there was any decryption needed no so this is purely you know you exploit WebKit you get user mode you exploit the kernel you get kernel mode we jump linic there's no security like there's nothing like stopping you from doing all this stuff there's a sandbox in freebsd but obviously you exploit around the sandbox like there's nothing there's no hypervisor there's no monitoring there's nothing like saying oh this code should not be running there's no like integrity checking you know they have a security architecture but as this tradition for Sony you can just walk around it so yeah the on the the ps3 was notable for the fact that the PS jailbreak which is a USB is effectively piracy device that was released by someone that basically use the USB exploit in the kernel and only a USB X within the kernel to effectively enable piracy so when you have it like a stack of security and you break one thing and you get piracy that's a fail this is basically the same idea except I have no idea what you need to do to do piracy and I don't care but yeah and Sony doesn't really know how to architect security systems that's it that's it you very much here we go that's your applause [Applause] [Music] you [Music]

Channel: media.ccc.de

Views: 112,047

Keywords: 33C3, CCC, Chaos, Communicaion, Congress, Hamburg, Works, for, me

Id: QMiubC6LdTA

Length: 53min 44sec (3224 seconds)

Published: Wed Dec 28 2016

Reddit Comments

[u/[deleted]]

I know nothing about what is going on but that crowd was rough on him

[u/Atemu12]

And he even got steam running, that's kinda the ultimate insult