37C3 - Breaking "DRM" in Polish trains

[Music] give a warm welcome to Redford QC [Music] okay and Mr [Applause] [Music] trick and it's an honor to announce the talk breaking DRM in Polish trains reverse engineering a train to analyze a suspicious malfunction [Applause] [Music] [Applause] hi I'm Redford this is kufri K and Mr TI not trick and we'll talk today about trains we'll do a quick intro tell the story and then we'll go into technical details so uh we sometimes play together CFS with dragon sector and Poland can into space I work for invisible things lab uh I mostly do lowlevel security and reverse engineering and let's and they they will introduce themselves in a few slides let's start with the story as you already know the story is about trains and the story actually starts a long time ago in 2016 when Kon it's a local polish train operator bought 11 impulse trains which one of them is on the photo then after some time the trains started reaching 1 million kilomet on the autometers and by this uh amount you have to do a uh a big maintenance and because the manufacturer warranty already expired they started a tender to select the best offer for Servicing and the offer was won by SPS it's an independent train Workshop in Poland and in the first quarter of 2022 the first train reached the workshop so let's see at the at least the public timeline the servicing started with uh the number 24 train uh the workshop to took apart the whole train sent the the parts to the manufacturers and then assembled the TR train back and but the problem was that the train didn't start afterwards then they took another train for servicing and it was the same the trains didn't want to start after servicing and what's even more interesting is that in the meantime another Workshop started Servicing trains for a different train operator and they run into exactly the same problem so it's it's getting a bit suspicious uh and the story got noticed by media in Poland because you had like less trains running so the manufacturer issued a public press release and they said that uh among many other accusations they said that someone interfered with the security system whatever that is and something happened in between and the the workshop asps started returning the trains which work so what happened and what happened in the meantime uh after the the workshop got into troubles the issues didn't look like normal issues because the the computer was saying that everything is fine and they had so uh pointers into direction of the manufacturers involvement so but they didn't have any idea what to do so they Googled polish hackers and found us so we we got in contact we got the trains but that about that later uh in August we managed to unlock the first train and by a few months months later we gathered enough evidence to notice authorities about this and that's what we will talk about today all right I think it's my turn so hi I'm Mr tick known in Poland as span clash in Germany as her he been he been a engrosser ban fan so I want to briefly introduce you I want to walk you through some initial terms here so before I tell you how to unlock a train uh let's define what a locked train is so uh we have basically a train you enter a cabin all the system reports that the train is ready to roll there's this device combined throttle and brake lever so you push it forward the train releases all the brakes and then it should accelerate but it doesn't we have H oh that's it the brakes nothing happens you can you can see that zero on the screen so we had a Lo train the workshop bought additional two CPUs of the same very small yep that's one of them and we got access to all service and maintenance documentation for the trains and yep we started so uh a small detour to actually see what a train is this is a very simplified diagram of how most of Moder trains work nowadays so at the top we have this three kvdc that's the Polish Uh current source for trains this cable that goes above our heads uh the green box is the PLC the CPU that controls all behavior of the train then we have the white or greenish power converter that actually converts this high voltage for all devices and power all smaller devices there uh at the lower part of diagram we have the stuff that is responsible to slow down the train so we have compressor that creates a pressurized air a lot of systems in a train works on a pressurized air like brakes for example more on that later and in top right Corner we have inverter and a motor inverter is used to convert the 3 Kilts to a current that motor can accept accelerate not burn and Y that's the thing that makes a nice sound when you start a train everything is interconnected with a canabas about which q3k tell you more okay I will hi I'm q3k you might know me from such posts are get as getting whand to run on iPod Nano but today I'm not going talk about anything about that instead let's talk about can now can is an automotive protocol but turns out because a train is made out of multiple cars that means it also uses can at least that's my understanding in reality it doesn't even use can it uses a layer on top of that called can open so we'll kind of look at that layer instead and can open effectively turns your network of or or network of can into a distributed mapping of ID into variable data so different parts of the computer can like read of the system can read and write some values and the first thing we effectively analyzed is like what is the difference on the can buas what's the difference in can open between a train which works and a train which which doesn't work also know as known as a trained and what we found out quite quickly is that the inverter would actually be told uh slightly different messages from the PLC instead of being sent a one which means uh please run please do not run the the emergency stop it would be sent a zero which means do not run and instead of being sent any power to any of the four inverters the power would be always set to zero on the other hand everything from the frontal lever up to the PLC looked fine so obviously we needed to look at somewhere in between the PLC and the inverter because that was the most likely corporate so this is what we can reduce our problem to there is a train it has a can Network it's actually a bunch of different buses it's five of them there's two plcs which are called master and slave and there's four inverters so let's just focus on that and in fact let's focus on the PLC first because I keep we keep talking about CPU PLC what is a PLC a PLC is a programmable logic controller it's like an Arduino but bigger what's a TCMS that TCMS is the entire system that you get from a vendor if you want to control your train electronically and in this case neac bought a system from selecton which is a Swiss brand part of a system called Mass M83 X which is based around the CPU 831 TG PLC which I'm just going to call CPU from now on and in in cont in comparison to Arduino is actually programmed not via C code but via a weird standard called IC 61131-3 we'll get back to that later this is the PLC in this natural habitat there's the PLC on the right there's a canbas expander to the to it left and finally to its full left is a power box notice that none of these are colored yellow and this is very important it means they're not safety critical so we can't touch them I am not is not a joke the important parts are yellow so whenever you look at the train keep that in mind so how do you actually program one of these we have the PLC here on the right we looked into it it's based around an infinion tc1 130 trick core core it has a bunch of RAM and has a bunch of flash which I'm sure you know what it is it has also n vram which is nonvolatile Ram which behaves like Ram but keeps across power cycles and for programming there's a software suite from selectron called CAP 1131 where you define your project in the I standard that gets code generated into C and then get gets fed through GCC to actually create a normal binary and finally that is uploaded over U UDP or rs232 so we know how to upload a program but how do we actually download it because turns out the software does not have a button that lets you get the software from the from the PLC for some reason but we did take a look closer at it and we found debugging functionality and that was basically what how we ended up acquiring the image from a running PLC uh the debugger is based around a subsystem called ciscom which is like this UniFi dll that is shared across different programs from selectron like the ID called CAP and some extra utilities we looked at that D we look at some UDP packet capture and we found more or less a protocol and what we found is a that when you want to debug a program you set you log into the device with username and password thankfully that cannot be changed at least not in the old software versions from selectron uh and then you query like hey I would like to get this variable range like I would like to monitor this variable range by address and I like this memory that I previously requested and turns out yes you just give it an address so we can just write a real debugger that just asks for all memory range from the beginning to end and that's how you get an image of your project from a PLC and now I must apologize but I'm going to the most boring part of this talk which is talking about the standard that is used to program these things let's start easy this a this is a sample poou there's going to be a lot of acronyms PO is program organizational unit and this kind of PO is an FB which is a functional block function block which is implemented in St which is structured text it's a mouthful don't worry about it it's going to get worse top part are the variable definitions bottom part is the actual code what happens is the code is just executed every single tick of the system and we have free variables we have start stop running these are two inputs one output and X state which is a persistent State and the code gets executed from the beginning to the end you if x start is set X state is set to true if x stop is set X state is set to false and then the output X running is set from X state so far so good not terrible let's look at how it looks like when it's generated to C this is the structure that gets generated you can kind of think of it as object orientation right like you have a structure for your FB and that's fine like all the variables we've seen are embedded as parts of a structure not bad this is the emitted evalu this is the emitted evaluation code so this D St code converted into C you'll see it's pretty much a one to one mapping right so again the same code if x start is true set X state to true if x state is false sorry if x stop is true set X St to false and assign the result to X running cool uh this is another function that gets admitted this is called the init function and that is like any object orientation as well it's a Constructor for your struct so that gets called the first time this thing gets initialized since everything we start off is false this is all just zero assignments okay not too bad now for the it gets worse part this is another poou this is uh program which is another kind of Po it's like an FB but it's top level and this one is implemented in a language called fbd which starts for function n blog diagram and you see this language doesn't text or at least it's not mostly text is actually graphical language and what we see here is we embedded our instantiation of the previously defined uh FB and we wired some some other inputs and outputs so X start X stop and X running a and X running B and these are wired so that effectively our two little motor controllers control to different outputs and are crisscross connected to the start and stop let's see how that gets generated so structure as expected the generated structure for the top level program still has the all the variables we defined it's missing X start because we messed up sorry and it has two s and it has the two embedded po motor control one motor control 2 but you'll see the type here is unexpected it's SC C in C it's not the you would expect that it might be the the structure we defined previously but this just like weic type put a pin in that let's look at the uh initialization function it's more or less what we've seen before things just get set to zero but again nothing is touching the embedded pus so what's up with that well maybe let's look at the body and that will answer our questions so this is the generated body code from that diagram we saw and if you look at the the two little uh things in Brackets these are the two evaluations on of the subordinate pus we set the x start of the first motor control to the xart of the top level module xtop to xtop and then the first second one they crisscross so xart is xtop xtop is xart and then for each one of them we evaluate them and you see the evaluation goes for macro code o and actually if you look a bit higher up the getting the reference to the struct goes for macro code o so what are these and these turns out are the worst part of this job because the macro OD and O are just defined as the references for a global table called oop so effectively every time you have one po accessing another poou it does so for a global array that is effectively untyped and sometimes the indices into this array are Dynamic so this successfully breaks any sort of attempt of doing doing static analysis the traditional way with traditional tool link just for complete completeness this is an oopt we generated and if you remember in this code sample we had o of 12 to get the function uh to evaluate both motor controls and here the index number 12 is indeed the SC body function of the motor control so you can follow that but it is definitely not pleasant and the SC Inc types within the structures we have we still haven't seen how they're actually set turns out there's another Global Constructor that sets all of them and we'll just skip that because again you already have falling asleep so we more or less know how the Cen looks like we know how the generated SE sources look like now uh Redford is going to talk a bit how this looks like the other way around starting from binary back to C so let's look at the same toy project but now looking at something closer to what we'll have from a real train so the binary uh and it will get much worse because we first has to have to disassemble it or the compile and we had to use gidra because other tools just didn't work well for trior and actually G also didn't work but that that will'll discuss in a moment it's going to get worse yes it's going to get worse much worse uh for the toy example we had symbols but of course for a real binary we didn't have any symbols uh and let's see that's the the compiled code for our to toy example and you can see in the first lines that there are even more casts and here it's not that bad because we have symbols so it means that the compiler knew the types so it's not that bad but it's still annoying and also you can see that there are some troubles with function pointers you see this end f f f f FF this shouldn't look like that so the problems with gidra and trior there were a lot of them for example trior is a bit weird it has separate registers for data and addresses and as you can see on this example they're reflected in the calling convention if a function has data arguments that are passed through data registers and pointers are passed through address registers and the problem is that gidra doesn't implement this so we had all the arguments order messed up so we had to script gidra to fix that then you have bugs in the uh semantics of trick or instructions this is an example nor dot had this end one in a wrong place so gidra actually thought that this instruction work slightly differently then it really did and we found a bunch of bags like that so what to do uh we used a lot of scripting because uh you've seen this OKO PT type which basically erased types from everything so yeah a lot of scripting to try to undo the damage then some binary analyzis remember we have this spare computer we can upload the code we downloaded from a train and just do some experiments on the desk for example we can poke some variables and see what happens and this is quite quick because we can do it at home uh then another idea is to look at the differences between trains just div the software you may ask how is this possible why aren't they all running the same software it turns out that I we don't know why but basically all the trains had slightly different software so we had I think now we have 26 different versions of the software for 30 trains yes so that's a lot and maybe maybe there's something interesting in the differences between them and we could also look at the can traffic and try to find the corresponding place in the code and also we had some motivation in the form of a deadline because the workshop called us after some time passed hey guys no pressure but like you have a week because it turned out that the train operator after like running without the trains I think they they they were six of the trains were locked already and it was problematic for them so they decided that next week they will cancel the servicing contract and we'll ask the manufacturer to do the servicing because the manufacturer claimed that they can fix that so we had quite a short short deadline and in the end it turned out that the ding was the the most fruitful way because it as we noticed there are a lot of changes in the code responsible for locking the trains and we actually found the exact place where there's a logic which blocks the trains on purpose uh it was basically the the the the first part we found was operating on values which were almost like a million so our guess was this looks like a like odometer because we knew that the trains were supposed to be serviced at 1 million so the values look like odometer readings uh and we we notice that it does something with the values and then if something is wrong it swaps one bit in nvram and then we found more bits like that with different conditions which can switch them and we did a quick test because it looks interesting we can do a test on our desk let's oh and what one one very important thing we noticed that the those bits differ between locked up and running trains so this is a bit suspicious let's change the bits to be the same as the the trains which are healthy and we uploaded this to the train controller [Applause] and [Applause] okay but that that was just a test we need to actually understand what's happening there before we like allow those trains to run like it's all nice to just flip some bits and like confirm it does the thing but it's slightly weird to just like be fine with that we wanted to actually understand what's going on so there are two very important questions the first question is what are the mechanism which cause the train to actually not run and the second question is what triggers them so let's start with the Locking mechanisms the first mechanism you already seen in the message to uh power converters so you've seen those four bytes each of them corresponds to one power inverter and if the bit uh here you can see if uh this is very simplified like the code like we wanted to post the code hours to fit the code in the slide it just doesn't fit so this is a simplified data flow diagram still treat this with a grain of salt I actually tried I gave up at I don't know four slides or something like that with code so it's something like that uh you have a bunch of different checks which are those triggers we'll talk in a moment they got ended and if if they decide that the train should be disabled then the uh throttle is gated through this and it's always zero if the checks don't pass and that influences the can open message another way which you also have seen in the that message is this bit which is uh usually it's documented as reserved for but in some documentation we found uh the description as emergency stop uh so we are not sure how exactly it works but the locked trains had zero and the working ones had one and here we can actually show you the code we had this function which decided whether the train should run or not and it was basically directly written into this scan message the result of the check you can see this the building of this one bite of the message you have four four bits which is in the message you see one s f o f f is those four ones which decide which power inverter should be enabled and which should be disabled and then you have this magic bit which comes from a function FB call which we call sneaky checks it's a technical term but that did a lot of the calculation for the like required conditions like this gets piped into a whole bunch of other parts in the code but this is one place where you could show it like almost immediately flow into the the C messages all right my turn so uh another nice failure how we can lock a train is to tamper with some aor stuff so we have this pantograph it's this thingy at roof of the train that connects to the uh wire that basically provides all the electricity that train needs but this device is operated on a pressurized air so if the train is cold and dark we need to generate some initial air pressure uh for the pantograph or uh electricity pickup to go up uh but we can't use the main compressor because it's really a power hungry device and we have very limited batteries in the train so there's a secondary compressor uh installed in the train that can be uh started in a cold train then it generates some small amount of pressure that is just enough to raise the panograph then we have enough electricity to engage the main compressor pressurize everything else and we even have a brakes then so uh the idea here was to it would be a real shame if the secondary compressor just stopped working yes so one of the look mechanism was to disable secondary compressor and after a while if it's disabled and it should be enabled it's reported as a compressor failure and we can't start the train so another another very important question is how to trigger all these lock mechanisms so we found a few ways to do that first that is one of the most common in the trains we investigated is what we call lack of movement or idle timer so I want to mention here that all these trains are Suburban kinds of trains so like asban they run for like 20 hours per day to generate any Revenue so there is a check that verifies if a train was running over 60 km/ hour for over 3 minutes and if this condition is not met the train would just Perman permanently lock so if you may ask yourself a question when the train doesn't move for 10 days when it should move 20 hours each day probably either something is broken and it needs servicing and yep somebody has to do that servicing or it's not used but this is not really a case except for few cases for example uh before uh the trains went to the workshop to get serviced uh they were just withdrawn from the service they waited few days at the sidet tracks and it was enough to trigger the lock so colon the operator asked manufacturer to fix those trains and they fixed it by extending the lock mechanism from 10 to 21 days and they've added a very clever mechanism Jo fencing so the train would only lock if it stays [Applause] so you can the train will look only if it states in these random locations so let's try to draw this random locations on map so the first location is the main competitor of neak pesab bidos that's the second biggest manufacturer of trains in Poland and their workshops that's the third Workshop owned by PESA in M maski that's the SPs Workshop that was still in construction when those stuff was implemented that's that's called future proof that's another competitor fabl from very nice sound city of shanu then that's the SPs we were hired by but wait there's more and there is the manufacturer Workshop in nov but it has an additional condition which was disabled all of the train so for debugging purposes they could enable also the geofencing trap at their own Workshop to test if it really works [Music] there are also some another additional locking mechanism that were not present in all the trains so for example we had a um you saw it in one of the initial pictures the can 831 that's the model that extends number of can buses if the serial number of the uh extension device doesn't match the one is stored in the CPU the train will lock itself permanently there was also a similar change uh similar check to check the c number of WTB that's the bus that connects multiple trains with each other then there was a check if the inverter firmware inverter's firmware version is correct that's actually a same one because we want to uh have a quite aligned inverter firmware and the main software there was a hardcoded check for 1 million kilometers after which the train will not run and there were some additional checks for odometer consistency so uh values of odometer were uh compared from few different devices and if they were off by more than 100 kilm the train would break yeah but one thing to remember here is that what as as we said before there were many different versions of the software so it's not like every every train had all of this like every train had a subset of those checks and except for the first one the least ones are way less common so we also had a very nice date check in one of the trains so that's the train that had these secondary compressor problems so uh the train was supposed to be serviced on 21st November of 2022 so21 2021 yep so yep if you spend a few seconds staring at this line of code you realize that this doesn't exactly do what they intended probably unless they intended to break the train repeatedly from the 21st to the 30th of November and 21st of to 31st of December granted this was only in one train but it's still H because all of these that's not how you compare dates I mean it is apparently the good news here is that in year uh 201100 this lock will not work anymore for 21 years for 21 years this is actually one of those examples where we can show you the code because we did manage to fit it on two slides this time so we can kind of follow along on the left hand one you'll see that we are running the function dfb to get the current time we do the comparison as in the previous slide and we use that to set a set reset latch which we then save to local storage code is date after and then in the right hand Pine you'll see there's a a bit more uh combining of that with other inputs into a trigger L variable then this goes through two other set reset latches and finally that ends up in an nvrm variable called nvrm lock enabled so you can kind of follow it from here to the to this nvrm lock enabled of course the names are ours symbol and like none of this was um had any names so and then we keep keep keep following and we go actually to the function that decides to enable the compressor in general like does the compressor actually need to run and one of the conditions is that it's not locked in in vram and you can see at right at yes the second statement that it actually set the output line of one of the can IO extenders so we can use these free Snippets of code to like go from date directly into the line on the can IO extender and as you can probably guess spara pomot is secondary compressor yes this is on one train yep that train is now famous because it did indeed break on the 21st of December this year yep [Applause] but don't worry New Year's is going to run just fine what else did we find there is a secret key combination you can input in Cabin that disables each of these locks it's not actually an economic code it's much more boring than that but you know yep this one is the famous one so at some point Unfortunately they removed it so oh well then there was this mysterious device that was bridging one of the can buses can 3 uh with uh passenger information system which has a direct access to the internet so in theory there is a possibility to somehow uh gather Telemetry of the locked values because we know that this device received uh some metadata about the locks in few cases it was able to the trains were able to lock by themselves using values received by this device but uh analysis of the device itself was not yet performed that's what basically we're doing now there's a few other things we this discovered because like as we were interacting with follow these operators some of the trains ended up going to neag and back so we did the you know kind of sensible thing of of doing comparisons before and after but also not just we didn't just get our diffs back from what they changed but we also got a few gifts for us including an update to dhmi which is the software that runs on the cabin that would display the scary message about intellectual property violations like as any other system alert like oh you know your breaks are engaged oh you're violated copyright uh I am not going to bother translating all of this it's all legal but let's actually find out because maybe there's some some kind of like hidden legal check behind this it wasn't like showing all the time something was triggering this message let's see thankfully the HMI is just Linux plus cute so like reverse engineering it's like just free F to seconds so we found out that what actually does supposedly violate nx's intellectual property laws is stopping the train for 21 days and then getting it to run [Applause] again now I am not a lawyer but I'm not exactly sure about this interpretation okay there's a few there's two case studies that we still want to show because they were somewhat interesting one of them is poo K which is a Transport company and with their workshop in Su bitka yes so so quick context after the train started from successfully from the from our Workshop we got a lot of other companies having troubles asking us to look at their trains yeah so one of them we we we learned about four trains that just wouldn't start and turns out you could unlock them but if you power cycle them they would lock up again and by our understanding this was just a programming bug so because we didn't want to actually modify the code to fix it like all of the things we did we never modified any code um we they the trains just got sent ended up being sent to neak and neak fixed them so of course we looked at the difference and what they did is they fixed their bug so they wouldn't lock up again they extended the deadline to 21 days just in case and they unfortunately removed the unlock code in the cabin another case studing was poio stretching where uh we found that the blackbox kind of external device that's kind of used as a date and odometer subsystem a lot of different things yes it's kind of like this external subsystem that's just you see it's orange one so it's important right and it would report that is currently the year 2037 and as we all know 13 15 years is more than 10 days so the train just locked up uh quick summary of all the trains we had like we tried so hard to make like a vent diagram or a table of like all the different lock mechanism it it just didn't work so here's a quick summary we analyzed 30 trains 24 of them had mechanisms the most popular was that like the train not running for a few days mechanism that was in nearly all of the ones that had any lock in the first place the kind of combination like the kind of registering against a can 831 seral that was also very common the more spicy ones were more rare the G the GPS Geo fening was only two trains the compressor failure simulation was on one train and the UDP can converter possibly remote lock was also one TR but we have a few more interesting leads on that one more thing when you once you build one of these projects on the SDK with you the the cap11 31 software a local file on your Windows computer gets updated an iny with additional build metadata like the build time and also the build path and that then gets embedded in the in the software that gets uploaded to the PC and then if everything goes well a log entry is created on the PLC when the software is updated so we can actually look at some of these log entries in those plc's and the ones that we can fairly sure that we can trust we can cross reference against other documents and other timelines and we can reconstruct the story of some of these trains so let's look back at our timeline in August 2021 the tender was one by SPS and then in January 1st 20 Jan sorry January 24th the first time I was supposed to go to to SPS for serving and just 3 days before that it got a software update another train on this on the 26th of February was to leave to get to SPS and just two days before that it got an update I'm not implying anything but it's I would like to take a look at this if I was um Kad speaking of suspicious things let's talk about n's response uh so these are it's very difficult to kind of understand what they're even trying to say because none of their statements make any sense so we're kind of trying to piece together an argument from their side just so that we can bring it down ourselves it's it's really bad but this is more or less what they're saying first of all the hackers are someone else did it and we've looked at the code and it's obvious that whoever did it had access to the source code like you would have so many side effects of trying to Binary patch like it's obviously integrated whoever had the source code and I don't know who has the source codes other than neak maybe someone stole the trains and stole the code from n and changed it and upload it just doesn't make sense second point this is slander and defamation truth is I haven't heard of neak before this so why would I do that first there's no proof have you seen the slides fourth point oh we haven't touched these distraint since 2018 so what like you can write code ahead of time what what fifth point if I'm counting correctly this a violation of Ip laws our lawyer backs to differ we'll see you in court about that next point the software was interfered with and those trains are now unsafe we either use the unlock combinations that you left in the code or when you remove the actual reading of the HMI interface we just trigger the same functions we just use your code if your code is unsafe I don't think that's our problem right Next Step they actually came up with a new defense l oh SPS didn't have the right documents or the right software to fix these trains oh you didn't have this special servicing software how did you expect to get them running well when the tender to buy the trains from neak to KD was published was was uh won by neak neak was supposed to give all the docks required to get the train service so maybe as maybe neak is now saying that they didn't fulfill the part of their procurement and finally the last point they like to make oh SPS are incompetent idiots they just keep breaking the trains and nobody can be trusted and like it's a train give us the docks and like SPS are competent they they're able to fix it like we've seen what they've done we'll still we're still waiting for more from neac because right now none of the recommendation makes sense of course they don't admit to any of this next steps we were supposed to release a technical report about this it's much more work than we expected because just collating all the information from different firmware version kind of having comparison tables between all of them being able to like talk precisely about the data flow like getting all the code samples in a row it's a lot of work sorry we'll try to get this done in in in January second next step we haven't been sued yet uh [Music] if I would like to say hi to all the nag lawyers on the room and you know where to find us if you need us to be a bit more serious the we have we have been in touch with the various different uh governmental agencies about this basically since we learned this the Polish antitrust office W is aware of this since forever we've been talking to them the there's a public prosecutor's office case against neak probably right now going on but we don't have much comes with that it's it's like they only said that they are looking into the case we're not really kept in the loop but we did speak to all them and we did present our case and finally oh yeah there's also like a whole bunch of free letter agencies that we are more or less in touch with and finally we are likely to speak about this to a political audience at the Polish parliamentary uh Commission I I think that's a talk thank you for listening do you have any questions wow woo woo yo yo yo yeah yeah yeah so cheat codes for trains so what a world um I have heard we have a lot of questions uh in the room and also from the internet we start with the internet okay um the internet wants to know how did the trains that got the new a software update that inserted the locks um how did they get there where that was that done remotely oh yeah uh that's something we should have spoken about in the presentation uh not they weren't done remotely just to make it clear we are not aware of any mechanism to up to update the train remotely all the updates were performed by n n service people or at Nag's service Workshop yeah so like when we said we compared the software it was like the operator sending the trains to the manufacturer and we did the dump right before and right after the train was sent to them so we could see like what they changed yeah microphone phone 4 are you uh implying that these idiots hooked up a canbas transceiver to the internet and to the train uh controls maybe there is to be a bit more prec it's a bit more complicated because there's the UDP to can converter box I mean it's UDP to modb rtu and can and the can the can side of it is connected to one of the can networks and the UDP side is connected to a local telematics network for the passenger information system and that passenger information system c matics network also has a GSM modem on it yeah but for sure it it wasn't just like an open Internet that anyone can send it there's probably some VPN or something like that but that's something we plan to analyze in upcoming months so maybe there will be an update presentation someday H then the next question from the internet uh yeah you mentioned that there were some software updates um a few days before the trains got sent into service um how did those updates get there um do they actually have 24/7 access to the trains or was this done remotely as far as I know they they had before this whole thing got more public they they got access to the trains my understanding is that there's like shared workshops between a lot of these comp sometimes they they just have like shared Workshop or some free access like when they request access the at least that was in the past now it surprisingly it changed then microphone one please um it's the 27th of December did you travel here by train I didn't unfortunately there isn't good train service microphone too uh so my question would be about the I call it Orange Box the autom autometer thing uh with a weird year like 2037 was that thing affected by the year 2038 bug would the train actually run again after a year box that you're supposed to send back to the manufacturer every so often to get it fixed otherwise it breaks like that or is that a different box we we heard some rumors but that that's not confirmed we just heard rumors that those particular Red Box had a back with dates and it's it wasn't the only case it's probably just back in the code some integer overflow somewhere or something like that but we don't have any like more concrete data then the next question from the internet uh yeah the internet wants to know if you actually found any um legitimate locking mechanisms so for example if there's something like I don't know battery damage or whatever and that prevents the train from starting or is it just um workarounds yeah like for example the uh if the fir if there's a firmer mismatch between the inverters and the PLC I think it's it's a legit thing to like block the train because like the PLC software is present a very particular version of the firmware so that's a safety reason to like not run against it however this fed into the same mechanism of just like silently stopping the train I would have expected all of this would just like have a documented Behavior same for things like replacing some of the parts like I could argue okay this needs to be done uh you know not have Hazard so this should be like explicitly resynchronized afterwards but no docks so even if we argue that some of those checks were legit and I think some of them we could actually call Legit the way they manifested was sketchy yeah there were more more checks which skipped which were like very legit like testing the brakes and stuff like that or if doors are closed so we had we had a theory that it started with like a legitimate system for stopping the train if something is wrong but then they started adding more conditions to it which were less legitimate yeah then uh microphone number one please um you said that you are only using functions the train manufacturer provided uh does that mean the firmware running on those trains has remained unchanged yes we have not changed any of the code and that's the reason some of the trains we analyzed we couldn't unlock for example the one with the comp the compressor failure it was uh there was a key me like key cheat code to unlock it but it was buged and if you power cycle the train it will break again yep we are very careful because uh altering in our understanding altering CPU firmware would require a new certification of that train to be allowed to run on the public tracks but yep so that we didn't touch manufacturer did but then microphone number two please yeah so I've got a question about the origins of all of this uh I don't know about uh train maintenance but I know in car maintenance they look at fall codes and they look at OBD data and stuff like that I'd imagine it to be uh the same or similar with trains um how did SPS get so suspicious that they reached out to polish hackers that they needed to Google that because normally they've got helplines service stocks whatever what made them jump to this well you have to understand that SPS and neac might have some conflicts of interest in helping each other so there might not be a way to just call Ne and ask yo what's up if the next thing that neac says oh you messed with our security system so like kind of any sort of you know hoping to call someone and get this resolved area kind of wasn't was dead on arrival for them yeah the manufacturer wasn't really helpful and as as you've seen it almost ended up like manufacturer taking over the tender yeah and in in terms of like debug codes and debug utility We There is some utility some very limited Diagnostics utility yeah but it does it doesn't show the the locks so it's not very useful for us thank you okay then microphone number four uh have you looked at if there are any valid safety locks which lock and report an error compared to the ones you describe which just silently fail plenty of eror codes like for examp example even with the compressor failure like it would simulate an actual failure by not running the compressor and then another subsystem would report oh there's no pressure so something probably went wrong and all of these have actual effects on the HMI that are you know understandable by humans so these exist like there's nothing that should have prevented this from also being a human readable error code instead it was this silent failure mode of like don't report anything is wrong microphone number one please train was actually saying ready to start yeah ready to start but it didn't then ready to start for microphone one yes so after your story got a bit of news coverage did any other train operators reach out to you already maybe even from Germany about some dodgy train Behavior no [Applause] comment [Music] yes I have to laugh um uh microphone number two please Perfect um hi at first thank you for the nice talk it was very delightful just to watch you talk about it um did you at any point in this whole journey of unlocking the train regret the decision to accept it that's a very good question um personally I've been a bit like you know there was this one day before deadline where nothing was working yeah so there's that part for me personally just even like this is probably going to go forever for a long time it's going to be in the media for while it's going to get political so there's always like maybe we shouldn't have gone public with it but at the end of the day that's the correct thing to [Applause] [Music] do and yeah like when when it was like one day before the deadline my two friends here were like crunching really hard it was a 24 shift system to discover this so we we actually forgot to tell tell this trivia in the story we managed to unlock the first train 43 minutes before the the train operator arrived to cancel the contract so it was really really [Applause] tough my microphone too please am my microphone too oh yeah that's everything thank you perfect uh microphone one hi you did mention that uh you have to get the train recertified when you change the firmware but uh doesn't that need to be certified when they change the uh date range to 20 days or is the software OB curent for the certifier it it sounds suspicious no that they just can update the firm like that without recertifying doesn't it and that's that's part of the that's part of the low that we're like not 100% confident to comment on the law isn't exactly uh clean on this it I think it says that if the change is significant or something like that so it's I I don't know this is something more for a train lawyer on the other hand that the change was made and it was not mentioned in any paperwork from that service yeah of course they didn't mention that they updated the locking mechanism so or even update that they updated the software right that wasn't loed anywhere some in some cases the train operators didn't even know that the software was changed but is the software obscurify to the certifier that's another good question uh we don't know okay um microphone number three um you mentioned that the code is executed at every tick or something um does this mean that the kill switch conditions are also evaluated every now and then or only at the startup so like if the train during operation reaches the 1 million kilometers did uh would it emergency break or or something that's another very good question yes so I can tell uh the condition for 1 million K kilom was in the same train that had the secondary compressor problem so it would just disable the secondary compressor that is not needed in a normal train run just to bring it up from a cold and dark State yeah normally when when the train was running you had pressure in the main pressure tank but what about the other trigger mechanisms and Trigger the other mechanism like standing in place uh had more checks to Ure sure that it didn't it won't engage when the train is running uh then signal Angel the next question from the internets yeah the internet wanted to know if that train manufacturer actually sells elsewhere besides Poland yes yes then microphone number two uh yes uh besides this sheet code code did you find any other uh delightful Easter eggs or any anything of that sort in the code not really nothing immediately comes to mind yeah uh I think the the best Easter egg was just the date checking because we now are observing twice per year the international compressor failure day then microphone number four uh hi you mentioned that the cheat code wasn't the Konami Code are you able to reveal what it actually was and if it was documented anywhere it was not documented it was very simple we're not revealing the particular code because we're afraid of people like just randomly pressing the buttons in the cabin because there was more than one code like every trigger mechanisms yeah you you could do different stuff with trains with the codes and we don't want to hint people about the pattern how the code looks look like so they won't start randomly experimenting and they they also differed between some trains so better not experiment with them we always like check the code and only then we use them but it's like something that you could press with two hands within seconds it's not like you have to sit there and like type in something with binary code it was uh a very quick combination all right thanks then the next question from the internet yeah the internet wants to know what was or is the um relation between you and SPS and if you got anything from it like besides publicity and maybe free rides in trains good question we should have declared our possible conflicts of interest we were contracted to do this we were paid money changed hands uh yeah we were basically paid to analyze and unlock the trains although not for publishing this we just got like a green light if you want you you can make it public but the the workshop was interested mostly in checking why the trains broke and just providing a a technical report describing what happened then microsome 3 hello uh given the ramifications are you concerned about any reprisals be they political or otherwise and hello from Ben hi Ben uh I mean we are mostly worried about just being dragged through courts forever uh that's it because that's like the easiest way to just annoy us I don't want to give NE lawyers any ideas but oops too late I guess otherwise no we are we are like 100% sure of of that we were in the right we're 100% sure we're acting the public interest it's n that should be scared not us and we are [Applause] 100% and additionally we are 100% certain of our findings presented here yeah so unless they start sending ninja squads our way I think we're safe Microsoft number one is there any chance that nag is going to get her hands on those trains and would try to destroy evidence or do something like that could you repeat the question please if if there is any chance that naag will get their hands on those trains which were like with those locks and we'll try to update and destroy that evidence like update firmware removing those locks yeah we we are prepared for that we have done dumps already some of them are audited by big four companies so even if they try to like the worst thing they can do now is start try to hide the evidence because I don't think the court is going to like that yeah we are running out of time which means uh you can reach uh here all the colleagues Redford [Music] [Applause] qck and Mr tick [Music] yeah

Channel: media.ccc.de

Views: 220,094

Keywords: 2023, 37C3, 37c3, 37c3 eng, 37c3 ov, Day 1, Hardware & Making, MrTick, Redford, Saal 1, ccc, chaos, communication, congress, q3k

Id: XrlrbfGZo2k

Length: 61min 46sec (3706 seconds)

Published: Fri Dec 29 2023