Disgusting Secrets of Real Hardware: HOPE 2020 Voidst-Archives

ladies gentlemen and cyborgs welcome for the very last time to void star lab new york this week brooke and i are getting in the van and we're driving all the way to colorado to uh well get to our new place and to set up our new workshop so instead of the usual video we're going to dig into the private reserves this one uh is about well the sloppy decisions that get left in commercial hardware and i gave it at hope 2020 that's awesome hacker con next week we'll get back to our usual programming in our new workshop enjoy the show good evening hope attendees and good whenever it is uh youtube people i'm zach friedman prototype developer and today we're making fun of some electronics welcome to my workshop in the beautiful west village of new york new york uh don't listen to the president uh the city is not being sacked by barbarians that's me wearing an eye tracking headset i built and uh this is my workshop at the fat cat fab lab the original plan was to throw an all-night party here at the fat cat but no this viral bastard just has to invade our alveolar cells and ruin everything i am digging this whole like hope over the internet thing i i definitely miss all the late night shenanigans but damn there's so many excellent speakers and we've got all the time in the world to listen to them for maximum immersion i've brewed my own mate and ran it through the soda stream it tastes like crap which means i succeeded all right let's talk about filthy hacks how many of you have ever busted open a piece of electronics to do a hardware hack how many of you have built electronics yourself from the ground app how many of you know that i actually recorded this a week ago all right i've been building electronics for about a decade and i've been a freelance prototype developer for around eight years i do a lot of engineering but i'm not an engineer i got my degree in business and technology at stevens and i taught myself all the programming and circuits and stuff i built a lot of projects for clients and myself as my data glove my eye tracking headset and a pc case mod that endlessly generates fake boot text as a professional prototype developer i build things like concept hardware art pieces custom projects and even the occasional product my clients are satisfied but i used to get awfully insecure about my work my projects were made of copied and pasted reference designs arduinos running off-the-shelf libraries modified example code i just didn't think that real engineers like the ones that look like this would consider my designs to be professional quality i was a little terrified that one day a client would send my work to a real engineer and they'd shoot them an email back and say like did you pay that jabroni or like maybe i'd forget to close some crazy security hole and some russian hackers uh would pwn the crap out of it it ends up on the news and somebody does some sort of presentation at a hacker conference about it uh i really did stay up late tossing and turning because i left a label debug header in the release product where i forgot to set a security fuse or i made so many mistakes and none of my footprints fit the first time and my heart would sink whenever a client sent me a picture like this and i panicked like how did i let this happen how could i have caused this time went on and i got more experience i saw other people's projects inside and out warts and all i tore open products uh to modify and reverse them i read news stories i sat through talks i listened to podcasts swap stories and it turns out many of those bonehead design decisions that i bumbled into are not unique uh many of them are actually very common or even industry standard so today we're going to rip into real hardware that's been really released into the real world and discover filthy engineering we're going to see just eye rolling levels of sloppiness laziness corner cutting short-sightedness and uh creativity before we begin uh we're gonna do three just three points of context uh a just because something is disgusting doesn't mean it's bad most of what i'm going to show you is deliberate and came from calculated decisions with reasoning behind it the a lot of these things were done on purpose with intent engineers are generally diligent and good at their jobs and it's usually the business realities that sully the product not you know anyone being a smooth brain uh point b i'm i really wanted to use my own projects here but uh my clients are not okay with me walking randos through the security flaws and and mistakes i i put in there uh most of the examples are going to be authentic mass-market hardware with pics from sites that are okay with with me doing such a thing uh finally i'm not a cyber security expert uh i am a prototype developer my job is to implement the security flaws and i don't really know that much about exploiting them i'm not purporting to be any kind of security expert so i'm going to take the academic approach specifically i leave the pwnage as an exercise to the viewer okay so all that said uh let's let's get rolling when you think of product design you might think of a racially and culturally diverse group in comfortable but sensible attire writing on whiteboards annotating tablets and rotating things in 3d but okay you probably don't but it is safe to assume they at least you know have meetings or at least the guy designing the enclosure to hold the circuit board and the lady designing the circuit board itself would work together well no uh pretty often they work for different companies most companies don't have in-house industrial designers uh or mechanical engineers they engage a design firm and mechanical engineers to sculpt the the physical stuff or pick a model off the shelf and then the engineers design the pcb to fit uh the ee in fact might be an independent contractor themselves like you know like myself electrical engineers are often left out of the high level design process instead they get an email with hey fit a workstation pc into this okay thanks bye and you gotta make something that looks like this weirdly shape boards are awkward and they're ugly they're harder to design they're harder to produce and they're ugly they also like they also affect performance they emit more noise the and time goes from improving the design in the schematic into cramming is you know all those components into that bizarre profile uh really thin tiny and irregular boards flex more and they need special programming fixtures and production steps but they i guess they make the design look sexy and your boss doesn't have to pay the engineers uh and the designers simultaneously so for here's another example look at this pocket-sized uh bench top power supply this poor bastard had to cut this giant hole in the ground plane so that it didn't get in the way of the wireless antenna and by the way like look how the uh look how the module is only supported on one end like a like a tiny diving board look at that control board like it had to be made of two scraps of pcb swaged together because those mounting bosses in the corners are too big for the main board to go all the way to the edge of the enclosure and as for those weirdly shaped cutouts and that strange board shape that's there because the other board needs to pass through that board in order to fit that enclosure just like look at this noise in a power supply is usually considered a bad thing but the priority here is making it look like this here's another example uh this is a speaker from ikea just like just take a look at this board here's the bluetooth audio bit on a daughter board and the audio it receives needs to make it all the way across the the and the audio it receives needs to make it all the way across the board around that cutout to the speaker why does it need to do all that so that you can ram a cork through this hole and stand it up aesthetics uh you know where nest thinks the best place to put a battery is literally in the middle of the sensor uh just look at this board just look at it last example uh cameras cameras are the absolute worst because people expect them to look and feel like 35 millimeter film cameras from the 1960s so boards are jammed in wherever they'll fit with ribbon cable snaking around components everywhere just like look at this imagine designing electronics for this you have to cram your super advanced image processing and digital signal processing chips onto a board that looks like a subway map i also like how they put this heavy duty insulating boot around this massive capacitor but they use the super fine pitch connector with like no creep edge so like a spec of anything conductive would short it out nice so every electronic component has a data sheet this is a document written by the manufacturer that explains how the chip works its specs tables of stats what the package looks like etc but it also includes this a reference design this is sort of like a ready to go schematic it's a schematic that implements the part that the manufacturer has tested or each of its features are ready to use manufacturers sell evaluation boards and developer kits and third-party cell breakout boards and all of these usually implement those reference designs these are extremely useful in the early stages of product development because you know you're starting with known working vanilla schematics you don't need to engineer all that yourself you can buy them pre-implemented and just chain the boards together for testing you combine that with some open hardware that's easy to program arduinos etc and you know off the shelf modules for complex stuff like wireless and you can jump right into prototyping without having to start from scratch the product development process ideally looks something like this first you slap a bunch of pre-made schematics together to sanity check your idea refine the specs and really focus in on your business logic what what the thing does you then put everything on a big roadkill board there's lots of open room so you can make adjustments and really you know perfect that schematic finally you take that and get it into the correct form factor and that's your prototype you trim unnecessary features you add protective components you tighten up your design you replace these expensive modules you solve problems and you'd think that by the time you're done very little of those off-the-shelf modules and pre-made reference designs would remain in your finished product i mean it's not even like you're going to take your prototype straight to production right well the boss sees that the prototype works fine and dfm ends up being cut 25 off the bill of materials and go straight to the contract manufacturer so things that you put in early in the process to speed up your development end up making it all the way into the finished product uh here's an example this is my desk and it features a 5th generation ergodox mechanical keyboard because i'm a giant nerd the left side is suspiciously similar to an mcp23018 i o expander reference design for driving button matrices such as the ones and mechanical keyboards the right half of the keyboard is literally an arduino leonardo that design i was showing you earlier that was a car stereo mod that i designed for a client a while ago it's an lm 358 amp reference design uh combined with an ap6502 switching regulator reference design and literally an arduino uno uh it's a bunch of off the shelf circuits and god damn it i am proud of how well it did the job it's tempting to think that only small time outfits would build projects by copying and pasting but that's that is just straight not true this is an fbi tracking device that kathy thomas an environmental activist found stuck to her bumper it's made of two stacked boards the top is a u-blox all-in-one gps transponder module this was cutting edge in 1999 and has a very helpful data sheet the bottom board has a z-mix three to five hundred megahertz transceiver chip and an rfm rf1172 filter and both implementations look an awful lot like the reference design uh actually it looks like the fbi did change the design a bit on the top side it looks like they added some capacitors by hand i just love the idea of some lantern jawed fbi agent black suit sunglasses earpiece just delicately blowing away flux fumes to see solder this up implementing reference designs is a lot like copying code from stack overflow we pretend we understand it we hit control c we hit control v and then we make pre-recorded conference talks about how good we are at engineering implementing reference designs or at least starting out with them is usually a good idea uh these they're design engineers of these companies like real humans with personalities and families that make these things uh these guys are well are often friendly and will help diagnose issues in your implementation some data sheets have although have sleazy reference designs that recommend specific parts from companies that they have relationships with or they even recommend their own parts especially the expensive ones for some reason why sell a chip when you can sell a solution the point is uh reference designs and breakout boards are starting points as a rule they are not ready for production and it's up to the engineer to find out why good engineering should require rigorous testing on the bench and in real life i mean after all when it comes to breaking stuff the user outranks the engineer uh it's up to ever it's up to everyone to find out where those reference designs fall short and don't get too over committed to them right it always makes sense to use the right part even if it's harder to develop over the long term speaking of testing as of 2020 electrical engineers are still incapable of jacking into their electronics projects i consider this a grave oversight and instead they add debugging interfaces to upload and diagnose software and to snoop on critical signal lines and other other critical paths uh by the way this right here is the best slide in the deck i wanted a picture of a guy in a vr headset messing with a circuit board but the best i could find was this was this guy messing with a whiteboard and he's drawing a cat i i know because he wrote cat anyways the next best thing to jacking into your circuit board besides jacking into what into a whiteboard uh is adding a debug interface some of these things like these spy headers just expose the lines of communication between the microcontroller and its peripherals these make it really easy to connect logic analyzers and scopes and they can also be used to upload firmware other ports are dedicated specifically to the software to diagnostics and programming uh this is a jtag header don't worry about the acronym because it's just it's basically a programming interface for a microcontroller it also provides access to debugging tools like setting breakpoints which doesn't like sound impressive if you're used to writing real programs but trust me this is this is hot stuff on a microcontroller anyways using this port you can burn new firmware onto the device extract the existing firmware uh snoop the memory and just generally do all kinds of handy low level debugging functions uh you can order microcontrollers pre-programmed and this functionality is extremely dangerous right so it makes sense to remove this header when you go to market especially if it's one of these fancy headers that just accepts this convenient spring-loaded programmer otherwise somebody could say take your commercial off the shelf hardware jab a convenient spring-loaded programmer into into that header suck out your firmware steal your secrets modify your firmware to make the device catch fire and re-upload it in the past 70 years of electrical engineering real engineers trademark have made many powerful anti-tamper encounter measures you have access prevention write wants fuses code signing and code integrity checks they're increasingly common and easy to use that said i don't think i've ever used them and i don't think i've ever met anyone who used them you might you might think that it doesn't get any more nauseating than a convenient unsecured port with firmware access so allow me to direct your attention to this little bastard this is a serial port and it's even more common than a jtag header and in my opinion even more dangerous all serial ports have at least three pins you have a receive transmit and a ground if you ever use an arduino or you had a computer in 1995 you might have heard of a serial port and that's exactly what these are what's neat about a serial port is that they're often used to communicate with humans not complicated devices and they communicate in plain text all you got to do is get a usb to serial adapter and you could barge right in this is an ftdi basic from sparkfun and i highly recommend it but there really are like a bazillion usb to serial adapters out there and they do the same thing i keep all mine in an altoids tin so they stay fresh mmm asynchronous data collect the whole set it's really important by the way to to first use a multimeter to figure out what voltage it is because you know you connect a five volt uh you five volt serial adapter to a three volt device and you're gonna need a second device anyways you just look for three to six parallel holes on the board and that's probably a serial port uh you'll often find multiple headers on a board if the device has multiple microcontrollers which which is increasingly common uh one of these holes will have a little starburst around it or like an extra thick trace where it connects to the ground plane and that's that's your ground you might need some guesswork to find the receive and transmit pins really that just comes down to try out the various combinations until something works but sometimes you get lucky and a friendly engineer will do you a solid and label it right there on the board ah debug headers uh what do you think we'll find on this cereal part uh does it stream maybe some debug logs some core dumps maybe some printf style status strings left over from development could it be a little bare bones command line to access hidden functions and factory diagnostics maybe it's an encoded channel for a technician to connect some super secret maintenance tool or maybe it's a bash shell with root access and default username and password and you bet your ass that's what it is every embedded linux distro i know exposes a shell on its dedicate on that dedicated serial port uh usually that command line is root because creating you know separate accounts with reduced privileges is hard um usually the username is root and the password is root or its admin or its password some fancy devs change the password and like that said i've never seen a manufacturer provision unique passwords onto the devices even though they totally should uh at best they'll derive it from the serial number uh or mac address and if you can find that algorithm like you're in like flynn many manufacturers will change every one of their passwords to the same password and then put that password into a manual that's available for download nda free from their site this is especially common for routers and other devices that should really be secured better than this um you might have been smart and like set your ssh server on the device to use to not use pass keys and only certificates but you should know that this has no effect on the hardware serial port uh i don't actually think you can set that to require a certificate and not a pass key uh anyways like you just take that device with crazy hard security log in install a new private key and you now no longer need to sneak into the boiler room to mess with the hvac this brings us to the filthiest crevasse of the embedded world i must stop showing you pictures of circuit boards because this foul miasma is invisible it's firmware and it's the libraries that these systems are built on trigger warnings unpatched vulnerabilities decade old kernels unencrypted networks and literal fire you guys are pretty plugged into security penetration and you're probably aware that embedded devices are notoriously insecure uh not much has changed in preparation for this talk i went to shodan uh remy mel rami malik's search engine of choice and i i did a search for dropbear dropbear is a super lightweight ssh server that's included with busybox which is in turn included with most embedded distros the overwhelming majority of these devices are running linux 2.6 the latest version is 5.7 and i can't imagine how many of these have been how few of these have been patched for shell shock and and all those other vulnerabilities we've seen lately yeah this one was my favorite this poor canadian bastards cable box is running dropbear ssh version 2012.55 as in version 55 of the year 2012. the last time this guy's cable box was updated barack obama was beginning his second term there have been some security up there have been some security improvements since then uh here are some iomega nas devices that back up people's sensitive files uh iomega was kind enough to announce through the open web that these are nas devices presumably full of sensitive files uh the these things support only tls 1.2 which i have read leaves them vulnerable to the golden doodle variant of the poodle attack i don't know what the hell you guys are smoking when you name these like when you name these vulnerabilities but this is a device designed to keep your files safe and up-to-date and omega's devs have made them unsafe by not updating them uh but surely it's it's better in industry than it is in consumer products i mean these are devices that are going to be in the they're going to be in service for decades not something that you're going to like throw out when i don't know in steve job i guess back in the day when steve jobs got on stage uh anyways there are a lot of industrial control protocols used to coordinate heavy-duty equipment these things have failovers backups and like other useful properties that come in handy when your software can kill people uh or if you've hacked a car and come on like it's 2020 who hasn't hacked a car uh you're familiar with the can bus there's also bacnet which is used to connect like hvac and building safety equipment and you have the modbus which is used to connect heavy equipment and factories and processing plants and stuff um as a rule these protocols are unencrypted which which isn't as bad as it sounds like if you're connecting a conveyor belt to the control console like it's your own factory you know you don't need to uh you don't need to secure it that bad it's not like you're hooking your industrial control equipment into the open web right if you have dubious morals and you want to control 40 relays in romania then you're in luck my dude see the mistake wasn't putting the ethernet port on the controller uh the mistake was putting the other end of the ethernet cable into a router here's another intriguing device in castel novo de soto italy what's neat about this thing besides the fact that it's probably an italian temperature controller that you know shows up in a search engine is that invincent supplied the safety devices that were hacked in saudi arabia with that super advanced triton malware i'm not sure if this is the exact device we're talking about here but you get the idea you connect dangerous industrial equipment to one end and you connect the other end to a direct link to everyone on earth with the command line triton was this malware that disabled gas detection safety systems in a chemical refinery presumably with the intention of flooding the facility with toxins uh through pure chance uh it was caught and thwarted but had it succeeded triton would have been the first malware with casualties uh they really should have kept a better eye on these key switches um it was what's interesting is that uh in the triton attack they actually did prevent these things from connecting online but they had poor internal controls so uh somebody who got access to the facility was was easily able to do a firmware attack and break that security wide open uh moving on let's set some stuff on fire uh embedded devices increasingly have consequences like not only are we sticking processors and everything but we're hooking these processors up to more and more subsystems this fire here was caused by a firmware attack this usb powerbank was connected to a usbc quick charger there's a microcontroller in charge so to speak of the charging subsystem so that it can so that uh the device can demand uh as much current as it can handle so this attack is launched on the wall wart by using it to charge an infected phone the phone puts the charger into device firmware update through that usb port and then it dfuses on a hacked firmware that instructs the power management chip to wait for the target device and then run 20 volts through it this means that a malware attack can blow your phone up neat it's a good thing that these that these uh wall wart manufacturers aren't just copying and pasting pre-written firmware onto an unmodified reference design if they did that uh the attack would hit tons of different devices from different manufacturers and it would be a total nightmare to patch out of existence anyways this is where we are we cargo cult schematics we leave all access debug ports open to the public we deploy outdated software we never update it we open it up to the to the internet uh we connect more and more of the circuit to the hackable part and we make ugly boards uh these are just some of the disgusting secrets of real hardware so what do we do about it i mean as a rule design defensively uh the bad guys are gonna buy some units and then they have all the time in the world to bust them open so don't make it easy clean up after yourself uh get rid of those debug ports prevent one part of the circuit uh from damaging another use fuses and thermistors to limit hacker to limit hackers reach uh don't blindly trust reference designs uh force those force any bad actors to like have to start desoldering parts get yourself involved earlier in the process so you have more influence on the look feel and configuration of the prod of the project don't put something online or even pro even add provisions to bring it online unless it's absolutely necessary there are tons of count there are tons of measures like intranet private apns and just straight not putting a connectivity on it that can provide a lifetime of protection and finally use the anti-tampering devices that are already available to you like it's kind of a pain in the butt but you know like life sucks for a hat as for the hackers the makers and the modders like don't be discouraged by how sophisticated and polished other engineers work looks like that cat that gizmo might look really sleek on the outside but on the inside it's just full of disgusting secrets and they're waiting for you to unearth them thanks for watching uh i'm zach friedman and i really hope you've enjoyed my presentation and the entire virtual hope conference this has been a blast to research and uh and record and i really i really hope you guys have had a great time over this past week uh huge thanks to the organizers especially bernie for letting me do this talk and uh thanks brooke for being a most excellent camera person and of course thanks to you uh you keep you keep hope alive through 2020 and far into the future and i'll see you on the other end of this pandemic and when we do i got a cold beer with your name on it

Channel: Zack Freedman

Views: 85,820

Keywords: DIY, hardware hacking, reverse engineering, electronics engineering, electrical engineering, hacking into the mainframe, bad security, worst security, serial port, fbi tracker, zach friedman, zach freedman, zack friedman, zak freedman, voidstar lab, voidstar labs

Id: YP8dZKYMxZ4

Length: 28min 8sec (1688 seconds)

Published: Sat Mar 06 2021

Reddit Comments

[u/rockax]

Can't recommend this guy enough! Every video is interesting, great narration and humor.

[u/beepbingboop]

I found this explanation of the electrical design process really useful for getting started with reverse engineering hardware. Here are my favourite points (timecodes):

• Identifying reference designs in products
• Locating and using access ports
• Penetration of network connected devices

[u/ceojp]

I can totally relate to the frustration of management or marketing seeing a working prototype and wanting to ship that. At work we call this "protoduction".

On another note, we inherited a controller from the company that bought us that is basically an ST dev board schematic with a bunch of IO. It even has the parallel flash chip that we do not use, but it was carried over from the dev board design. I don't know if there were plans to use it at some point so they were afraid to take it off, but to this day we don't use it.