"a$$word" LITERALLY SAVED PayPal | Prime Reacts

Shamir secret sharing it's 3 A.M only good things happen at 3am okay my mom told me that when I was younger and I wanted to go out with my friends you know what she said well only good things happen at 3am so you go out kid you go out and you go have fun at 3 A.M nothing bad's gonna happen only good things Paul the head of PayPal database Administration carefully enters his elaborate passphrase at the keyboard in a darkened cubicle of 1840 in bacadero Road in East Palo Alto I bet I bet him I I know this place you were born you were born at 3am 100 true for the fifth time he hits enter the green on black console window instantly displays one line of text sorry one or more wrong pass phrases cannot reconstruct the key goodbye so Paul obviously a lot like me trying to log into my Instagram got a little bit of a boomer moment couldn't get logged in okay totally normal okay sometimes we forget our passwords not a big not a not a big deal by the way this would literally be the most terrifying thing in in the entire universe okay if this happened I have many questions one of them is just being able to just straight up log in with the password this just seems terrifying anyways there's a nerd pandemonium all around us James our recently promoted VP of engineering just climbed the desk at a nearby cubicle screaming god ice if we can't get this key right right away we gotta start brute forcing it ASAP it's Gallows humor wait what he knows very well that brute forcing such a key would take millions of years and it's already 6 a.m on East Coast the first of many why is Paypal down today articles is undoubtedly going to hit CNET shortly oh my goodness just the pure Terror one I mean I've introduced a few bugs at Netflix okay like uh late let's say like here's a good one uh Lady Gaga's billboard for for Netflix uh she had a show come out and if you were on the web when the countdown timer had zero I accidentally created an infinite recursion Loop and everything Frozen broke okay I did that I accidentally did that my fault I'm sorry I'm sorry to everybody I'm sorry to my family I'm sorry I've made repentance before God and and my country it is my fault I did that okay sure all you have to do is refresh the site and everything works but if you were waiting if you were waiting for that exact moment specifically on a Chrome browser at midnight Pacific Coast time the United States of America the only country in the world then guess what you were disappointed you had the refresh and yet here I am still at Netflix okay we've all done that hey we have all done that our single story cubicle maze office is buzzing with nervous activity of pay paleoons paypalions I hate that name uh who know they can't help but want to do something anyways I poke my head above the cubicle wall to catch a glimpse of someone trying to stay inside a giant otherwise empty recycling bin on Wheels while a couple of senior Engineers are attempting to accelerate the bin up to dangerous speeds in the front lobby what the hell is this entire paragraph what I am what I'm hearing described here is that the world's about to explode you apparently hired senior Engineers that are also six-year-olds what the hell is happening here I lowered my head and tried to stay focused let's try it again this time with three different people is the best idea I can come up with even though I'm quite sure it will not it will not work who here hasn't done this three people one password prompt okay who hasn't done this who hasn't been the person that's like you know what it hasn't worked yet but maybe we haven't used enough people let's try it again all the paypalions came in senior engineers in shambles in the front they've given up the ghost right they they've gone completely insane just driving recycling bins as if they're vehicles at dangerous speeds within an office you can tell shambles Shook and shambles the key in question decrypts PayPal's Master payment credential table also known as the Giant Store of credit card and bank account numbers without access to the payment credentials PayPal doesn't really have a business per se seeing how we are supposed to facilitate payments and that's really hard to do if we no longer have access to the 100 million credit card numbers our users added over the last year of insane growth I just want to take I just want to take just a quick moment and I want you to think about the idea of being the person that has to tell a higher up that you lost it all like I just want you to feel that I just want you to feel it for a moment how that must just be this is the story of the catastrophic software bug I briefly introduced into the PayPal code base that almost cost us the company or so it seemed in the moment oh let's go I've told this story to a handful of times always swearing the listeners to secrecy and surprisingly it does not appear to have ever been written down before 20 plus years since the incident it now appears instructive and a little funny rather than merely extremely embarrassing I you know what I haven't told anybody that it was a Lady Gaga's billboard until this one that's like eight years later okay I understand it can get very embarrassing when you screw up uh before we got back to the uh fateful night we have to go back to an another decade in the summer of 1991 classic Summer by the way my family and I moved to Chicago from Kiev Ukraine uh while we had just a few hundred dollars between the five of us we did have one secret Advantage science fiction fans my dad was a highly active member of I don't even know what that is kievs uh possibly first and possibly only at the time sci-fi fan club the name means Star Trek in Ukrainian unsurprisingly he translated uh some uh Stancil Stancil La Lem of solarus futurological Congress frame Fame from polish to Russian in the early 80s and was generally considered the core of I do it okay I don't know what any of these words mean am I outing myself as not actually a nerd at this point or is this just is this just a new level that I am okay not achieving well it's okay we're just seeing it's translated content week yeah I'm weak okay while USSR was more or less informationally isolated behind the digital Iron Curtain until the late 80s by 1990 or so things like phytonet wriggling wriggled their way into the Soviet Computing world and some members of zish I'm just gonna call it zish because I don't know zsh that's obviously not this it's not it's not like it's not like oh my zish right here right it's something it's something else okay I don't know I don't know pick pick what is it uh we're now changing electronic mail with sci-fi fans for free world uh of the Free World the vaguely exotic news of two Soviet Refugee sci-fi fans arriving in Chicago was transmitted to the local fandom before we had even boarded the Pan Am flight that took us across the Atlantic that is such a cool story that is such a cool story right there my dad and I by extension was soon adopted by some kind of Chicago Science Fiction Geeks a few of whom became close friends over the years though that's a story for another time holy cow that is incredible a year or so after the move to Chicago our new science our new Sci-Fi friends invited my dad to a birthday party for a rising star of the local fandom one Bruce schneier uh we we certainly did not know Bruce or really anyone at the party but it but it promised good food friendly people and probably silk I don't know what filk is is that milk what is Phil because that Ukrainian milk I don't know my role was to translate as my dad spoke limited English at the time I had fallen desperately in love with secret codes and cryptography about a year before we left Ukraine walking into Bruce's Library during the house tour this was a couple years before applied cryptography was published and he must have been deep in research felt like walking into Narnia nice I promptly abandoned my dad defend for himself as far as small talk and kind of this okay I'm not a well-learned man all right this is embarrassing we're concerned uh and proceeded to make I don't know how to say these things when I see an oxidegu I don't know what it means okay that's French for a little thingy on top I don't know I don't know what it means you're doing great shut up pick uh and proceeded to make a complete ass out of myself oh I see there's two of us doing that out of myself by brazingly asking the host for a few sheets of paper and a pencil having been obliged I pulled a half dozen cryptography books from the shelf and went to work trying to copy down on some of the answers to a few long-held questions on the library floor after about two hours of scribbling alone like a ma a man obsessed I ran out of paper and decided to temporarily rejoin the party on the living room table Bruce had a stack of copies of his a fanzine ramblings thinking I could use the blank size of the pages to take more notes I grabbed a printout and was about to quietly return to copying the original s-box values for Des what a door when my dad spotted me from across the room and demanded I help him socialize the party wrapped soon and our friends drove us home the printout I grabbed was not a ramblings issue it was a short essay by Bruce titled sharing Secrets among friends essentially a humorous explanation of shamir's secret sharing say you want to say you want to make sure that something really really important in secret a nuclear launch code and database encryption key Etc cannot be known or used by a single friendly actor but becomes available if at least n people from the group of M choose to do it okay okay uh think of two on-duty officers a Cadre of five say turning Keys together to get ready for a nuclear launch okay okay the idea proposed by Addie Shamir the a of RSA in 1979 and as a simple let's see is as simple as it is beautiful let's call the secret we are trying to split among M People K oh we're about to get hit with math pick should I get you on the phone you do a lot of Haskell am I about to like do I need to know Haskell to look at this stuff you know I only write o camel which is like toddler functional programming it's toddler functional programming right is this quick math I don't know quick math no Haskell needed okay I'll try um first create a totally random polynomial that looks like this classic let's create some polynomials okay start with uh power to the end all the way to X Plus Cave okay create here just means generate a bunch of random Co fish and C1 through CN now for every person in your trusted group of M evaluate the polynomial for some randomly chosen X of M and hand them their corresponding x m y m each classic when you really think of it we have if we have n of these points together you can use the language inter interpolating polynomial to reconstruct the coefficients and evaluate the original polynomial x equals zero which conveniently gives us y equals zero which is K the secret beautiful da da da I still had a printout uh with me years later in Palo Alto it should come across the no surprise that during my time as CTO as CTO PayPal engineering had an absolute obsession with security no firewall was too many no multi-factor authentication schemes too onerous Etc anything that was worth anything at all was encrypted at uh at rest okay to decrypt a service would get the needed data from the database table transmitted to a special service named crypto serve the original Sun Hardware running Solaris sitting on its own especially tightly locked down Network okay a and a special service running only there would perform the decryption and send back the result decryption request rate was monitored externally on the crypto serve and if there were too many requests the whole thing was shut down and purged of any sensitive data and the keys from memory until manually restarted okay I I feel like some of those I feel like some of those uh uh crypto Services probably could have used maybe some similar security huh Mt gox right now seething and coping um all right it was it was this manually restarted that nod me at launch a bunch of configuration files containing various critical decryption keys were read decrypted by another key derived from one manually entered path for passphrase and loaded into memory to perform future cryptographic services four or five of us on the engineering team knew the passphrase and could restart crypto serve if it crashed or simply had to have an upgrade what if someone performed a little old-fashioned rubber host uh Crypt analysis and literally beat the passphrase out of one of us that literally could happen right this guy writes well yeah the attacker could theoretically get access to all these important master keys then stealing the encrypted at rest database of all of our users Secrets could prove useful they could decrypt them in the comfort of their underground super villain Lair I needed to eliminate this threat okay I think we're starting to see where this is going I mean I hopefully you saw hopefully you understood what was happening right here clearly this is what's Happening Here the guy created the multi-key turn and then effed it all up and almost lost everything is what it sounds like I needed to eliminate this threat shamir's secret sharing was the obvious choice beautiful simple perfect you could in fact prove that if done right it offers perfect secrecy I decided on a three of eight scheme and implemented it in pure posix C for portability over a few days and tested it for several weeks on my Linux desktop with other Engineers classic Shamir moment just classic just Shamira on the just Shamir on the troubles uh step one generate the polynomial coefficients for eight shareholders compute the keyshards X 0 y 0 through X7 y7 get get each shared Shard holder Shard holder to enter a long secure passphrase to encrypt The Shard right out to the eight Shard files encrypt with their respective passphrases and reconstruct pick any three Shard files ask each of the respective owners to enter the passphrases decrypt The Shard files reconstruct the polynomial evaluated to x equals zero launch the crypto server with keys dude when I read this I think about one thing what happened if six of your eight people are traveling on an airplane together like you could you could act you could ruin your life like this stuff is crazy can't let that happen I don't do it dude it's so crazy you'd actually have to have these people like take separate you'd have to have like backups they would split up there's literally stuff that make sure uh yeah I know there's like there's also key man Insurance there's all sorts of stuff like this it's just crazy that's why you do the same thing uh they do during State of the Union addresses yeah you have to have people out designated survivors One Design detail here is that each Shard file is also stored let's say also stored a message authentication code a keyed hash of its pass of its passphrase to make sure we could identify when someone mistypes their passphrase these tests uh ran hundreds and hundreds of times on both Linux and Solaris to make sure I did not screw up a big Little Indie in this issue very smart man by the way that is like the most classic way to screw things up it all works perfectly a month or so later the night of the key splitting a party was upon us we are finally going to close out our last vulnerability and be secure feeling as if it was about my turn uh feeling as if I was about to turn my fellow Shard holders into cymax again this is a joke I don't understand I gathered them around my desktop as PayPal's front page but again sporting the we are down for maintenance and we'll be back soon message around midnight getting excited the night before I solemnly generated the new master keys and securely copied it to crypto serve now while push it by salt and pepper blaird from someone's desktop speakers the automated deployment script copied Shard files to their destination while each of us took turns carefully entering our elaborate pass races at the specially selected keyboard Paul shut down the main database and decrypted the PayPal credentials table then ran the script to re-encrypt the new with the new key some minutes later the database was running smoothly again with the newly encrypted table without incident you think they would have had a backup copy for this one and only thing in this moment cymac are from Dune really am I saying it right I only listen to Dune I've never read Dune out loud so I've only listened to it I must be saying it incorrect or with cymac and book like book three or longer Clueless is it book three that they get introduced because I'm almost done with book two the telexu is this Duncan Idaho the face oh that guy Tila oh yeah taylax or whatever his name is dude that guy is a weirdo he only like I only have like a small little little picture of him a really weird guy okay all right he like pokes that guy in the neck and then kills the Sun is a a Gola that's right he has an Angola uh dude we're excited uh apart uh you take us that way hold on you take us on side trips sorry sorry okay my bad my fault okay here let's keep on going all that was left was to restore the master key from its shards and launch the new even more secure crypto service the three of us entered our passphrases to be met with the error message I sh I haven't seen in weeks sorry one or more passphrases or one or more wrong passphrases cannot reconstruct key goodbye his own error message his literally own error message in his own face right there surely don't call me Shirley one of us screwed up typing no big deal we'll do it again no dice no dice again and again and even after we tried to numerous combination of three people necessary to decrypt minutes passed confusion grew tensions Rose rapidly there was nothing to do except hit rewind to grab the master key from the file uh still sitting on crypto serve split it again generate new shards choose passphrases and get it done not a great feeling to have in our first launch go wrong but not a huge deal either it will be okay in a minute or two a cursory look at the master key file date told me that no it wouldn't be okay at all the file sitting on crypto serve wasn't from last night it was created just a few minutes ago oh no no you overwrote it there was no copies you during the salt and Peppa pushed or theme pushed from stage we rewrote the master key file with the staged version whatever key that was it wasn't the one I generated the day before only one copy existed the one I copied to crypto serve from my computer the night before zero copies existed now not only that the push grips appear to have also wiped out the backup of the old key so the database backups we have encrypted with the old key are likely useless kid wrecked I moved to the Bahamas after this yeah I'm going straight Bahamas sit rep we have eight sharded files that apparently cannot use to restore the master key and zero master key backups the database is running but its secret data cannot be accessed I'll leave it to your imagination to conjure up what was going through my head that night as I stared into the black screen willing the shards to work after half a decade of trying to make something of myself instead of just going to work for Microsoft or IBM after graduation I had just destroyed my first successful startup in the most spectacular fashion still the idea of what if we all just continuously screw up our Master passphrases swirled around in my brain it was easy to check to perform it was an easy check to perform thanks to include Max I added a single print F to bug statement Into The Shard reconstruction code instead of printing out the summary error of one or more the code now showed if the passphrase entered match the authentication code stored in The Shard file oh no oh this is so good I compiled the new code directly into crypto serve or directly on cryptoserv and direct contravention of all reasonable security practices what else did I have to lose entering my own passphrase I got prompted bad passphrase error I just added it to the code well that's just great I knew my passphrase was correct I had written it down on a Post-It note I had planned to rip up hours ago another person same error finally the last person JK entered his pass race no error the key still did not reconstruct character uh correctly I got the goodbye but something worked I turned to the engineer and said what did you just type in that worked after a second of embarrassed mumblings he admitted he chose the password ass word [Applause] word saved PayPal take a moment here and realize that ass word saved PayPal oh [Music] the guy the gall I asked everyone entrusted with the grave task of relaunching the crypto serve to pick really hard to guess passphrases in this guy still this was something it worked but why I sprinted around dude how much do you want to bet before looking on this is gonna be some sort of stir and buffer length problem how much do you want to bet that only small passwords worked and everyone's password got cut off to a certain length because when they were testing it they all just chose small passwords how much do you want to bet that was it that's my guess that's my guess right now so okay we got some dollars in there okay we got my guess as well I bet you that uh I sprinted around the half-lit office grabbing the rest of the shareholders demanding they tell me their passphrases everyone else had picked much lengthier passwords of uh yes yes I manually tested each one and none decrypted correctly except password what was it a lightning bolt hit me and I sprinted back to my own cubicle the far Corner unlocked the screen and typed in man get pass oh yes on the command line while logging in the crypto serve in another window and doing exactly the same thing there I saw exactly what I needed to see today should you try to read up the programmer's manual AKA The Man page on get pass you'll find has been long declared Obsolete and replaced with a more intelligent alternative in nearly all flavors of modern Unix back then or but back then if you wanted to collect some information from the the keyboard without printing what was being typed onto the screen and remained posit compliant get pass did the trick because he did a posic posix only C implementation did pass did the trick other than a few standard final manipulation system calls git pass was the only operating uh system service call I used to ensure clean portability between Linux and Solaris except it wasn't completely clean plain as day there it was the manual pages were identical except Stellaris had a special feature any passphase entered that was longer than eight characters was automatically reduced to that length anyways who needs long passwords am I right oh come on s word dude ass work did it oh ask word saved the day no way no way I screamed like a wounded animal we generate the key on my Linux desktop and entered our novel length passwords right there attempting to restore them on the Solaris machine and they were being clipped down to eight characters long would never work except of course for ass word that one was fine the rest was an exercise in high speed coding and some entirely off-protocol file movement uh moving we reconstructed the master key on my machine and all of our passphrases worked fine copied the file to Solaris running crypto serve re-split it there in very short passphrases reconstructed it successfully and PayPal was up and running again like nothing ever happened by the time the unsuspecting colleagues rolled back into the office I started to doze on the floor of my cubicle and that was that let's go when someone asked me later that day why it took us so long to bring the site back up I simply responded with EH should have read the [ __ ] manual um RTL rtfm let's go let's go rtfm indeed P.S post scriptum a few hours later John our general counsel stopped by my cubicle to ask me something the day before I apparently gave him a sealed envelope and asked him to store it in his in his safe for 24 hours without explaining myself he wanted to know what to do with it now that 24 hours have passed ha I forgot all about it but in an about of what if this doesn't work paranoia I printed out the base64 encoded master key uh when we generated it the night before stuffed it into an envelope and gave it to John for safe keeping we shredded it together without opening and laughed about what would never actually been a company-ending event what a smart guy dude what a smark I forgot he even had this and it turns out he had a backup printed version post post scriptum if you're thinking all the ways this whole SSS I don't know what SSS stands for I'm sure there's a great thing design is horribly insecure it had uh some real flaws for sure and planned to poke around PayPal to see if it might still be there don't while it serves us for uh well for a few years this was a very first thing eBay required us to turn off after the acquisition pretty sure it's back to single passphrase Now notes one a member of the Chicago land sci-fi Community let me know that the original news of our move to the US was delivered to them via posted letter snail mail not Fido net email a hand Written Letter sent guys this was this was the best article I've ever written in my lifetime or read my lifetime that was is Shamir smarter than Tom no that was the best article I've ever read my lifetime that was fantastic the link will be in the YouTube video of course for any amount of sharing beautiful here's the link now absolutely outrageous the name is the Shamira gin look at that okay this is what makes PayPal databases drop

Channel: ThePrimeTime

Views: 285,148

Keywords: programming, computer, software, software engineer, software engineering, program, development, developing, developer, developers, web design, web developer, web development, programmer humor, humor, memes, software memes, engineer, engineering, Regex, regexs, regexes, netflix, vscode, vscode engineer, vscode plugins, Lenovo, customer service

Id: MzescXc5SW0

Length: 27min 56sec (1676 seconds)

Published: Sat Aug 05 2023