BSidesSF 2020 - If You’re Not Using SSH Certificates You’re Doing SSH Wrong (Mike Malone)

all right welcome to the lightning talks my name is Andy I'm the rim Proctor for this this is Mike Malone he's giving a talk on if you're not using SSH SSH certificates you're doing SSH wrong so Mike really antagonistic hi I'm Mike founder of small step and no this isn't working but my keyboard does nope it's not working either I'm here today to talk about SSH more specifically this presentation is about SSH certificates certificates have been a part of OpenSSH for 10 years but most people haven't heard of them so I'm here to tell you what they are and why you should care so we'll be following a typical three-act narrative form in the first act I'll introduce the monster SSH operations and security will learn that the real problem isn't SSH in general but the way most SSH deployments handle authentication using simple public key authentication and act 2 will introduce our hero SSH certificate authentication then we'll wrap things up with a summary of our findings some code and resources to learn more I got to do all of that in 10 minutes so I'm gonna be moving really fast heads up the URL for these slides is in the footer of every slide first let me introduce myself your narrator again my name is Mike Malone I'm a software engineer my happy place is distributed systems architecture I've made some open source and open standards contributions over the years these days I spend most of my time wrangling kids and running my company's small step which provides solutions for securing distributed systems anyways back to our monster to put it bluntly typical SSH deployments suck for pretty much everyone involved ssh user experience is terrible operating ssh at scale is a disaster and ssh encourages bad security practices most people probably haven't given ssh enough of thought to really even form an opinion about this stuff so allow me to elaborate by walking through a typical new user experience for ssh so imagine you've just joined a new company you've got a brand new laptop and you're trying to get SSH access onboarding starts with some baroque implantation of ssh-keygen hopefully pulled from a run book but more likely cribbed from Stack Overflow most users won't actually understand what's going on and are already confused at this point either that or they'll know enough to be dangerous and do something like copy they're really secure 4096 bit RSA key that they've been using since college from their old laptop to their new one so we're already off to a bad start next you have to give your public key to an administrator to approve and distribute so this process is often pretty ad-hoc usually it involves manually opening a ticket or sending a message to some administrator let's ignore the fact that whatever messaging app you're using here is now critical for SSH security and we'll go ahead and move on so in step three you wait you've initiated some opaque process that's eventually going to result in SSH access but you won't get access immediately and while you ate some poor operator is interrupted to perform some mundane and security critical tasks so for simple public key authentication to work your public key you needs to be in an authorized keys file on every server most likely that means someone has to add your key to a manifest in some repo and trigger a deploy there may be some automation around this process but it's usually shoestring and bubblegum and remember that the security of this entire pipeline is absolutely essential to the security of your system so once that's done the admin will close your ticket or message you back note if you notify you somehow that you now have this SH access and finally you can SSH and you can keep SSH in using the same credentials forever because user on-boarding and off-boarding is typically the only time geez are updated rekeying requires running back through this whole process again so people don't do it removing fuse to be provision at D provision access is also manual so you have all these keys sitting around on hard drives for people like your CTO and your VP who SSH to a machine once a year and if you forget to be provisioned access when you're off board someone you might have end up giving some disgruntled former employee access to prod so all of this is pretty terrible but our new user experience is not done because when you SSH to a host for the first time you get this warning on the left you've probably seen this before if you're like most people you've been trained to ignore it by just typing yes that is a problem because this is a legitimate security threat the onboarding process we just went through configured hosts with our public key but we didn't configure our client to know hosts public keys so this warning is your open SSH client telling you that it couldn't identify the server that you're trying to connect to what you're supposed to do is verify the key fingerprint out-of-band by asking an administrator or consulting a database or something like that but no one does that when you type yes the connection proceeds without authentication and the public key is permanently added to the known host file on your client so this is called trust on first use or tofu it's a bad security practice it's basically the same as ignoring a certificate warning in your browser but everyone does it and God forbid you ever want to rekey a host or reuse a host name later on a different server if you do your clients will freak out because the new public key doesn't match what they expect this is called host P verification failure it triggers a much scarier looking error it's also harder to bypass so it usually results in a bunch of engineers reaching out to CEQA worried that they've been hacked so that's our spooky monster managing SSH credentials this way is bad for users it's bad for security and it's a huge operational time suck but the real monster here isn't actually SSH it's simple public key authentication and the fix is to stop using this authentication mechanism and certificates are the answer so a certificate is just a data structure that binds a public key to a name certificates are signed by a certificate authority so you can trust them and certificate authentication eliminates key approval and distribution so all of the annoying and time-consuming stuff that we just discussed with certificates hosts and clients don't need prior knowledge of one another's public keys to authenticate they only need to know the ca'se public key when a client connects to a host the two peers just exchange certificates certificates can easily be reissued on demand as needed and since hosts and clients both have certificates they can mutually authenticate and tofu and host key verification failure goes away so again certificate authentication was added in OpenSSH 5.4 10 years ago it's battle-tested it's used in production by massive operations like Facebook uber and Netflix and all of the tooling required to use SSH certificates is available today configuring OpenSSH to use certificates is stupid easy on each at and each host you edit sshd config specifying the CA public key the hosts private key and the host certificate on each client you add one line to known hosts specifying the CA public key and that's it that's literally all you need to do to start using certificate authentication you can even use it alongside public key authentication to make transition and easier you will also need an ssh certificate authority to get certificates issued to clients and hosts there are a bunch of existing open source tools you can use for this SSH key Jen can do it you can generate a root certificate and sign user and host certificates bless is Netflix's SSH CA that runs an AWS lambda and uses I am cashier's intercoms SSH CA uber has Pam you SSH that lets you use certificates to authorize sudo use and vault has an SSH secrets engine of course as CEO of small staff I'm kind of partial to our are our own tools which are also open up doors the step command-line tool makes it easy for users and hosts to get certificates from step CA which is an x.509 and SSH certificate authority we have a new release coming soon that will also configure OpenSSH and hosts and clients for you and automatically get a certificate from step CA when you tried to SSH to a host so this enables what I think is an ideal SSH flow so let me show you what that looks like real quick so users just use as such an SSH like normal typing SSH and a host name and we hook into the connection process with a proxy command to check if the user already has a certificate in SSH agent if the user has already logged in and has a certificate there connected like normal if not a browser tab is opened and a single sign-on flow is initiated with your identity provider so users log in with a familiar web-based single sign-on flow which makes it easy to leverage strong MFA and other advanced authentication capabilities your identity provider offers and adding and removing a user from your canonical identity provider also adds and removes SSH access once a user is logged in a new key pair is generated and they get a certificate from the CA using the assertion issued by single sign-on the certificates added to SSH agent and the connection perceives like normal so in this setup your private key and SSH certificate is like a browser cookie it's an ephemeral credential that never touches disk that authenticates you for one SSH session there's no magic here this is all using vanilla OpenSSH with a few lines of configuration the step tool chain makes it really easy to implement this flow but you can do it yourself without this without our tool chain we're just filling the last couple gaps to make it super easy so this all sounds kind of complicated but it's really not that bad and the important thing is users don't need to know any of this detail from their perspective they're just using single sign-on to SSH so step also includes a simple utility that configures an open SSH client to use this flow so when you're on step SSH config step will automatically grab your CA root certificate and configure your open SSH client to use certificate authentication so there are a few obstacles to using SSH certificate authentication unfortunately I don't have time to cut to go over these and running out of time so definitely download the slides for this or grab me later and I can answer any questions you do need to manage view things differently overall though certificate authentication is a better way to manage SSA SSH access it eliminates trust on first use it lets you connect us SH to your existing single sign-on the multi-factor all solution it makes SSH keys ephemeral makes retain easy and access expires automatically so that's it that's the sage certificate authentication makes us is easier to use easier to operate and more secure if you'd like to learn more we have a blog post with a lot of the same info in it check out step in steps yeah and github bit of shameless self-promotion if you want an easy button for all of this we also have a that also solves access control user management audit logging we have a SAS product coming out with this with a 30-day free trial so if anyone wants early access to that find me later I can make that happen other than that thanks so much for having me I'll be wandering around after this if you have any questions but I think I'm out of time [Applause]

Channel: Security BSides San Francisco

Views: 1,189

Rating: 5 out of 5

Keywords:

Id: P-Yq_6Da1b8

Length: 10min 52sec (652 seconds)

Published: Mon Mar 09 2020