DevCentral Connects: Troubleshooting SSL & Certificate Management

by the way greetings from germany i would just say guten tag marcus or maybe guten abend at this point in the uh man it's i feel like we're just one big happy family right now hanging out right all right welcome to dev central connects this is episode what number four already this is number four man the quattro the quattro for those that speak espanola jason and i love that you're hanging out with jerry seinfeld and uh the kramer cosmo cramer is that right is that the is that the living room that you're sitting in today it is you know uh jerry himself said you know i would love to have you over in this time of social distancing please come hang out in my living room with me but i won't be there it looks like yeah i was gonna say it looks like you're the only one there so that's good you're practicing you know the normal social distance that's yeah i'm doing my best but yeah so patrick i don't know if patrick's on today but you know he said my living room was was was pretty nice and so i thought you know maybe we'll maybe we'll cycle some different living rooms here but yeah we got a good show for you today we're going to talk about ssl um but before we get to that today is bring your kid to work day and here at f5 they're having an entire virtual day and so you know in honor of that i'm going to bring two of my kids on there they all couldn't be with us today but andrew and emma look at that yes man all the robs representing today emma emma is my big f5 swag girl she she likes to bring all of my you know like f5 stuff what i go to she's like dad can i can i have those dad no i really want that so she's rocking all the f5 swag at f5 virtual day emma what's your favorite activity you've done today so far at f5 uh f5 virtual workday definitely storytelling i'm super excited about that um we i've already done the one but i get to do part two later today and i'm preparing a sales pitch so i'm excited about that nice tell give us give us just a hint miss emma what what are you going to try to sell this afternoon i am selling a one hour a day time of um going outside so oh nice you know what you know what i can do that for free well it'll be a good sales pitch i think the i think is this is the sell on that though that you're gonna you're gonna encourage people to do that you're going to tell them how beneficial it is to go outside for an hour so hey yup don't sit at your computer all day long at least take an hour right yeah i love it man i love it all right so andrew andrew is my mini me you know it's like he's he's birth he in birth order he's like number four so i figured like my genes perfected their craft over the course of my children and so so andrew how are you today i'm doing well how about you i'm fantastic how's your f5 experience been today who is your favorite host in any of your sessions so far okay so i had a lot of great teachers but probably my favorite so far has been mrs boddy um because she elevated me and because she knew me so apparently i featured in a lot of your meetings for some reason oh because i all the time even when i have a in a meeting sign up all right well i know you guys are in the middle of your sessions so we're going to let you guys go thanks for coming here to be part of f5 bring your kid to work day enjoy the rest of your sessions guys thanks for having us thanks for coming to f5 rom kids man looking good looking good emma rocking those shades and the hat and the whole thing man i love it i love it all right so we got some comments out here we got uh dalip kamar let's get started absolutely we're getting moving and uh hey jason and john from grown up uh shin-chan um hi all hello back to you and um all right so oh yeah so yeah i saw my daughter's so i we've been sharing as part of this uh uh bring your kid to work day uh they're having a a thing to where if you share out your pictures uh maybe that uh an in-house artist is going to sketch you up so you know i've been posting stuff yeah nice yep uh that's right tony he is my mini me sure when i was like jason when i was looking at the screen a second ago you were you know you were on the top and then andrew was just below you and i was just like wait a minute that's you know big jason little jason old jason young jason nice that's right akash hello welcome to the show yeah good to see everybody out there today all right so we do have an ssl question in here but we'll cue that up a little bit later john how's how's life going for you in this uh in this pandemic you know what i'm uh i'm sitting at home just like everyone else in the whole entire world and um you know but like emma just said a few minutes ago it's good to get out for an hour or so a day so i'll try to go take a walk or maybe go for a little run or i uh my actually my next door neighbor he had a basketball goal and he let us basically have it it's one of those little portable kind of things his kids are older and moved out and stuff so me and my son have been out there shooting some hoops and i'm doing my best to you know every time he goes up for a jump shot then i just come in and just bam you know hakeem elijahuan like not in my house so i'm trying to start him early you know trying to get the uh you know trying to encourage him let him know what life really looks like no but it's been good it's been good it's a good deal yeah we're you know we're we're doing our thing not not a whole lot going on in in that realm we have been trying to get some outside time and you know it's raining today and we've got uh i guess another three days of rain forecast so we'll be inside for a little bit but yeah well that's that's the way life goes right now you know so it's uh but you know it could be worse we're all sitting here hanging out on dev central connect so it's good good to kind of be with everybody good times all right so what do we have on tap today well i tell you what for those that uh have ever heard about a little thing that we like to call encryption or ssl or or maybe more more modern than that tls we still say ssl around here sometimes like profiles and all that but it's all tls uh that's what's that's what today's show is all about it's all uh it's all tls all the time we're just gonna encrypt the entire thing um so hopefully you brought your decryption keys but i was gonna i was gonna kind of set this up a little bit we've got a couple of amazing guest speakers uh today one is rodrigo albuquerque from the dev central team he lives out in london area and then we've also got um the uh the foremost expert on ssl certificates peter um mr encryption scheffler and we'll have him on here in a few minutes after rodrigo but i wanted to kind of set the stage really quick i was looking at this uh this new fancy finagled website called google and apparently they have a uh they have this encryption report this transparency report is what they call it and they they um they basically analyze and you know and and look at and count and all that all the different uh encrypted pages that come across their site which of course are many many pages um so what they have done is they not only have encrypted all of their properties so if you've got a gmail account or a youtube account or you know google itself or whatever um then that is an encrypted experience almost completely across the board for google uh there's some google maps pages that are not quite fully encrypted um and there's some google news sites that are not fully encrypted yet but for the most part they're in the very high 90 of everything that google produces is encrypted right and then they also look at the top 25 sites i'm sorry not the top 25 the top 100 non-google sites that traffic flows across on the internet and their statistics show that 96 out of a hundred of those other non-google top 100 sites 96 out of 100 default to https to the you know ssl tls encryption right but all 100 of the 100 will work on https so the point with all that is everything is moving toward encryption right so if you uh if you're on the internet today you're you're using an encrypted channel an encrypted experience in some way shape or form and so the things that we're going to talk about today are extremely relevant and extremely timely and so people need to understand what does it take to make this encryption happen and then more importantly you know what are the what are the parts and pieces of f5 that come into play here and how can we how can we look at that traffic if it's all encrypted how can i make sure my certificates are up to date how can we do all that stuff you know so it's uh i think it's going to be a really exciting session today i think so too and and you know for everybody here watching wondering what we do with your questions uh both of these segments that we're doing today a result of questions we got on the first show and so you know don't don't be afraid to throw something out if you have questions so we're going to jump into it here in a second i did want to mention you know we're pretty new at this and so we're trying to find you know the time that works best for you and so you know we have a two question survey on you know what what day the week works best for you what time of the day works best for you if you want to um you know peter's peter's gonna drop that in in youtube chat here in a second and uh and then you can go fill that out and then we can kind of look at you know if this isn't the best time for you guys to join us live but you would like to be part of the live audience you know we'll we'll try some different time slots and see if that works out better for you guys and so uh you know as john was mentioning our first guest today uh mr rodrigo albuquerque we're going to talk about ssl troubleshooting and so this this comes from uh the question we got in the first week about hey i've got ssl running um and i'm trying to i'm trying to pass the 301b uh exam and i'm having trouble doing that because of the ssl profiles and the wireshark and all of that stuff so you know we brought mr wire shark himself rodrigo albuquerque um in to share some of that goodness hello everybody um today we're going to talk about what i used to do when i work for engineering services at f5 when it comes to ssl troubleshooting so i'm just going to show you firsthand what it looks like to troubleshoot some ssl eq i mean we're not going into troubleshooting an issue itself which i'm just going to show you the process itself really quick so um are you can you see my screen i'm yes so i'm sharing oh so that's the topology that i'm currently using so i've got one big ap in the middle i've got a client and i've got two back-end servers so my my virtual server is 10.199.3 so basically that's the topology that we're going to use and for the certificate keychain that is applied to the to the uh client ssl profile i've just applied a default uh pass certificate in key which is obviously just for the for the sake of testing and that's it so let me just jump on to the let's jump on real quick to the big-ip so basically that's my big-ip i've got one virtual server here as i said and in this virtual server i've got a profile applied which is this one it's a client ssl profile with the default key and i actually uh excluded the dhe and uh diffie-hellman uh cyphers uh because that's one of the things that i do in when i started uh with uh troubleshooting because we didn't have the uh capability of uh decrypting uh diffie-hellman key exchange now we do with eyebrows which i'm going to show you as well and the other thing that i i used to do was to set i think i can find it here cache yeah it's actually not set but i used to set the cache size to zero as well because uh the once once you're trying to decrypt something to display on wireshark what happens is if it's not a full cls handshake then there's no way we're gonna grab we're gonna be able to decrypt it so uh let me bring over my terminal window hopefully that's going to be big enough yeah while you're getting that set up rotated can i ask you a question what what does uh moving your cash size down to zero i mean would you want to do that like full time on your profile or would that be just in the in the temporary time where you're trying to do a little troubleshooting yeah so before we had other ways of of decrypting uh a client traffic we the setting the cache to zero would effectively make the ssl uh so it would affect the effect effectively disable ssl session resumption so ssl session resumption is after one session where you have the full tls handshake the big ip or the other side is going to store these ssl caching ssl session in in its cache locally and then the next time we have another handshake we're not going to exchange a session keys again we're going to reuse the previously negotiated session key so if you if you happen to capture not the full handshake but only the session that has been resumed you're not going to get the this is not going to get decrypted because obviously we would need to see the full handshake to see the pre-master key so that the session can be decrypted on the big-ip that's the way it works because we're not exchanging the the keys so we're not able to see it and decrypt it even though we've got the private key yeah okay okay so uh now i'm gonna do a tcp dump to show you guys oops since it's gone it's gonna be back up here so i'm gonna go tcp dump i'm going to show you real time uh a session actually i'm gonna present you a new uh way of that we've that we introduced on version 15 of decrypting traffic on the big-ip uh so basically this is a tcp dump syntax so basically we are capturing traffic on the tmm which is our forwarding plane we are making uh all the headers f5 headers available for us to visualize and the p flag means that we're going to capture the flow that is specific to this ip address here the new thing that we introduced on version 15 is the minus minus f5 ssl up parameter so basically what that's going to do is that's going to copy the master key to the to our ssl packet capture here this one that we're actually doing this so i'm going to hit enter you're going to see there's a warning here caution just just just to mention that the master key is going to be the master key is going to be copied to the capture then we're gonna go to the client this is my client i'm gonna do just a quick test uh and and just uh just to interrupt you real quick while you're while you're getting that typed out john what is the the big caution they give you the caution kind of there in that warning but you know what what's the what's the danger in doing that yeah well so you're using i know when you when you uh configured the the cipher suites on the big-ip you know you said don't use diffie-hellman diving with ephemeral all that stuff you're so you're effectively saying hey let's use rsa so the key exchange you know can have a have a master uh secret you know a private key that can then be shared and if you share that private key that's the keys to the kingdom as it were you could you could take that and decrypt all of this stuff so while it's really cool for troubleshooting you would not want to just hand that out to anyone and everyone i mean that is the that is the thing that is is causing the encryption to be you know secure um it's the thing that would allow you to decrypt so you wouldn't want to just hand that out to anyone that's why they're saying hey warning you know warning you know be careful with yeah be careful with how you store that capture and and who you give access who you give access to right right right yeah sure so if i hit enter here you're going to see that i issued an http request obviously that's encrypted because it's https and we're going to see that big-ip capture 42 packet i'm going to hit ctrl c here so we already have that actually previously so we can just gain we don't waste much time i've already done the i've already captured another packet capture before that one so but the thing that i want i want to show you is for example let me just jump into the capture which is exactly the same as this one uh yeah because in that tcp line right the the syntax there you were writing the output of that tcp dump to apply to a file and then that file is what you're going to pull up and wire shark so that you can see like what you just dumped right exactly very cool yeah so let me make it like slightly bigger for you guys to see it right so we can see here uh actually that's not encrypted so but i'm going to show you what i want to show you that the f5 uh trailer protocol and actually that's something i need to show you on the go here if you click on analyze and enable protocols you can type in that's actually that's really small but i'm typing in f5 and you can see here f5 tls right normally that would be unchecked so you need to make sure that it's checked yeah so you can see the f5 ethernet trailer protocol so if we expand that we're going to see one header here called f5 tls and that's the sweet spot here because if we move through some packets let me just reach a good one yeah there's your yep but there is one that's got all of them i guess no none of them has all of them so basically we have client random server random so what we're going to care here is about client random and also there is one that is the master key here so the master key yeah so that's the thing that is going to make us decrypt the packet capture so what you can do here is you can right click on it you can click copy and then value and then you copy the master key and then you go here you you paste it i'll show you where you're going to paste it you you copy also the you also copy the client random the value and then what are you going to do here is i i actually have it here so i can show you so we don't waste much time uh it's actually i think it's ssl provider dot yeah so what you do is you you type that you type in client random just like this and then you first paste the client random and then you paste the master key and actually you're going to do this for both because we had two connections here right so let me show you we had the first connection which is the actually that's the client side and then the server side so we have two connections so if we want to create decrypt both uh you're gonna get you're gonna you want to have to get the value from the master key from both connections so what we're gonna do here is we go to wireshark and we go to preferences that's going to be fairly similar in on windows or other or linux so you go to protocols you expand protocol and you look for tls that's a lot of protocols rodrigo yeah marshall small man yeah okay so we've got pre master secret log file name you click on browse and then once you get there you're gonna load that file ssl provider where is it uh which one which file was it was ssl provider.pms i guess yeah so it should be uh jesus where is it oh over here so when i do this and i click ok you're gonna see the green bit here that's the decrypted thing so basically everything that is green is the http traffic so if you click on the http traffic let me just you're going to see my head request had us and we actually see that it's decrypted because we have the tls layer just above so we have http over tls yeah so that's that's pretty sweet so the thing my point here is that f5 made it really easy so before the only thing you have to do is to add that uh to add that parameter that i showed you minus minus f5 ssl to your capture the master key and the client random will be copied to the to the uh packet capture itself and you won't need to ask anybody to like john said which is dangerous uh sometimes we would need to copy a private key to somewhere else try to decrypt it that kind of thing you don't need that and the other thing that is really cool do i i can show you guys if we have some if we still have time i can just show you i don't need to show it it's not going to take you take that much long but i'm going to show you real real quick here if you go to there you've got about five minutes left yeah so it's cool yeah so on resources on our on on the so that's the virtual server that we that i'm running here there is an eye rule called tls kls decryption so what the i rule does is there is something that i didn't show you when i took that capture actually i need to show you right now which is real nice so if i bring that over here these are the big ip logs from that machine and if you see here the master key actually is on the tmx where is it master key yeah so here's the master key it's in actually in on tmm logs sorry ltm logs so we can that's another option we we can also use eye rules to to display the the the master key so we can copy that to the to a file and then we can decrypt the file so that's that's the first thing that i used to do when i that's the the thing that i used to do when i used to troubleshoot issues uh with our customers so we were most of the issues they they were related to the ssl handshake like i would say 99 was because the ssl handshake failed or some s cell handshake failed like inter intermittent and we need to work out what was happening and every time we would need to take a packet capture to see what's going on because uh when ssl fails they usually there's usually a kind of message called alert and that message is going to give give us a clue so we can uh understand what really was the reason why it failed so that that that's that's pretty much it i just want to show you one one quick thing in terms of troubleshooting real quick like literally a couple of seconds so if we go to the ssl profile there is something really interesting here called generic alert so it's enabled by default because that's actually a protection we don't want an attacker to know that uh if they are able to sniff our traffic we don't want to know what the reason why there is a handshake failure but actually if we are troubleshooting something you want to disable this because this is going to tell us exactly why the ssl handshake fails like is it because of an unknown ca that we don't recognize or anything like that so that's that's that's yeah so that's roughly it so i want to show you that new new feature the minus minus f5 ssl parameter on tcp dump i think it's rodrigo real quick um anis is looking for can you please show how again how the file looks like i'm i'm assuming um i've asked but i assume that's uh the key um your client random uh the format for the client random and master key yeah got it yes i can do i can do that so that's it so you basically you just type client underscore random and then this is the client random value from the wireshark capture and this is the master secret value you just paste it one after the other actually yeah so it's based i pasted it one after the other so okay yeah that's that's how it looks like and if we go to the trailer you just need to look for it here like flat random which is here copy it and the master key as well oops yeah i think he should be here yeah we got another question uh for that uh flag is very handy um that will make it into my toolbox that was not actually the question but it scrolled right as i clicked the question from daniel was in what version was generic alert introduced i can't remember but that was really early because as far as i'm aware i've uh i can't remember not having that generic alert option at all so you can you can never remember a time in your life rodrigo where you did not have a generic alert available exactly yeah so hopefully hopefully daniel's got that and i don't know what version he's on but hopefully it's there all right i'm going to pull your screen off sorry yep real quick i'm going to pull your screen off and i'm going to bring chef in just a little bit early because we have a few questions i thought maybe we could work through and then we'll move into uh to chefs commentary uh first of all peter scheffler welcome uh we'll give you like a full awesome introduction here in a second but i wanted to go back to the beginning of the uh the session um we got a question from uh sajeeth uh connection air ssl underscore hs underscore rx hello unsupported version 70 how to fix this issue these servers pass through yeah so basically oh that's not right so when when i had that issue when i used to troubleshoot that i used to obviously request a packet capture but that usually means that you're negotiating a tls version that one of the sides doesn't really like or doesn't support uh there are there may be bugs as well so basically um let me sh oh no i can't can i share my screen again though or is it yeah go ahead yeah so i'm gonna show you the handshake uh the ssl handshake here real quick so if we go to the ssl handshake we see in the client hello i'll show you the tls layer so we can see something really weird here right we see the tls record layer which shows us tls 1.0 and then we see the handshake protocol client hello which is gls 1.2 so basically that's the way the client uh signals to the server which versions it supports so we what we're saying here my client is say is telling big-ip that it supports tls version 1.0 1.2 and 1.2 sorry 1.1 and 1.2 so what happens is sometimes there was a bug a really like that was an old bug on big-ip where we would just send tls 1.2 over here and then tls 1.2 here and some some servers that was on the server ssl site some servers they at that time they didn't support tls 1.2 and that's exactly where i saw that error where the server said well i'm sorry we don't support tls that tls version and yeah so basically that's it okay thank you and we also had a question on https monitoring between an xia and f5 ltm is failing due to either handshake failure or lt i'm not liking ixia certificate any suggestions on how to make it work um yeah i definitely would want to see a packet capture because probably something to do with again it could be a handshake um it could be a the product a protocol mismatch that maybe some of the um like i without a without a packet capture i'm not sure exactly what the option what the problem would be but um yeah i don't know yeah and yeah it'd be good open a question on on on dev central and uh submit you know ixia inversion and ltm inversion because i know that in later versions of ltm there's some protocols by default with the default ssl client ssl profile that it doesn't support like ssl 2 or any anymore and so you know you open a question you should get help pretty quickly on that per day and then okay so i think we have some more questions uh related to certificates or at least one more in here earlier that we'll get to with you chef uh rodrigo thank you so much for joining us today i know that you're ready to get to your evening there in the uk you're already uh late late uh i guess early evening but uh host work day so thanks for joining us man it's awesome thank you thanks for being here rodrigo always a pleasure my man my pleasure all right take care and so chef we've already brought you in so you know we'll we'll you know post introduce is that even a thing uh but peter scheffler is on our f5 team and you're in sales and he has um enjoy or he's uh volunteered to join us to talk about certificates today and we have certificates that are uh supported on ltm from a management perspective you can manage them on big-ip you can manage it on big iq and uh you know but not just where they're supported but what types of certificates are supported the packaging all that we've gotten lots of questions from that in earlier episodes and so we brought in the expert to talk to us about all things ssl certificates so peter welcome thanks very much um yeah i'm excited to be here um and um yeah so i guess when we we sort of went through the the the thought process for what we would talk about today um there were there are a lot of questions and interestingly enough i i just finished a a discussion with a customer um not even a half not even an hour ago um or maybe just an hour ago uh on this exact topic um completely unrelated to the fact that this is this was coming up here um and it's it's a it's a point that comes up a lot customers are asking you know um there's been a lot of change in a certificate technology over the last little while with the introduction of you know ecc over the last four or five years now um and and the adoption of that um really you know obviously mobile has driven that because the the old world of having big chunks of hardware to do the decryption on the client side um you know it's expensive uh eats a lot of battery so people have been you know moving towards uh you know ecc and away from uh you know from the from the rsa world or maybe the the dsa technology as well and um and then a lot of our customers are are going through a certificate glut um the the average organization you know has you know as many as 17 000 you know an average organization in the in the united states has about 17 000 different certificates um which is a mind-blowing number 17 000 chef that's crazy i can't even i don't even think i've ever counted that high in my whole life i don't i wouldn't want to count that high yeah but uh it it's it's kind of cool if you have fans oh yeah i do have if you can see them yeah carolyn man those guys are awesome check out their channel as well it's pretty good yeah um no but it's like and and the conversation that i have with the customer or just before uh was really on how to manage this so they were looking at using an hsm so if you're not familiar with an hsm it's a it's a it's a it's a hardware device that um that stores the the the keys right so the keys are generated on on the hsm um and then you don't pull them off of there so so it's a it's a it's an encryption technology so that your keys are secure and and one of our challenges today is just making sure those are those that are secure but then we have to you know distribute them all over different devices and we have to maintain them and and how do you how do you how do you automate that and how do you manage that so a lot of the time that that we've been spending over the last few years is um is is to kill the f5 ui um i you know i i i um i used to have a friend or i still have a friend but he used to work with a friend and he was a documentation uh guy and his whole uh mantra in life was to get himself out of a job because he thought if a ui needs documentation then it wasn't designed properly um and and so now we're moving the next step where we want to get away from having a ui completely and and we want to automate things so it's easy so what we've been able to do now is is automate a lot of the certificate management so um we now have a protocol called acme so um it's it's actually a standard uh so acme ii came out uh recently and it's a standard that allows us to uh issue and um um or issue certificate requests so that's a hey i'd like a certificate um here's my certificate can you go validate it for me um and and get it get a trust relationship built automatically for that um and so the the acme protocol is is something that's really making this much easier for us to to scale um so that we don't have you know nine ftes worrying about these 17 000 certificates that have to be renewed all the time um and the other problems that we're seeing now is um if you're familiar apple just came out with a new standard saying they will stop uh allowing uh apple devices to talk to certificates that are 13 months that have a a lifetime longer than 13 months um yeah like like that even makes it harder because i don't know if you guys remember but remember when we had 10-year shirts on our device certificates right i mean both days are gone um that's uh that those are those are crazy crazy long numbers because once that certificate's out gets out it's always out right even if we have a certificate revocation list or uh or or you know um we're we're we're validating the certificates we there's a chance that certificates aren't validated so we want to we want to lock those down as much as we can and as customers are adopting go ahead yeah chef if i could just kind of just kind of wrapping my mind around this whole thing you know you mentioned that there's 17 000 on average in these different organizations which again is just i'm still trying to figure out what kind of number that is it's crazy huge but then when you add on to a layer of that if i'm thinking about this correctly i've got 17 000 let's say i've got to deal with but now if they expire more than 13 months from now it's a problem with all these new apple devices or whatnot so that may be kind of the way the world's moving so now not only do i have to handle seventeen thousand of them i gotta i gotta i gotta like turn these things over super quick right that's a huge thing that's a what a headache man well google's going to make it worse for you because they're going to force 90 days 90 days 90 days thank you google yeah and that's the way the world's moving because some people would say well hey forget it man i'm not going to play ball i'm just going to do my own thing it's like you can't you've got to you know when when the world moves that way you got to just move with it you know yeah yeah so it's going to be possible so there's no way to do it other than automating yeah sorry go ahead yeah yeah i know i was going to say i think even you know we you know we can thank google for that on the browser side but i think google can think and they're just adjusting to you know the let's encrypt explosion right when let's encrypt came out um with the automated process which you could do either via http or dns uh to validate your domain and your your urls and and uh and push those i mean their their max was 90 days and so you know that people adjusted to that and they they built in the automation and you know i use let's encrypt in a couple of my personal product and it just works and it's it's fantastic and so you know everybody adjusted to that pretty quickly and maybe not the enterprise but but certainly all the uh the small potatoes out there that that hey this is a this is an awesome service and it's free and i want to buy to that and and so you know google adjusts to that and then everybody adjusts to google right so yeah yeah well i think i i think what we want to do in in in you know for you know this is a this is a a dev central conversation so we definitely want to drill down on what we can do on the on on the big-ip side um and and what we've been able to do is um in in some of our later releases we're now uh we we use the acme protocol uh in as a means to go and generate certificate and validate certificates um for the device so you can automate that so you can get away from having to generate and and you know manually load certificates um the one customer i was talking to this week um you know they're like okay you know our developers go develop an application um they they test devitt um they move it into you know a staging environment and they deliver a pfx file to us and it's like you know you can see the zip drive just or the or the or the the usb drive moving from hand to hand in this whole process you're like wow okay so we've got the whole development process automated but now we've you know now we're now we have to put keys in there oh i'll just put my key in my in my uh in my github repo oh wait wait wait wait what did i just say there be dragons right um that's that's not going to be a something that we we want anybody to go to so we we need a way to to automate that and again so the being able to to do this through an automation process and and we can do that with big iq um and big-iq has you know the ability to to to warn us uh before certificates expire um so we can we can even have a you know if we want to get the ui uh component to that we can we can see that uh also we can then automate the the the renewal of that so if you um uh if you use uh uh komodo or i keep want to call it gizmodo but it's komodo or digicert um uh then you can actually use our uh our management so we have a order certificate order management uh capability now in uh in 15. oh and and and newer and that uh that we actually can um use to uh to use the acme protocol to just automatically re reissue that certain so you can type in how many days do i want that certificate to be um so i want it to be 90 days or 85 days or whatever and then that certificate can can can be renewed and and the the the profile the ssl profile uh updated automatically um and i think we we we need to get there um because you know at the at the the crazy cadence that we're seeing now it's it's just it's not gonna it's not gonna be possible um and then we have some integrations with other tools too so um there's there's venify um which is uh which is a certificate management solution um so we've got some we've got some uh uh business development with them so um you can use big a big iq to talk to venify um so venify would be the the interim certificate of uh issuing an uh device and the advantage there is now you don't need to worry about what the nca is you can use your you can use your own ca um you can use whatever cas are supported by venify um and you get the get the wealth of that and then the big iq can push that to all your all your different uh data centers and and you can you can maintain that so now what you used to have a bunch of people reissuing certs probably on an annual basis like you know that's probably what we were looking at you know you know four or five years ago um now we can have this uh this automated process um and there's several steps in the chain that we have to worry about but that that makes life significantly easier yeah so from a packaging perspective for for certifications we do have a question um from daniel uh on you know do does microsoft pki support acme you know uh microsoft has its own implementation i don't know if i saw microsoft's ca as as supporting uh acme now um i'd be surprised if they wouldn't be in a not too distant future acme 2 was just ratified late 2018 so um it could be an issue that you know maybe they're it's coming but i i'm literally just throwing words out here i'm i'm not i'm not sure and i know dan pretty well so maybe i'll reach out to him and give him an answer nice nice hey so i guess chef from uh from a big picture perspective i'm kind of putting my i'm cutting my i'm putting myself in the in the shoes of the person who's got to manage these 17 000 certificates they're going to be rolling over every 90 days so it's just you know it's non-stop kind of deal and i want i would just want to hit the big fat easy button that's just like boom automate the whole shooting match right just the whole deal is is completely so is it fair to say i guess from a big picture perspective is it fair to say that we're not quite there yet where you can just kind of ronco 2000 set it and forget it um but but you know so so we're not quite to that point yet but we have made steps and we're headed that direction um you know where there are certain things that are automated but it's not like the entire you know the entire game is that fair yeah yeah your ronco it slices the dices but it doesn't put it in the pot maybe that's right actually right you still need to put it into the oven and then you can forget it right or that kind of a thing yeah but i think i think the problem with the with with this infrastructure with the challenge that we're seeing now um is we have so many things in this chain um yeah and there and previously everything was proprietary i mean we really didn't have um a means to to to automate anything prior to to acme it was a lot of hand coding and um jason i see you've got a devs you've got something on dev central where you've got a you've you've i think you've got an active in integration with with the big-ip it's you know that's that's that's you know probably a year or two old or something like that so we're just now to that point where we can build these things together um but we're not i don't know if if if the maturity of all the pieces are there yet for us to get there and and i and i and i sense the frustration of people when i tell them this because they're they they're like oh you're oh you oh the full answer also goes back to a lot of um layer eight issues too how certificates are managed inside of a company is very different um you might have the smaller companies you know maybe maybe with you know a couple hundred people in them where you have one person or a small group of people that manage um enough of all the infrastructure pieces that it's automatable because they they know all the different pieces but once you get beyond that we now have different teams that are worried about that we have governance and compliance issues that that come in because the certificates are are essentially you know uh ip of the company right and and you need to lock that down um and so there are um operational hurdles in there because they're operational hurdles they actually need to be in there because of the gnc requirements so some of sometimes the automation becomes becomes a challenge so and i think what we're seeing right now is some of the larger organizations getting getting through that but i don't know if we've got all the answers but like i said we have some of the pieces and i think what we've been able to do is is is make some of the the the api and as3 supports us too so we don't worry you know when i talk api i'm using api and as3 interchangeably and those concepts the apis are there but we also have the the declarative model so you don't need to know all of the you know the the the how to cook it and understand how to use that with the the ronco knife or anything like that just give me a chicken just give me a chicken right yeah specific specific to as3 you know i i think that you know for for those of us network background people who are used to clicking all the nerd knobs individually and building out services uh something like as3 seems a little mystical and and uh and scary and because it's abstracted and and you know we like to know all the nuances of how the thing works and how it's configured and you know when it comes to managing search you talk back to that um solution that i built with uh let's encrypt uh using the acme 1.0 protocol which i probably need to update um but you know it it is more brittle right because it's got to go through all these different individual iterative steps with the eye control api and and even once you get the certs up there you have the um you know you bang your head against the wall if you're not using transactions because you can't update the cert without the key you can't update the key without the cert but they both have to be updated at the same time if those ssl profiles are already attached to virtual servers and so you know if you're if you're just getting started out and you're trying to figure out well it works in the gui why can't i do that with the automation tools you know there's there's a lot of ways that you bang your head against the wall and and using something like as3 where you're just putting all your data into a blob of json and you click the easy button and then it's there and everything's updated and you know the uh the as3 application solves all that for you and it's supported whereas if you're if you're doing your own thing um you know that's a little less supported and and i wanted to add i wanted to add as well chef that you know i know when you talk about we are not quite there and there's there's so many moving parts and pieces to this whole discussion of just tls encryption certificate management all that stuff there's just this just very it's a very complex you know thing to get your arms really wrapped around uh anyway when when you say we you're not you're not in you're not specifically necessarily talking about f5 it's more we like the security you know um industry in general right it's it's not like f5 is lagging behind in what we're doing it's it's the whole industry needs to kind of really move forward on this right yeah yeah the cap the capital we not the f5 we right right because i think f5 is as we have again like a lot of the other oems out there we have the infrastructure and we have and whether it's as3 that understands how we do our stuff um or apis which allow um you know the people who have code behind them and want to want to type stuff that you know they give us the the ability to do that those those building blocks and the legos are there um but we still have to solve the layer eight problem and i think that's where the industry is today like i i don't know if if if we have the have have all the answers and you know then we have what's the next thing we're gonna have to worry about in encryption and you know you know people are gonna bring up quantum and all those other things and you know those those are all crazy long-term things but i think we just need to solve this operational problem when when you know like i said this customer i was talking to a reasonably sized uh um um you know uh financial organization not not not a top 100 company but you know they they they have a couple hundred you know 150 200 vips uh virtual servers that they need to maintain each with their own certificate because they don't want to use wild card certs and please if you're using wild card search get away from wild cards but you know so so they're making the good choices but the good choices drive operational challenges that they need to think about too um and so when you think about that okay if i got 150 dips uh 550 fips do i have 150 staging and prod and uh pre-prod so does that turn into 450 um certificates i need to manage right there and then what's what's my back you know what what's what what uh certificates you'll have in the back uh do i just have a you know a longer lived uh you know while uh you know self-signed cert or assert that's signed by my organization for those um so i mean that's typically what i would suggest people do um that alleviates some of the challenges um they might have you know you might have a an acquisition which now suddenly grows your number certificates or maybe the number of domain names you you require so maybe do you go to a you know to a to a domain search or something like that where you know where you could have multiple domains in it so there's a bunch of things that you can think through um but um those are all things that technology doesn't solve um those are things that we we need to we need to solve um and i think the building blocks are here um and and i think we're making that available to people uh but i think what we need to do is figure out how to make that work uh inside the operations side yeah yeah fantastic yeah because that's i mean that's a it's a critical part of the whole equation is that what you just described so yeah i really appreciate that i'm not sure we really do a whole lot of road map uh live and so you know i'll throw your question up here maybe chef he knows uh but we might have to get back to you or your sales team get back to you on on on that but if you had any insight there so so we do have some uh fips roadmap stuff uh for sure uh so fips is uh it we're using this term here as a as a as a industry standard uh us government standard for uh for for uh definition of of how a certificate is maintained uh so these are fips 140-2 uh boxes this 1050 10 350s um so the um those boxes are um so we're not upgrading that box but we are making other about boxes available um and um if you what we want to reach out to us or we can we can we can get back to you uh with the exact uh comparable model that would be and say our our i series and that does bring up some technology uh questions too and you know ecc versus rsa certificates and and you know i mentioned that earlier it's nice nice to have ecc certificates because uh they work really well on mobile devices and they're they're battery friendly um um but then there's challenges because traditionally we all worked with rsa offloaded uh cards so uh previous our boxes you know the the the 10 350 in that generation of technology um we had uh you know cavium cards in there that did uh our ssl offload those were rsa ssl offline not ecc offloads so some of those boxes as you move to ecc ciphers um that you have up you have a performance uh change to think about too so um so some of the boxes um don't do uh ecc offload it it's gonna be a challenge for those those boxes as they scale but again if you're looking at say a modern virtual appliance um so you know your hypervisor probably anything that's within the last three or four years uh those those those chipsets those intel chip sets actually will will are are optimized for doing ecc nicely so there are things that you can do there too but the fips question is again that's an hsm so that's a a hardware security module that actually has this the key stored on it on the 10350 if they actually open the case the the keys are are are broken burned um actually the chips are physically burned in them so they're they're shut down um and um so so there's a that's a protection uh technology um uh for the certificates it's not how the certificates have generated that they are traditionally generated on the on on the on the box um but there's no way to get the certificate off that box so that that's a way to lock it down so i i i know i didn't necessarily completely address the question but hopefully i talked enough to make it sound like i said something i guess a question for rodrigo for me then with him already being gone is that you know for for keys stored in the uh the hsm if you use the uh the dash f5 ssl flag in your in your capture does that work or or not yeah uh you're getting the ephemeral key at that point right so that's the key that that that would break that that will allow that traffic to be decrypted so i i don't think that's the hardware key i think that's just that the the actual uh handshake key but i i could be wrong i'd have to look so i think you're still gonna get that yeah i'm thinking about like mission impossible or inspector gadget when you talked about cracking open the case and stuff starts blowing up you know burning on itself so that's awesome man and then also we got a question from daniel why haven't more major cas adopted the acme protocol so um so there are there are several that have adopted it so i just i actually had the wikipedia open uh and so i'll look smart but i'll i'll be honest um so there's let's encrypt bypass uh digital cert entrust global sign venify uh sectio which is uh semantic um prime key which i'm not familiar with um uh manassasol.com they all support acme um so if you want to go get a certificate from them you can use the akb2 protocol to talk to them um so that's a that's that's possibly uh helpful so maybe i answer your question there yeah all right we have some more questions from uh suggest uh hey with this one why don't you um send me details in an email you just hit j.rom f5.com or just send it to dev central at f5.com and i'll take that offline with you um because that's probably a deeper conversation that we have time for here if you want to upload um a quick view uh to ihealth and and send me your your quick health id i'll uh i'll see if i can help with that or you know get a case open with support as well uh because they'll definitely be able to dig in and help you figure that out but i can do some um preliminary uh analysis but i i'll just need more details for that and then uh we've got uh thank you just a big fan so shot joe i appreciate that hey all i would say to sushant is uh join the club man we're all big jason fans and uh yeah cheffy's got a lot of fans on here we got daniel you know we put this up other and uh and then we had another one earlier yeah yep so big fans big fan we got we got big names on today this is big it's a great it's a great session great connection okay well i you know we're we're at time so i don't want to keep everybody and uh and i think we're kind of wrapped up with our cert discussion if anybody has more certificate questions that they they want to run by us or they want to run by sheffy get them to us and and we'll address those offline there we go there it is again canadian celebrity and uh uh great session guys absolutely let's let's be neutral all right so yeah we won't have any canadian uh you know u.s wars today hopefully right right and uh yeah another time and i'm gonna post it in the in the chat again real quick if if you guys want to take a quick two question survey just to let us know uh best time and day of the week uh that would work for you we can kind of move this around and maybe we'll do a couple different times and we'll uh you know alternate weeks or whatever and when is the next session hey next next thursday same bat time same bat channel uh 2 30 central next thursday and we're going to talk about all things for certification so we got dr ken and heidi themselves are are coming to talk about uh the certification program and then we have a celebrity guest joining us who is fi certified and uh has shared his journey in deaf central articles so uh you know and he's an mvp so you know we'll have a great conversation on on our certification journey next thursday and so uh thank you everyone for joining us and we'll see you next thursday all right thanks chef thanks to everybody man we'll see y'all see y'all next time bye guys

Channel: F5 DevCentral

Views: 1,658

Keywords: f5, ssl, devcentral, authentication, certificates, tls, tls 1.3, big-ip, encryption

Id: r0wdtAk7AZA

Length: 61min 35sec (3695 seconds)

Published: Fri Apr 24 2020