Security and Microservices by Sam Newman

well welcome along sight there's a lot of people frowning today I'm not entirely sure Evan are so grumpy the cricket is going very well for England so I'm not entirely sure what else would you care about in the world the rain seems to agree with all of you though so we're welcome along I'm going talking to you today about application security and micro services to an extent what I've done here is I've put the word micro services out like bait on a hook to bring you in to talk actually about application security we are going to talk about both things don't worry but you will you'll learn things I hope coming out his talk my name is Sam Newman I wrote the book you can go and buy the book downstairs if you want you don't have to buy the book I don't mind too much it's fine please really please buy the book though got a job at the moment so it'd be great anyway I'm currently working with a start-up called atomist we do stuff in the micro services space if you're interested in what we do go ask me later look at the calm but we should start off at the beginning with another plea to buy my book no not another plea to buy my book to talk actually I want to get a sense of the audience here so my second time in Antwerp last time I came here for the conference last year and I was here for all of about 16 hours so I didn't really get to get to know any of you or get to the kind of crowd that comes to devoxx Antep so I wanted to ask right at the beginning do we have any any doctors in the house I mean I can't quite see I'm shielding the lights or any of you a doctor some sort of sending any doctors it's a couple of hands up right there's a couple of hands up that's good right I mean that's not the wrong that's the wrong kind of doctor isn't it this kind of doctor a medical doctor hands up for any medical doctors in the room that any of you out there are there any immunologists virologists in the room people who have done research and bacteria any of those sorts of people anybody who's played pandemic I'm lowering the bar now there we go yes I thought there'd be a few people have played pandemic well if anyone gets sick of course these are the people to talk to the reason I sort of start off by asking this question is because we have a very very small representation of medical practitioners in the room it might be vero but I'm hoping and at these judging on my experiences thus far in the bathroom that most of you know the importance of washing your hands after you've been to the toilet hands up if you don't know that how important is to wash your hands after you've been all I'm going to say is if anyone does put their hands up stay went away from those hands this is a thing we do not necessarily have to have multiple years of study and expertise to understand the importance of doing some basic hygiene to keep us well and to keep our family as well but when it comes to application security we're kind of in the place where we still assume somebody else is going to solve the problem for us we're sort of where we were with testing say 15 years ago when you'd go and talk to organizations and say you know wouldn't it be great if you maybe when you wrote some code actually made sure the code worked you know this is brain blowing stuff at the time and I go oh no we have other people for that they make sure the code because the code works in my head and then they make it sure it works in the real world that's that's how things are and we've realized that maybe doing something around making sure our code actually working actually investing some time and energy in something like automated tests might be a good idea more recently we're starting to come to believe that also perhaps just sprinkling in usability at the end of the process like sort of like sort of bitter which is you know just a sprinkle some guilty would have usability or applications maybe actually that doesn't work either and we should bring some design thinking into how we build our products but when it comes to security what's too hard we don't even know the basics we don't need to be experts but there's certainly a subset of application security that all developers I think should have in the same way that we don't expect every developer to be an expert in the field of automated testing but we do now expect you to write a few tests so hopefully today I'm going to share with you some hand washing type stuff this is a talk about security and normally I feel the need to scare you it seems that current events may have scared you anyway but I'll give it a good old try how many people here have seen this site before this is North Corp right so this is the top tip that's going to come out the conference you're going to want to go back to your employees your employer's rather and they're going to ask you how great was the conference and of course you need to explain how awesome it wasn't show that you've learnt loads of new stuff because you want to come back again next year right you need to justify the investment best way you can do that so yeah I've been learning about security pop up a spare monitor and just have this running in the background and they say what are you doing is sound monitoring our perimeter because what it is it's like an online sort of video of near real-time attacks that are going on in the internet so this is sort of just showing you the general background noise and radiation almost that we live in it's sort of interesting information you can see up here you can see till the attack targets you can see the attack types you'll see lots of old protocols and well that was when my machine went very slow it turned out somebody was very cross with somebody in South America at the moment and they all know actually they were cross with somebody in the Middle East that's so weird because it's such a calm place normally and Australia is asleep and nobody cares about us anyway I'm from us and I live in Australia so this is sort of the backdrop of the world we live in now a lot of these tacks are targeted a lot of them aren't you'll often just this is background noise you know there are BOTS there just looking for people running old versions of WordPress it's not a malicious intent directly against you these aren't necessarily attackers that are targeting your site but there are just things that we'll try and do stuff to your system just for lulz and of course there are people that might want to do more serious things to you if you are running some kind of public facing technology company and you are non-trivial in size the question is not you know if you're going to face some kind of attack and maybe suffer some sort of breach the question is more when and the reality is a lot of our time and attention is sort of focused on stopping bad things from happening I'm also going to talk to you today about the things you need to do to make sure that if think bad does happen you can deal with it this is just the backdrop this is just every day now so we have to build our systems in a more resilient way and actually build them with defense-in-depth this is a talk about micro services so I have to talk about Microsoft's a lot and I will do it's kind of my jam my definition of micro services is it's sort of it varies I but I tend to draw them as hexagons hexagons are a nice shape they tessellate as a pleasant feel to them I dislike layers that's why I don't do I don't do rectangles because rectangles people to layer them up I talk in my book about onion architecture you know architecture with loads of layers and when you cut through them it makes you cry these are tessellate and they are they are sort of oriented around business concepts because we found that those boundaries are more stable the rough definition I use now use variations on it every now and then small autonomous services that work together modeled around the business domain when I mean autonomous what I'm primarily talking about is independent deployability and if there's only one thing you learn about micro services this year it's that that really you need to optimize for independent deployability of services that's going to get you most of the benefits that people strive for when their dots micro service architectures now obviously this sort of decomposed albeit connected relationship gives us some interesting questions around security now straight away I mean it-- when I first started looking these sorts of architectures I was looking I was seeing some patate potential positives in the realm of application security you know with a monolithic system I've typically got a small number of processes I've got maybe one type of data store if something happens to one of those running processes someone hacks into something they've got access to everything right literally all my eggs are in one basket or a very small number of basket with exactly the same eggs in it if I'm load balancing and that's sort of an interesting problem because you you're your ability to do defense in depth to actually provide multiple types of protection somewhat limited because of America so a system is someone hacks one part of your system the impact is therefore somewhat limited you can also do some interesting things you know you can probably how many people here are using something like Amazon or you know some of those sorts of systems or docker those platforms for example provides you what's called software-defined networking layers so the ability to write code configuration to define things like custom subnets you can do things like putting different types of services on different subnets setting up peering relationships between them giving you effectively multiple Network parameters or which you can provide protections that is a great boon great benefit the by opening up multiple scenes we allow ourselves to defend multiple points all the way down the stack that's great so what you're telling me Sam is that our micro services systems will be more secure than our monolithic previously past systems well not so much because micro services giveth and they take it away because also what happens will cause when we move from a monolithic process is all of those things that used to be in process calls on our network calls so now what used to be sort of fairly protected traffic is stuff that might be flowing over your corporate networks hands up how many people here you do anything basic like just run HTTP on internal networks how many people here on HTTP yes that's I'm pretty sure half of you are lying but that's still pretty low right you know we don't tend to think too much about what happens inside our perimeter and when we move to Microsoft X is like this we are opening ourselves up to more places where traffic can be intercepted payloads can be manipulated endpoints can be addressed and we also have more machines more processes more processes that can be looked at attacked so we've also potentially increased the surface area of attack as well so we have the opportunity to make our systems more secure and to defend them from malicious external parties and internal parties even but actually I think out of the box our systems can sometimes be less secure unless we apply a bit more thought I used to work at for work so many years and internally we had a model that we use to talk about application security think of doing we refocus on prevention let's stop the bad thing from happening and I think if we focus entirely on prevention although it's probably the deepest subject from a technical point of view you're going to miss a trick so I used to be a consultant I therefore have a quadrant as we start off with prevention so we need to think about how we stop bad things from happening we then talk about detection if somebody has attacked a system hacked our system gained access will we even know once we know about it do we know how do we need do we know how we're going to respond to that attack the nice thing about application security is there's so many great examples of organizations that respond incredibly badly to these sorts of attacks so we've got lots of fun stories to share there and once you've responded to say a breach to an attack on your system how would you then recover from that case and we're going to take a look at each of these in the context of micro services but we'll start with prevention it was probably about six years ago five years ago now I think when I was starting I was working with a company who were starting to get a bit more serious about application security and so they created an in-house InfoSec team and so that InfoSec team was starting to look across the entire organization everything from sort of laptop security data center security down to premise security and alike and we're looking for vulnerabilities and we're starting to put out a few pilots to make improvements and one of the first things they sort of raised was this idea of actually putting CCTV cameras on the front door of our offices and what they said was this actually acts as a two-fold thing on the one hand having a CCTV camera up prevents people coming in because the deterrent you know I premiere security from a technology point of view they actually could be quite serious if you can just walk into an office and jack into your corporate network have a have a camera up and you might deter people from coming in but you'll also have a sense of what happened you think st. does happen you can D see what did someone come into offices that we didn't normally expect to be there could they be the person you could then pass the information on to your authority is there like now this ignited a firestorm of internal hand-wringing and debate and angst and anguish ah this is you invading my privacy when somebody think of the children you don't trust me my rights are being violated developers too much time on their hands not dealing with serious issues history history history oh and this went on for a while and then bring a few weeks and it was it wasn't always a great conversation and then somebody said you know maybe maybe before we put the CCTV camera on the front of our office and then our reception maybe we should make sure the door works because the lock for the front door of the office I was based in hadn't properly worked in a good couple of years so frequently we just you know you'd think it's locked it would blow open wouldn't always closed properly you'd walk in in the morning and the doors wide open and the alarm hasn't gone off that's a bit odd so maybe spending you know a hundred bucks on getting a locksmith in to fix the front door might be a better way to spend our time and energy then you know pulling up a several thousand dollar CCTV camera and this is the issue when it comes to prevention in the field of application security we often get fixated on the wrong things and we're not always able to step back and look rationing it where should we spend our time where should we spend our energy and we fixate on the things we see and we don't think about other things and inevitably of course it's something you didn't think about I don't always think am i fixing a camera am i fixing the door so when it comes to prevention one of the first things you're going to want to do if you want to think sort of rationally holistically about sort of application security is is actually to step back and do some form of threat modeling now there's a whole bunch of really good threat modeling processes out there something were quite lightweight you can do them yourselves I'm going to go and talk a bit about a very very small part of that I would thoroughly recommend going and looking at Microsoft's threat modeling process stride and dread and all those sorts of things but just to give you a taste of what threat modeling can do for you I'm going to share some examples from Bruce Schneier he's a technique he called tap trees and it's this approach comes up time and again in different threat modeling processes it was on dr. Dobbs the dobbs is now dead so it's now on his site but anyway it's there and so the way he you know he breaks the problem down is it's like a well that what you've got is you've got a malicious party um and you've got vulnerable stuff and you've got things that you don't want them to achieve and so you pop the thing you don't want them to achieve at the top so in this particular situation the thing we really don't want that malicious person to achieve is to open our safe and so then what you do you start in numerating the different ways in which that could happen and some of the threat modeling processes go into more detail to help you think freshening about that but in this particular example so I want to open the safe well how can I do that I could pick the lock I could learn the combination or I could cut the safe open and you keep going okay well how could I learn the combination well I could find the combination written down or I could get the combination from the target and it's now if I becomes a bit Jason Bourne and it's all very exciting because this is like this is the world InfoSec they're all quite paranoid people often quite rightly so because they sit down and spend all their time in numerating potential risks and then start thinking about how could they get the combination from our people or they could blackmail them they could threaten them they could bribe them and these are real potential risks I mean you know let's think rationally about it you know finding a combination being written down I mean who here has been in an organization that's had a part of a policy so complex that you saw co-workers write the password down on a post-it note right few of you who have seen that awesome webcam where they the organisation said we got to use two factor auth so we'll had their little RSA key and they didn't want to buy RSA key to everybody it was a hassle for everyone to forget they kept forgetting them living at home so I just put a public webcam on it on the public Internet so everybody could share the same RSA token right it happens all the time I worked in a bank in a big bank actually and your log as a consultant and your login name was actually a nine digit number now aside from how warm and fuzzy it makes you feel when you're identified by a nine digit number it was whoever members nine digit numbers I've got space in my head for about four and that one wasn't one of them so now I've got to somehow know my you know my my cut might sort of login name and in the password policy involved you know slaughtering chickens and things so lots of people wrote both of them down on the same piece of paper so you know this happens all the time and so then once you've enumerated your risks you start looking at how hard it might be for an external party to use this sort of attack vector to gain access in this example to the safe and you start applying weights I mean there's one example which is just difficulty level of difficulty it might be cost to the attacker in general we're looking at prevention what you're actually trying to do is not necessarily make something impossible what you're trying to do is raise the cost of something to such a degree that there's no incentive or benefit to the attacker using that Avenue of attack for example if it costs our attacker a million dollars to get a drill capable of cutting into our safe but our safe is worth like five grand and we've only got 20 grand worth of bullion in it yes they could do it yes they could attack it it's not actually financially viable for the attacker we do these things on timing you know the when we start using long keep you know long and long bit encryption what we're saying is not that it's impossible for someone to crack a hash just that it will take them either until the heat death of the universe or so much hardware that they can't possibly afford it and so when you do this you start looking at your weak spots you see what actually it's quite easy to find a combination written down because I saw it this morning and it's quite easy to bribe somebody because we pay people really badly so maybe we should do things and put our energy in there to actually close off those attack vectors so I could really recommend some process like this I the Microsoft threat modeling stuff is really well documented and it's not that hard to process to actually do a first pass and there are external people that will come in and spend a like a week with you and also do a fairly good job of this out of this you know you'll start coming up with things like a prioritized list of where you should spend your time where you spend your energy but we should talk now really about some specific examples of prevention that we could carry out in America so it's environment so as many of you may know from previous talks in the book I often use a fake domain of a company who is selling CDs online and there's also streaming and selling physical CDs and this is the site so we've got the box which is our perimeter we've got sort of external UI clients in the form of a mobile application or websites we've got a royalty payment gateway we send them sort of information about what artists have been streams we're sending them sort of payment effectively information that is going to be used for us to pay royalties that's quite sensitive information and inside our perimeter we've got some services like a catalogue to show you how much things cost the music web shop is the main website back-end and some user data right so a very simple simple simple architecture and we start looking at problems well we should really start talking about the first one I raised earlier which is those in process calls and now inter-process course they're now exposed over our internal networks so what can we do in the realm of looking at things like Transport Security where should we start well as I mentioned earlier well let's talk about HTTPS right it should be very easy nowadays but HTTPS itself is not a solution to all problems so what does HTTPS for example give us by the way slight sidebar some people have claimed that micro services are HTTP based services that's not the case right HTTP is just a very common technology used to allow server to talk to each other I don't think you have to use it I think it's a good default choice it's fairly widespread choice but you can have services communicate in all sorts of ways like message buses and RPC and yes even exchanging files but let's talk about HTTP HTTPS is all the raid on the public facing web as it should be it should be a choice every single one of you make for your own websites what about internally inside the network what sort of benefits does it give us well the first thing it gives us is is server guarantees I mean I'm I'm assuming you're using sensible certificates and not just making these things up but look it's giving us guarantees that the thing we think we're talking to is actually what we're talking to why is that important well so gets access to your network they pop something up saying hello I'm the customer service just send me information I promise on the real thing now with this we've got some level of assurances the thing I'm talking to it actually the thing I'm talking to we also get guaranty to know the payload itself hasn't been manipulated that could be quite important but we get no client guarantees so a server receiving a request doesn't necessarily know who's made that request a bit again malicious party on your network starts just making API calls to extract all of your customer information that's kind of not necessarily great and so if there's nothing at the transport level confirming these client who they say they are that's that's problematic and of course the biggest downside historically to HTTP has been this certificates and setting those things up is painful right it's just awkward and ask so much work guys found a person and set an email and it's it's being difficult of course we now have let's encrypt our Chidori screen catalyst is now open hands up who here is use let's encrypt on those about let's encrypt okay well that's great the rest of you should know you can go look up Alice's now let's encrypt is very interesting it came around last year was MIT I first found out about it in glass opposed March April 2015 and it was it is a open free automated certificate signing authority so you can go to them and get certificates HTTPS certificates people fixated when it first came out on the word free oh it's free I don't have to pay money anymore as though the money was the blocker how much the root certificate from flank cost you 40 bucks a year you really telling me the blocker for you using HTTPS is $40 a year in which case how much you care about your customers that's not really the issue right the issue is this other word it's automated because actually the process of getting new certificates and renewing them was typically a multi-stage manual process when we're moving into the realm of miraculous services we're often talking about dynamic provisioning as versus dynamic provisioning of the machines they run on people are often using sort of cloud-based platforms they're spinning up machines left right and center and then to suddenly introduce a manual step in getting and installing and managing certificates or changing them became quite problematic now of course Amazon has solutions in this space as well which still very good this one has the benefit of being sort of Universal the backing let's encrypt is actually an open protocol called Acme and it gives you like a nice command line like if I want to grab a certificate and automatically configure my nginx server I just typed let's encrypt run and it and it kind of just does that work for me it is used to public/private key pairs to actually get the third this is really really cool stuff there are some downsides and support wildcard certificates but actually because it's so easy to issue these things at the first place you may as you don't need wildcard certificate so you have to be more specific so do look at that so okay maybe we say okay well HTTP isn't fixing one of our problems but maybe it's a good default choice now I'm actually a good believer when it comes to micro services to having some default choices so actually just saying look by default this is what our service you should do this is how they should behave because there's some areas where you do want consistency and it comes to security you may not want the link so maybe we just say look within our perimeter we're just going to adopt HTTPS Everywhere no it's not going to stop all problems yes it's fairly easy to do are the performance overhead of these things is pretty minimal nowadays given modern hardware you know we're no longer talking about 25% overhead on calls and things so maybe you just really not primitively pick that but we could maybe do better what about something like client-side certificates I really want to clash your thunder and like a bolt of lightning at this point is when we scares people give them the willies we're client scientific 'its we introduced the ability to have client-side guarantees that's really nice so now I know the person talking to me you know there's some some trust I can say that requesters actually come from them yes that's that's well that's good I'm happy with that but client-side certificates are even bigger pain in the bottom to use than to the server side HTTP times start certs you know we talking here about things like the phrase PKI public encryption infrastructure be used and the automation in this space is pretty poor historically there is stuff again Amazon have stuff in space says our lemur which is tall from sort of a pluggable PKI framework from Netflix I don't actually we should allow you to automate the process of issuing and renewing client-side certificates they've built it on top of Amazon's own to difficut stuff but it has it's theoretically pluggable I don't yet know of anybody else using lemur outside of Netflix if you are in this room please come talk to me so I'd love to spread the word about this stuff but even if the client you don't you're not able to automate these you may still decide it's worthwhile like maybe I start thinking well you know I'm looking outside here's perimeter this is actually going over the public Internet I'm talking to some third party we're worried enough about it that we are going to use some kind of client-side certificates and maybe what we do is just you know we accept as a manual process and quite often you'll do things like having some sort of aged service at terminates a client-side management at the certificates so actually you're sort of isolating the points at which you have to do that handshaking so that might be a decision you could make that's a different that's not opt out you know the d4 is hosted based everywhere maybe you start doing different things with our edge services we should also talk about auth and here I'm talking about both the AWS both authentication and authorization starting with the sort of the authentication part of this which is our users who you say you are when we start thinking at the top of our stack we start thinking of the traditional things we're all quite familiar with we're thinking of OAuth or thinking of form of so logging into my browser and dropping some kind of a wharf token that it's used to confirm that yes this person should have access to the system and then authorization is typically an internal concern where I say okay yes Sam is allowed and here's what he can see and now some things are an authorization you can do with things like directory services increasingly your application itself your services themselves are going to have to sort of make decisions for themselves about what they allow you to do what they don't allow you to do you can't necessarily offload all that to third parties nor should you in fact you have to do that internally the issue of course is with the market service-oriented system is that often when we carry out an action we call a service which in turn might call another collaborating service in order to carry out that operation so in this particular situation maybe I'm pulling up my customer records so I go to my history page and to render the history page the music web shop actually I go off down to the user service to fetch my records back now this this sort of interaction opens up opens us up to a class a security problem that pre-exists micro-services caused the confused deputy problem and the confused deputy problem is a class a problem where you can trick an intermediate party into asking for things they shouldn't be able to get so I could be logged in as Sam but I could formulate a request and its previous example to the music webshop saying can I have Alice's data now I am logged in I am authenticated as fan unless the music webshop is you know real rechecks that I'm also authorized to get Alice's data or there's a bug in how I handle our authorized code I may then allow that call to go downstream and what we often find is in these situations where we have these calls where maybe you've even got guarantees that the server's the server and the client is the client and so yes these really are the real things talking to each other but we then sort of embrace this idea of implicit trust so one's like a hive mentality in a way we sort of work under the assumption that the user service says I was at the music web shop asking me for something it is okay great but whatever it wants you can have but we don't revalidate whether or not the initiating call from me is authorized to access that information now there are ways of solving this problem some of them historically have been quite nasty there any sam'l fans in the room didn't there's a guy there's a few Wow you know we don't admit it now that theoretically you can solve this problem using nested sam'l assertions and I knew this was a theoretical solution and it seemed like it could work but it looks too horrible to even contemplate and so I assumed no one ever did it I met a team in Norway last year who did it they spent six months making it work and they backed it all out it was horrendous to make work the way it basically works if you have multiple assertions you sort of have assertions often for the services and then the caller inside and you end up these huge chains of things happening especially you have multiple stacks sort of a much simpler way of handling this is using things like tokens you can issue a token when you sort of when I'm logged in when I'm authenticated I'd say maybe if I could JWT token and that token can be passed downstream that token can be validated by services downstream and even interrogated for information about me so it can get information out about my roles for example and decide whether or not to give me access to information I've found extremely few companies that have tried fixing this by the way I'm hoping that will change over time okay so this that was a brief run over looking at Transport Security there's other things look at stuff like H mark different types of API keys which can help you but let's think a bit about data at rest this is the issue of course that someone hacked into my system they get access to my data all of my data is sitting there and they might steal you and run away that's not great typically we'd start talking about encrypting databases that's often painful with a monolithic system because the overhead can often be an impact often databases of a bottleneck in our system performance so we start doing things like selectively encrypting individual tables that gets complicated because sometimes you miss a table and you've leaked information you have seen a couple of places where you're leaking information in the relationships between things themselves aren't necessary encrypted so it becomes a bit of a pain and of course with micro-services what we're doing is we're also blade breaking up our data stores so we can actually selectively target our energy and efforts and just you actually be simplistic in how we handle this stuff we can just say look when we look at our system it's our haxing and gets access to the database for our catalog service it's like that's all public information anyway do I care well again the worst they could do is maybe go in and maybe change prices but there were different ways of handling that other than encrypting the data and so maybe I just say actually the data at rest I care about is the user so so user service now is saying have to now run unencrypted type of data store of course then we have to have separate conversations about where keys are kept but for that sort of stuff you can go do some more research back with your databases support and look at stuff like vault which is awesome a brief aside this is I hope a fairly buzzword compliant talk so far because I said the word micro-services every five minutes at least I need to make it I need to amp up the buzzword enos so I should say Dakka Dakka Dakka Dakka Dakka Dakka Dakka Dakka Dakka go-go-go dhoka dhoka in the context of security and mariko services the reason I talk about this is because often they go hand-in-hand no you do not have to use docker to do micro services no you do not have to do micro services if you use docker but the tools are highly the tool and the technique are quite complimentary some things to be aware of a lot of excitement around docker initially was from developers who could suddenly sort of go and just run look I'm running Redis an elastic search on my machine and what scale on my laptop and they run single commands and what they're doing is they're pulling down containers to putting down images from the docker hub and one here used the docker hub to pull down images and stuff yeah okay so things to be aware of the docker hub there are things there is a certain level of trust around the docker hub there are official images official contributors what that means is that docker have taken steps to ensure that the person who uploaded that image is really who they say they are they don't really necessarily do much beyond that in terms of verifying whether or not that image is safe and secure they do some stuff so this is about 18 months ago Banyan ops did some research that they found that 30% of all the official images the official images in docker hub had critical vulnerabilities in it now this wasn't people uploading malicious stuff necessarily the issue is when you create a document is it's immutable so you upload it in its base and say summer bun to image and time passes people find bugs and they find critical vulnerabilities those things are patch but your image is not patched goo a docker have now exposed the security vulnerability to do some vulnerability scanning now on the puppet docker hub so you can go and look at any image you depend on and see the vulnerabilities interesting thing to note of all of the abun - images from the official Ubuntu user the official bundle images how many of those do you think have critical vulnerabilities in them of the ones that have been scanned like 5 percent 10 percent how about a hundred percent every single one of the a bunch of official images and docker hub has vulnerabilities on them actually the viability is they have our bash vulnerabilities that were patched in 2014 even in the 1604 kernel at 6 no 4 of disk drive which is kind of interesting right so when you grab these things they are not necessarily in of themselves safe or risk free and you have to understand the transitive chain of trust effectively you have and you download these things I will not install stuff in the public docker hub in production because for me personally that feels like I've found a thumb drive in the gutter I've plugged it into my production systems and go yeah she'll be right now there are things that we can do there are increasingly are great tools to help us scan and look for vulnerabilities the way doc images a built up based on layers allows things like Claire which come so CLA ir which comes from core OS to actually do those scans inside your bill pipelines right so you know you can actually build in scanned on the images you yourself create normally the docker hub I'll actually copy what some I'd look at what they've done I maybe copy there and I'll pull it into my own private repo some organizations that allow you to hose private doc repos will also do that scanning for you so increasingly you're seeing that stuff being automated just please do be aware of it now there is actually an argument for running a lot of things in docker because docker by itself can actually provide protection but I don't think it's very don't think you could consider docker a sensible way of isolated untrusted code and that's effectively what they're doing when you being a public image down from docker Harbor running it in production so just be aware of that you should also also mention the Microsoft stuff so the links will be in there I'll send around links to all these slides afterwards say you don't have to take photos now one last thing please patch all of your stuff this is the making sure the door lock works or prevention a large amount of data breaches are caused by Pat yet bugs that have been around for months if not years that people do not patch you need to be patching your production infrastructure at least once a week you're doing things like rolling custom a.m. eyes and docker images fantastic there's some reasons why that's great from security point of view but if you haven't actually updated those or rolled out new versions on a weekly basis you've potentially got unpatch machines running their their systems out there like up guard that you can run on your on your infrastructure to look on passion machines so that can be worthwhile patch your stuff like weekly and I'm really serious about that otherwise you're gonna get bitten bit hard let's move on to detection a lot of this is about stopping bad things from happening but if it does we need to know often the very first places do this just even know when people alert you that oh by the way we found a huge vulnerability in nodejs version 2.0 I had not versions up to I don't do note I'm afraid so you know subscribing to services like qualies and things like that can be useful to see those CVS as they roll in and there's actually increasingly automated tools out there that will actually look at the critical vulnerability to your technology stack and will scan your software and say oh by the way there are three new versions of libraries that you use that are out there that fix critical vulnerabilities those things are things that were increasingly building to our CI pipeline now those things they are often helpful in terms of determining if we've been affected so for example when I was that thought works I help when I'm internal systems shellshock came up so shell-shocked was this really really serious exploit path that was discovered within bash and the bug had been around for at least sort of a couple of years and was sort of made a large number of linux servers vulnerable to external exploits now the interesting thing when I was talking to the Intel InfoSec team was they said well look we've got everything patched we got it all updated and we worked really hard to get that done but what we don't know is did someone actually use this exploit against us in that time and so I started we started talking about how we'd known is that the very first thing he would want to know is where are our locks where are our application logs are we are not you know our access logs for our machines this is one of the best things you can do this from the point of view of micro services in general aggregating your logs is a great way of helping you understand what the hell's going on in your system my security point of view this is really useful using something internally like the elk stack getting a hosted service like see my logic get your logs off of your machines and get them somewhere else a security professional can use those to sometimes look at see have we been affected by by some sort of attack even the absence of information can be useful some of these root kits will actually remove traces themselves from logs if an attacker does gain access and wipes out the logs on the machine and you've already shipped them off then you may be in a better place if you are doing this talk to your security professionals because often they'll want a longer retention cycle than speed development purposes will want so I think from a security point of view we'll keep it like five years worth of our logs what was the process we wanted to go on there was also things out there like mod security you know application firewalls these things can often stop things that obviously look malicious they can also do things though like that alert you to things that might be suspicious and you can of course with microservices because we have the ability to have multiple perimeters you could apply multiple different services on those different perimeters and that can often be useful because sometimes these things are expensive and so maybe you actually apply these protections just around the most critical services when it comes to knowing if something bad happened um we do have to address one of the things that we talked about a lot in the context of micro services and that is the ability to embrace a polyglot development style so having multiple different programming languages this is often cited as one of the big benefits of micro services but if you're trying to keep up to date make sure your stuff is patched make sure you're running the latest versions make sure you using the right coding practices for your text stack to ensure you're creating secure services having a really polyglot developed environment can be a challenge especially at the beginning if you've got a high degree of automation maybe that's okay but this is really difficult this is partly why yes you get the option to you have a polyglot environment when you're in first moving towards micro services this is one of the reasons why I advocate actually keeping things really simple and maybe sticking to one technology choice at least for your main development stack until you sort out sort of your automation and tracking around these things the flip side of the polyglot world is that yes there are potentially more things to track more things to keep up to speed with but there's also more things to break an exploit that gains access to your Ruby app is not necessarily going to work against your JVM based service and vice versa and so when you do have say you know a tacher trying to gain entry into your system they're going to have to use potentially more tools at their disposal to gain access to what they need let's move on to response who here has heard of target and when he retargeting you have so this is kind of interesting this is actually target is an example of a an organization that did a fairly good job about detection and did a fairly awful job about response so target is one of the largest retailers in the US and unlike most of the large retailers or at least at the time they actually have an internal information security team and a pretty good one well ich so what happened was a malicious party installed malware onto the tills in target stores that more where just was like memory resident you sat there and as people came in and use their credit cards it just siphoned the credit cards and then at some later date all of those credit cards were bunched up and exfiltrated out of target I learned the word exfiltrated from CSI cyber which is where I get all of my cybercrime information from and then they were sold on the dark web it's always information went out there and that's not good the I saw some estimates that around 40% of all you credit card holders were impacted by these credit by this particular hat target is used by a lot of people now it's sort of there's a bunch of problems here the first was when they found out about this they were very very slow to let people know very slow to let people know and that's not a good look I think it was actually the someone the Senate the Congress that actually forced them to become pub to go public with this it's interesting to note that outside of certain regulatory type boundaries there's a lot of very few peer places in the world where there's a legal requirement for people to notify of security breaches for a few countries actually have legislation on their books where they have to tell you about this stuff I keep her in the US you have to but you know that doesn't cover target now this was bad right it's a huge impact but it got worse when people started digging into what actually happened so turned out that as I said target did actually have an internal InfoSec team that InfoSec team saw them all we're being installed and they said more were more more we're bueller bueller more we're and then they saw all the malware taking his credit cards admin no credit card good bye bye hello and they were ignored they had the team they had the tools they had the technology they didn't have the integration and internal communication pathways with which that information was taken seriously enough anyone to do anything about it so interesting right from a response point of view you've got a team how are they structured how they're wired into your organization how they who they report to how do you get information flowing for your organization just as I was leaving thought works has sort of been talking for a while actually about taking the InfoSec team inside that you know historically reported to this CIO and actually moving them to reporting directly to the CEO as a way of sort of improving the flow of information this is very well for the company that picked this stuff up which was fire I this is they then actually did themselves no favor with the application security community with a whole bunch so they did it later on I won't Leslie bad-mouthing because they're run by a lot of people they're ex-cia and they probably know where I live if you want to know more about the I get breach or just information in general about sort of cybercrime and the things that happen I can thoroughly recommend Brian Krebs and so he covered this in detail he goes out on the dark web to find out how much credit cards cost so you don't have to interestingly enough some of you may remember the DNS attack recently took out paper like Spotify and those sorts of things that botnet their first target which I think was a dry run was actually his website so people don't listen like what he does but he writes very well very approachable about this stuff there is a comms angle to all of this response right it's it's sort of how do you get out when something awful does happen there's a great deal agree value and actually just coming out I mean part of the reason the target situation was made worse is they didn't get out early enough I think get in front of the story comes is really really important in this world and I'm gonna show a couple of stories of you in a moment but the one the first thing I would say to you is you probably want to have a bit of a chat internally about if the worst happens who's going to send the email to your customers who's that going to come from is that going to come from your chief security officer your CEO your CTO what's that email going to say maybe come up for a few disaster scenarios which draft different emails because let me tell you this none of you are going to be thinking clearly when when the hits the fan and you're running around like headless chickens no one in that mode goes let's draft a really well considered email it's not going to panic everybody we can definitely find a little space to do it now it doesn't take long just draft it up because a good well-written email that you get out there early really can help to give you an example of a company who did is very badly let's talk about actually medicine in the actually medicine I'm not going to ask you to do that now this is an interesting story when it came out because of the slightly salacious nature of the site and but the interesting thing was sort of how they held that sort of the story grew around this and it was partly about you know they actually make sure this site is for but it was also partly about the way they handled things so what happened was a group gained access to a large quantity of the information behind Ashley Madison and they said unless you shut this site down we can share this information on the web and actually medicine say you don't have our information you don't have it they go yes we do I think I know you don't and they go yes we do and then these independent security researchers verified this they did they said okay maybe you've got a very small amount they said no we've got all of it they said I okay you don't have all of it we've got all of it all right you've got all of it that's all you've got and they go well actually you've also got the source code now you don't we have okay you have we've also put you emails now you don't we have all your emails as well this is bad right but this sort of drip feed of denial and from Ashley Madison themselves stretched out this story way more than it needed to be and actually if they'd just been sensible and hired a decent security researcher they could have verified probably in 24 hours that the information that had was what they claimed to have I know at least one person actually did that at the time so this was bad right and they stretched this story out Oh sort of a Troy hunk put me onto another case as a site called Adult Friend Finder which is where you find an Adult Friend for adult activities now that site was actually a little bit more the data it store was slightly more sensitive because you would actually sort of declare what act adult activities you would like to engage in with a certain special somebody or somebodies depending on the nature of your desires and all of that information was breached and was made available and currently it was some of the senior execs a came out that organization said don't worry they haven't got your credit cards like what happens if someone steals your credit card you phone up the bank you get your money back it's normally resolved within a week and not only ten days five to ten days you've got a new credit card and see like a problem to me it's painful it's awkward it's irritating if someone gets to learn about all of your sexual predicates credited them predictions by information isn't going back in the bottle the lack of empathy displayed by that person making that statement is staggering it's often not just the speed and the response it's actually about empathy and caring you really need to have that otherwise you're going to make missteps a good counter example are would maybe be the tylenol poisoning scare in the u.s. so many years ago I think it was 70s or 80s where it was several people were killed by tylenol which was at the time the biggest sort of off the shelf pain medication available are in America and so it was in Chicago only they never never actually worked out who did it or how they they achieved it and I think the seven people died very very quickly Johnson Johnson the parent company pulled tylenol from the shelves all across the US not just in Chicago but all across the US they developed sort of anti tampering mechanisms that are now industry standard to improve their say improve their supply chain security and then they went back out the top the recall at the time cost them a hundred million dollars and with inflation now be significantly more now the interesting thing was the public saw their response the moment he went back on the shelves went back to being the number one pain medication and their PR boost off the back of this was was huge they didn't do it for the PR boost they did it because it was the right thing to do and the public reacted I think the whole public your customers understand that sometimes something bad is going to happen they're going to look to you to handle that problem that situation well it's always interesting when you work with you know you have a company you love something goes wrong they just screw it all up and your view of them changes you actually might think better of them when they come out and are very clear about it there's a similar situation recently in the u.s. it's only Australia where I live so Telstra is the biggest telco it's sort of their incumbent sort of previously state-owned monopoly there are other ways you can respond badly in a ways that actually affect your company and your employees directly so there's a huge outage took out mobile telephony if you fixed their network telephony for a large proportion of Australia's Reds quite a distributed country there are an awful lot of people live in the country that rely on mobile phones and IKEA nativity this is a big now they did the right thing they came out very quickly with a yeah we're really sorry we're looking into type email the problem was what was actually said the statement came from the CIO so good you're coming from a fairly senior person in the organization that shows that you care and it's a senior person taking that was doubts good but what they said in the statement was a bit concerning the employee responsible didn't follow procedures and clearly that's not a good thing but I wouldn't want to preempt the proper investigation and we'll figure out what the right response is when we've had a chance to dig into the detail it's very interesting because they don't know what happened but they definitely know that someone's to blame and they say so right at the beginning the employee singular didn't follow procedures this sends a whole load of interesting messages firstly you haven't done your actual post-mortem you've already decided to blame somebody secondly how many people in tells you I can work out who that individual was very quickly what sort of message does this send to you if you work for Telstra and this thing happens ah so if an ink goes wrong that's another thing what kind of telco has a system where one person can make a single mistake and it takes it offline for a country or 24 million people right that's fascinating the CIO is no longer in their job there were multiple continuing outages afters because guess what turns out it wasn't just that one person's fault there were actually issues with the system there's a great keynote by Sydney Dekker used to be an airline pilot and then he worked looking at air travel air traffic disasters he specializes in looking at failure and failure in systems he argues I think very convincingly that as human beings were almost hardwired to look for a person to blame when something goes wrong and that's really he says it actually we create systems in which people a person making a mistake can cause a failure we can't look at an individual as being a source of failure we need to look at the system that person operates in John all sport puts in a bit of a different way he says the root cause of a failure is like finding a root cause of success we're quite good when something good happens we talk about the efforts of everybody or how everybody contributed how it was fantastic when something goes wrong who the last guy left the company that always the best person to blame on because I've gone they can't defend themselves there was a Lisa a silver lining tell she didn't have a free day today the first of many happened to be on a Sunday said businesses can take advantage of it but one lucky soul managed to download Norma's a terabyte of data over as little mobile 3G dongle so he was quite happy let's finish up now I talking about Wis recovery there's not too much to say here to be honest with you let's start with backups you've all got them right they're all tested right backups are important because it's not just people nicking your data is people doing things like encrypting your day care data rather how many people I mean it's probably a couple of you here that have either been the victims of or have family members that have been victims of things like laptop crypto lockers well they encrypt all of your information say if you don't give me one Bitcoin or 0.1 Bitcoin I will delete your laptop of course your backups you're in care what that taught me until your backup don't pay a Bitcoin don't pass go don't pay 200 pounds oh good tell me if you actually automate your backups how many of you test those backups right now with a monolithic system you've probably got one maybe do it maybe you test it might not be automated you may be you know it works you start moving to a micro service-oriented system those are more backups you've got to store separate data sets automation is the key another thing you may you can recover burn the thing down you get root kited and someone's in there and they've patched PS and they've patched rpm so those funky things you try not date it you can't live with that machine anymore you have to wipe that machine out now if you're practicing or you know using things like automation and configuration management systems like puppet chef ansible we are version controlling how you build your machines you can build them automatically maybe you're using docker images where the way the recipe effective of creating those those docker files are checked in destroying your machines and rating it is easy just burn it all down again your stuff all becomes much harder with micro-services if these processes rebuilding a machine reconstituting backups if those processes are manual this is going to be difficult for you also save if you done post mortems review them from time to time maybe if a quarter as a team take a look at what you discussed last time look at the action plans did we actually do those things did that work so just to summarize when we're thinking about application security we need to think a bit more broadly we can't just think about prevention prevention yes is important but you're not always going to get that right so we need to be thinking about how we know if we were if we failed in preventing something did it occur how do we know know how to respond have those conversations early with your team talk about the comms plan talk about the emails you're going to write who's that going to come from who's going to be the contact point do that while things are calm not while people are running around like a bunch of Americans I mean headless I mean anyway a recovery with recovery automation is key I don't think you need to have everything fully automated around build and deployment necessarily to start using micro services maybe he only got two or three of them that's probably okay you've got 5 10 15 20 you need automation you may need some standardization to make that work I meant about the book you don't have to buy in all seriousness but it's out there and I've got a I've also got my blog where I'd sort of write about architecture patterns around all this stuff and it's also a blog post I do write interview interesting techy people and then without that I think I've got some time for questions not a lot of time for questions maybe I could time for one question one question there yeah so you said so instead of using the public docker hub you sort of do unattended upgrades so I mean the key thing for me is the source of truth where's that information coming from so you I mean I don't necessarily think an unintended upgrade is going to Lesley fix that problem if you're still using the dock hub as a source of truth and there are some ways you can work around some of the issues with the docker hub things not being patched you can extend from public docker hub images and you can automatic an straightaway you know run your you know your apt-get upgrade and those sorts of things to run those patches when those images get downloaded but not quite I might be missing something but maybe we can talk more afterwards I'm going to go let you get lunch I'm going to be around today in tomorrow I'm on twitter at sam newman if you wanna ask me questions do so there but thank you for your time

Channel: Devoxx

Views: 17,049

Keywords: DevoxxBE2016

Id: ZXGaC3GR3zU

Length: 60min 48sec (3648 seconds)

Published: Thu Nov 10 2016