Moving from DevOps to DevSecOps

[Music] welcome to the sei podcast series a production of the carnegie mellon university software engineering institute the sei is a federally funded research and development center sponsored by the u.s department of defense a transcript of today's podcast is posted on the sei website at sei.cmu.edu podcasts hello i'm suzanne miller principal researcher here at the sei this is part of our ongoing podcast series and this particular edition is coming to you from our homes because we are still in quarantine so um today i have the honor of interviewing hassan yasar my colleague my boss's boss the technical director of the continuous deployment of capabilities directorate and we're here today to talk about the move from something called devops which many of you have heard about to something called devsecops which many of you have also heard about but not everybody has so it's time for you to hear about it so i want to introduce hassan and say welcome thank you for joining us today thanks for having me susie we are part of team as a devops team thanks for having me so before we get into our main topic we usually like to get a little bit even though you've been on the the uh podcast before if you give our viewers just a little background of what it is you do here at the sei and what is it that sort of makes you excited about the work we do here well so i have been working at ci since 2010 i came from industry with the lots of experience in the software engineering software deployment delivery and at sci when i start and really bring that experience of practitioner perspective to the community of dod now i have been leading a group such great group including you and talking about the agile and devops and fc cups and helping the dod mainly but helping the other industry as well it's not just the audit but helping the community to improve their software delivery with faster and better quality another faster and fail faster and better quality now we are calling up a devops maybe in the future we're going to call something else but our intent is always to deliver software faster better in high quality i do like the work because the work is never going to stop we have been demanding more on the software world we have been using a software and everywhere now it is a time to really look at what we can do improve our the level of the software the high quality in the various applications it's not just a web application it is in everywhere the set the mobile platform iot devices and everywhere even if you are talking about ai component right now but still it's a software at the end so i see it really is encouraging me and also to motivate me the work is gonna stay forever seems to me and i'm really happy to see challenging environments and various platform various components with the team cultures and technical stacks so i'm really excited to continue to work i i i will say you know we've talked about this before that every time we solve a problem there's a new one that comes up that's related somehow to software so yes we're still busy i've been in this at the sei for uh decades and uh you know so it's it's amazing how even though we've solved a lot of problems there's still a lot to work on and so devsecops is actually i think a nice example of this um for those that don't know what devops and devsecops are would you just summarize briefly sort of what is devops what is devsecops and why do we even care that there's a difference great question so i'm going to start with about the how the devops started so you came from agile community you know the how the agile started i didn't want to go that route you can cover up that pieces devops started in 2009 with the agile movement actually it's not against it it's a side effect of the agile movement so the why that happened actually 2009 a group of people from operational team they were really not upset but they would like to be do something different in their operation and environment to support the needs from agile team which is dev team are making more incremental iterative changes for their application they are building their focus is more about the customer work their focus is more about getting things done not spending a lot of work in the paper or anything else just spending more under the content of the work can be delivered so operational team they said why don't we change our mindset be an agile in operation world so we can receive the needs from the agile team that started in 2009 as a movement with patrick d boyce and clay shaffer david's as they get together they said let's change our mindset make it as the dev devops wait like not that objective they will operate agile in operation environment they came up the term as a devops term in the twitter handle and they started devops days in back in october 2009 2009 so that term actually stay with the community as a devops it was a devops days originally that how it start so initial intent was getting a developer and operational team working together but people realize that the community realizes it's not enough having a more business oriented more supporting user needs requiring other stakeholder involvement so that work got expanded more including other stakeholders not that even ups other stakeholders are acquisition people architects and the security team etc so when we build a stakeholder it was still oriented for business needs animation and then it's more focusing delivering the capabilities then we go really fast and fast and fast built and security people see the another side effect of that devops movement such side effects are not able to test enough for the security needs not able to test enough for vulnerable analysis not able to connect the community with the security world so secure the kite seems like not unintentionally unintentionally forget include into the devops process you know by definition of architecture we would like to get the secret as part of architecture but it didn't work out with the devops community a couple of reasons i can say because when we start to use more contingency integration concept or continuous delivery concept or mainly using any type of containerization concept which is very hype right now or using an existing code multiple times in existing environment like a platform or backhand they will start to use a lot of ready code so ready code has to be analyzed from the security themes like developer galactica or architect they would like to use existing frameworks or using existing libraries and technologies that type of mindset again not an intentional kind of unintentionally open up the vulnerabilities or mindset into the devops pipeline such as we're not able to address the dependencies such as we're not able to address the enough security testings because we're going to go fast and fast because focus on more customer needs and customer never says under the security customers i need that feature that's that's expected but as an organization that's our responsibility to secure that they need security but if you don't give them security then they're not exactly nobody will be happy to see their data anywhere else we see a lot of bridges happening almost every day nobody's happy to see their information is being sold in a black market i'm not happy but you're not a happy either you know like nobody's going to be happy so we're not saying as a customer we need to protect our data but that movement was not properly addressed that mission specific secured or specific compliance perspective so while we are doing a lot of great work and people likes to see the great impact and we start emphasize more secured into the context of the devsecops so devsecops start and adding a security into the every phase of lifecycle it is not in one elements but adding a secret into every phase of lifecycle that's how devsecops terms started actually have another story in this one and i have been participating the many devops conferences you know that and then i have been also the organizing and running the a couple events with the community together so in as part of the big convention for the security convention which is rsa and rsc conference has been happening almost 20 years so we had a day for the devops days we run as part of rsa conference but we were calling differently we are seeing like a secure devops rugged devops we called many definitions about three years ago as a community we said i know there's a lot of people are calling different things rugged devops and secure devops dev ops sec sec devops something so as a community we asked the audience we said what are you gonna call what is the purpose like i know everybody's calling different things about 750 people joined to the our kind of polling and everybody liked the since that day we're using a devsecops turn as defining the that concept of adding a circuit in the devops process so it's not a in between dev and apps but it is basically covering up all left live so we can open up more that's what the devsec app start actually i like that as a term because it talks it's really saying security is embedded inside dev and ops it's not outside of it and right and that i think i suspect that's one of the reasons that it was popular so what are some of the challenges when you start making this shift both in terms of organizationally from your product viewpoint what are some of the challenges of adopting a devsecops mindset especially if you've already been doing devops what are some of the things you're going to have to think about changing so honestly susie based on the survey we have been running almost five years if organization has a devops as a maturity level when i say the maturity like not really implementing one component of devops so let's open up devops a little bit so i can cover up the implementation detail so when we say devops we are looking for the full life cycle perspective if organization has just a integration server like a jenkins with the repository management with the tooling perspective it is not devops right it is the one elements of covering up continuous integration maybe just not a continuous integration just integration of the code then we adding a continuous term into that definition we would like to see a continue to in the life cycle how i set my requirements what is my case i'm working on it how many sprints i'm running which sprints i am in based on that continuity we would like to see the continuous integration process for any code changes then we would like to deliver the code we've build it that will go to the staging or the any test harnesses then we automatically deploy the production that will continuously deliver it so all this fashion is all connected either a tooling perspective or the process perspective process our running agile process process how are we going to do interactive incremental development and technology pieces are how we can architect our component that will support iterative climate development so all this continuity is really helping eventually add a secret into it so based on the server with what we find out if organization has good maturity in the devops perspective and practices with the process and technology they were able to add a security into their devops process more than 300 percent it's interesting not a couple times 338 percent they were able to get better or faster adding a security why is that so starting from ground up adding a security component it's not going to work out if you have a good devops environment it's very easy to secure the component into it why is that possible multiple reasons we can open up so one is we have a team structure already available so we can either secure that into our daily scrum if you need to we can add the pi planning we can add the sprint planning postmortem etc with the process section of it we have a platform that the security people can share the information with the team members easily can share that so now we are covering up the technology pieces now we have automation is in place which is doing the integration or the testing or automation perspective that's what the security people they would like to see as an artifacts so if organization has a devops they were able to easily add a secret into it and now we are calling it devsecops right now we saw that as based on the statistics one of the things that uh in working with my customers i know that the security people have appreciated is that because we've got this idea of a continuous pipeline we're testing the code many many times as we make changes and that builds a body of evidence for the security community that vulnerabilities have not actually entered in so even though we're changing the code the fact that we're doing all this testing all the time gives them confidence that many aspects of security testing not all of them but many aspects of security testing they can kind of check off and say yes i have confidence i have evidence right that i have data to support that that this aspect of security has been dealt with in the way that they built this code right and and that opens up the security people i've talked to said that opens up them doing the more creative you know sort of uh interesting threats that they don't always have time to think about because they're really just dealing with a lot of the mechanics and so that that's one of the benefits that i've seen with my customers right what are some of the other things that you've seen in terms of the benefit to the security there are many benefits before going to the benefits suicide forgot at previous question there are some road blockers still which is the trust problem so like the think about the two communities now typically and i like that for a while for as a developers and i kind of played that role as a secure i was i was thinking acting i think we should do some play with the how the interaction between interaction between developers and security right i think we should role play sometimes so playing that role for myself and i realized that also i'm teaching a software segregated class at the cmu i know how the secret things since our thinking are there is a trust problem between two entity why because first of all security people are assuming that developer knows the concept of security they know what the vulnerabilities are they know the best coding practices are they're expecting they should know the all the network stack and ips and the buffer etc but in reality developer is not trained with the security they're not trained they don't know i mean like i have a student they take a class like exactly so like you might i if they if they have time probably yes but in my class and i have about 30 35 students susie i ask that question all the time like most of them are the cs departments graduated and good universities in all around the world it's including cmu they don't know the concept of the security i know they innovate a high level but they don't know the what needs to be done with respect to the technology with respect to the tools and also they don't know what is the specific networking and stuff because they train writing a code or building a systems good architectures or good database design or good algorithm or anything else but the security is a living thing susie when i say living things it is not as static as learning some language it's a native tanks because you have to really follow up up to speed come up in the technology of concept new idea so hacker community or the or the adversaries communities are very up to date so really difficult to developer following that what is really happening is a trend understanding the all the vulnerabilities all the possibility of thinking implementing the code so that's kind of like a without knowing each other's now becoming developer are blaming the security fox as the gatekeeper and blocker secretly people are blaming the dev team are creating a crappy code so it's basically crabby code is a bad code it's creating a evil thinking between the both parties but both intents are delivering a good soft face so develop intent is really fixing a lot of functionalities but the security people intend is making resilient systems so they don't really converge that yeah and how we're gonna build up that trust as you said in your statement susie we can at the build up the component as the people can trust each other so how they're gonna trust each other that's the what devops comes in the picture because as a security people if i know developers are working such component which is libraries and code then i can see a full transparency like a full application lifecycle including the artifacts including the backend stuff it makes me comfortable what developers are working on it so i'm not going to look at a suspicious way they will operate anymore because i can see how they are working what they are working what type of they were using it you know me but i can give them a you know open like kind of environment they can use the existing library type already verifies the security people so basically transparency is building up a trust between the two entities and second things also with automation concept automation testings and analysis stuff now the security people can see that how we are able to speed up the process by creating art effects as a security teams right and also in other things is important developer will use the existing operating systems or the boxes that secure the team or advising will then give them enough confidence that hey i'm a security person i already gave the what operating system they are going to use and developer you lose that they're not going to use by themselves and pulling somewhere in the internet maybe going a docker hub pulling something from docker hub which is creating kind of trusted sources and security people will use that so biggest biggest benefit for me for both entities creating a trust which is important trust is based on the transparency and visibilities so they can see what is going on they have a full visibility of the pipeline and which is creating a lot of good input for the secret another important factor now going back to the other direction so whenever the security people are analyzing the code they were able to send the feedback to the dev team as a digestible vase like if there is no devops which i work in the years when i send my code to the secure the teams they were analyzing the code they were sending me a 50 pages of the pen testing result yeah do you think that people have enough time to look at the 50 pages of reports eyes roll in their heads exactly nobody has enough time with the mindset of devops now we are using a lot of static analysis tools or dynamic analysis whatever the tools we are using it we are able to create a feedback on time and a specific context another important factor is at the beginning so we are expecting that everybody knows everything it's the wrong assumption we it's not nobody knows everything i mean if everybody knows everything i'm i'm happy to hire it we don't have that people we should not be that either because we're not we're not going to create a a person knows everything it's going to be a bus problem eventually but we would like to increase our team knowledge so team knowledge should be as the institutional knowledge in the organization where we are building so with the feedback mechanisms with the right tools sending feedback to the chat window that can be a heap chat can be any type of mechanisms they were using it the feedback has couple components which is almost all aesthetic analysis house result of the analysis what the problem it is what the solution it is so technically we are we are teaching the developer what the solution is so revealing the knowledge now the secret themes instead of sending 50 pages of documents now the system is sending the information as in in a way that developer will know what the problem it is how to solve the problem which is another great benefits which is continuous feedback we are calling it right now we're expanding feedback including seeker nada not a business not a user now the security teams are able to send feedback to timely the developers what needs to be done so these are all things that and automation the automation of the tool ecosystem i mean it's an ecosystem now it's not just a pipeline but the automation uh inherent in the tooling ecosystem is one of the things that has really made a lot of what you're talking about possible when i was writing code in the 80s you know that was not feasible i remember the very first automated testing software that my company brought in for us to look at everybody would just laugh at it if they saw today you know how cumbersome it was and 50 pages of of things to look at i mean it was not a useful um way for us to learn about solutions to different kinds of problems many organizations struggle with this and um you know what are the right tools and and do we help with criteria are there ways that we can help them not say oh you should use jenkins or you should use this but help them understand what are the choices that they need to make when they move in this direction and are there criteria i mean you can say things like you know your tool set is going to have uh you know low vulnerabilities okay that's great but if i'm evaluating you know a an offering for doing containerization or something like that how do i know how do i what kind of criteria do i use to say this is going to be more secure than that are there do we have artifacts resources that we can help people with in that area excited so let's open up that question a little bit further so i think as we said at the beginning when we get the devsecops we would like to have a software delivery cycle should be in place which is devops we are calling it but what are the principles of devsecops are the principles are communication and collaborations principle automation infrastructure record and monitoring which is sharing so all this concept is is the must-have things so if i'm picking your tools if tool is not able to communicate with me in a proper way which is i'm trying to address the communication principles of the devops or devsecops let's call dev setups now also they want to confuse anyway so how can we address the better communication collaboration that tool i'm using so as now we are saying like try to address some principles of devsecops how can i do automation in the devsecops pipeline or how can i add a monitoring concept into the pipeline so all these principles we have to achieve in our devsecops pipeline if i'm going to pick the tools we have to say what is the reason why i'm going to pick these tools in this the integrated pipeline concept what is the the puzzle it's going to fit into the my pipeline when it's going to fit into my pipeline can i integrate that into my pipeline which is integratability can i get a data from that tools for a monitoring perspective can i use the infrastructure pieces like standing up the tools if i need to can i deploy independently can i maintain that tool independent which is interoperability so can i swap it out exactly right how much is flexible at this so two should give me that perspective so i picked the reason that i'm going to use that say i'm going to pick a tool that do tool with a aesthetic analysis for me great so now i'm going to ask a question myself can i ask the static analysis tools into the my pipeline is it connected with my build server maybe a jenkins maybe a team cd maybe a bamboo it doesn't matter what it is can i integrate into my pipeline can i integrate into my idea environment i saw some tools are capable to really give the feedback early in life cycle the earliest feedback we can give in the life cycle when they develop write and get caught when the developer are writing a code they were using an ide integrity development environment so ide is a good place to integrate the tools as an example if i'm able to as a kind of tool owner if i'm able to give a feedback to the developer when they're writing an object or a code give the syntax error give the some type of code practicing when they write the code this is the perfect way of shifting left the secret the concept all the way to the left side which is more early in the life cycle so now i'm looking for tools perspective so i can integrate here can i integrate that great i integrated am i able to pull data from that tool so i can monitor quickly in my life cycle now we are addressing a monitoring perspective now can i share the outcome of that tools into the various collaboration portals like issue tracking systems can i tool create an issue for me instead of creating manually see now i'm connecting to the the issue tracking system see that now we pick the tools we put in the place we are asking the principles of devops how can we increase the communications how can i do automation how can you know the monitoring perspective if you are able to address all these common principles that's the biggest criteria but what we'll start the first was we start why are we going to use this tool what is the purpose it is very important we don't want to go to tools that we like it we don't want to go to tools it's everybody's talking about the communities everybody's using cucumber but what is the reason how they come up with these names i have no idea right so like exactly like use the tools for our needs and i give another analogy maybe i give many times before i always describe when i teach the class say when you go to the home depot you're looking for a tools which we do all the time for like a you know some work in our house and some little projects we always look the tools that meets our project needs we don't like the tools that we like it we are not paying a lot of money we have a project in our head we already put the list of it we just bind it to what we need to do for a specific project may not be an exactly the same thing but in software in the same world we have a project we have to look at why i need to do that what is the purpose you know what what does that bring is actually really relevant because there are people i have some in my family that go to home depot with a project in their head and they get completely distracted by all the tools that they didn't know were available to do something like that and then instead of coming home you know with a screwdriver they come home with this automatic thing that you know dials around you dial in what you want and the right screwdriver thing pops out and it's like i just need a flathead screwdriver what are we doing here and we see that in in our tooling environments in devops and devsecops where you know all of a sudden people have said oh we have to do that and you know this whole list of things and nobody's asking the question that is important right why why and so yeah i definitely resonate with that and i love the example because i resonate with that too that's end of the day we are creating a lot of technical depth either keeping in the garage never use it at all not effective in the software role in a worst case scenario we can sell the tool in a yard sale probably but in the software world it is done we cannot say anything it's not you are we are part of it and then the biggest biggest challenge actually in the devsecops it's a sustaining of environment susie we build up the tools we build up the pipeline who's going to maintain the pipeline who's going to update that if he distracted ourselves a lot much about the fact that it's not just the technology environment that changes but our threat environment changes the hackers that are doing you know ransomware didn't exist 10 years ago nobody thought about you know the the way that you hold somebody hostage you hold their data hostage to get money out of them that's but now that's a thing that we all have to deal with and i don't know what the next thing is i'm not sure i want to but you know there's always going to be something new in the threat horizon and threat environment that we also have to pay attention to so this is one of the reasons this is a never-ending problem set because there's always going to be people thinking of how do i get around you know what you the blockade that you've put in front of your code how do i get around that so that i can you know do something that you don't want me to do right it's a good point susie like what is next when we say what is next we would like to be resilient enough we should be ready for any bridges down the road so how can we ready for any type of disaster situation like a bridge maybe ransomware is one of the examples maybe other bridges in a software world to be ready any type of bridge we have to know what we have our assets which beat our code if something happens we should be able to respond quickly for any bridges response may be fixing a code response may be a patch of the software if you look at any bridges i can analyze the bridges in in my study as well the most of the bridges data bridges is based on the common vulnerable which is maximum 24 maximum 24. out of 24 and mainly are the input standardizations very basic in the in the cw examples so that's the common thinking but to fix the very basic vulnerabilities just the input sanitization we have to look at the code we have to recompile push the new version of it so we have to really know that in the context of software world so you have been in the world almost 30 years when we switch the context of different the tools or different program or different projects do you remember anything six months ago say that yeah especially in covid world i don't even know what day it is sometimes yeah six months ago i just know that six months ago was pre-coded right for the for the coder for the developers not useful as you say not useful for trying to fix something i did six months right so now that we need to have the context to get the context in the software role we have to know we cannot go back to six month we don't know the time travel but we have to look at what artifacts are what things test we have done it what the dependencies are where the version it is so we can quickly look at our code and fix that specific very basic vulnerabilities that is required back to what you're saying earlier about we we have to have that transparency right i have if if i'm a security person who understands input standardization for this particular type of data set i probably understand that better than the actual coders so if i can look at what is out there if i can look at the code and i can see what's there i can help them to say oh this is the spot that probably caused this problem let's fix that where it may still take the coder who originally did it a longer time because they don't have the same mental model of what it is they're looking for so this teaming of the security and the coding community and facilitating that through these many many new ways of communicating i think is one of the things that's making a difference in the environment exactly and we didn't say another word it's another another buzzword it's a real word is traceabilities devops is giving traceabilities so exactly we know when we develop the code which components are what type of library has been used we can automatically fix it or pull up and then push the new version so we are racing and then another things i have to say susie the unfortunately i don't want to say the dark like you know the communities adversary communities are sharing more but as a team for the software developer engineers or the good people we can say we're not sharing enough information between us we don't have that type of sharing mechanisms but the advisors are sharing a lot they are faster they are sharing so we have to be proactive in the concept of the security productive means we should be ready we should be resilient we should be sharing information amongst each other so we should be ready if something happens which brings up another trust issue um which i think is one of the things that prevents the people that are trying to protect their environments is who do i trust who who who is it who is it safe to share with and and now you get into a whole bunch of different layers of trust and um you know a whole nother topic is the zero trust architecture kinds of stuff but but i think there are um challenges and i think the kinds of things that we do with the devsecops days and and things like that um you know conference events we know that there are bad actors that go to those just like there are good actors that go to those kinds of things and i think the resilience idea that you're talking about is like we have to just accept that and understand that whatever we say to anybody can get perverted but at the same time we can't be so paranoid that we don't say anything to anybody because then we're going to miss the new ways to protect ourselves in different areas and so um you know it i mean i i don't want to oversimplify this for people this is a complex area to work in but we were paralyzed if we don't do anything and and we aren't able to take advantage of the wonderful technology i mean just look at what we're doing right now i mean five years ago if coveted happened i don't even i can't even imagine how we would be having this podcast oh yeah and if we didn't if there wasn't a certain amount of trust that we had with whoever provides the video and the audio and you know the data going across the links we would never be able to do this and so you you have to make a decision that you're going to act in a way that is resilient but is not a way you know you can't be naive but you can't just stay paralyzed either sorry i got a soapbox there you're great you're great why don't we wrap this up by by helping the people um that are going okay so what do i do next if i really haven't been thinking about devsecops what should i be doing and what kinds of resources should i look at look to the sei for to help me in making this transition because i need to make this transition great so we have so many resources either through the sci or through the communities so we have a space for devops we have a space for agile i really encourage our listeners and take a look at sci resources we have been publishing the webcast and webinars blog post and technical write-up technical paper research paper we have been publishing and also as you've heard of it suse we have been hosting deficit cup stays now it's another one is going to come up october 1st as a virtual event so please participate that type of events as you know the attendees you will get a lot of lesson learned from others but we have to be really careful because we don't want to be exactly copying and pasting the same infrastructure we should remember that why questions what is my project why i'm going to achieve that what is the purpose of it we should remember that based on the what we know what we have internally let's build up devops if you don't have a devops start the build up devops environment then we build up the devops angular the secret to hawks at the beginning into your pipeline either a process with the team cultures or the tooling perspective that has to go together parallel don't just literally just build up the pipeline say i'm going to build out the pipeline first and then go look at the process later on it's the wrong assumption or just covering up the process we don't get to learn wrong so we have to go as a parallel we have to go like a raise up and raise up and we can get better on the three classical term process and technologies and then then talking about the people section so we have to go together we have to train the people we have the right processes in place we have the right technology balance so we can increase the bar me from devsecops there is no limit on devsecops we can get better and better and better because it's it's a journey we're just looking for what things is not working we can go fix it sometimes we may see some training problems we may see some tool problems we may see some process issues we may see some compliance perspective so we may see many type of the barriers and we have to enable also the barriers based on what we see is and then that's what enablers are all right so i want to thank you i always enjoy having a conversation with you about these topics i think the next one might be on sort of what are the future expansions of that middle part because we're hearing about people saying not just devsecops but dev devstarops as in devress up so so that can be our next conversation for our viewers we will have in our transcript links to all the kinds of things we were talking about assets of different types the sei has provided community portals and things like that so you can get that all when you look at the transcript we will also have this podcast available in all the places that you get all your other podcasts whether it's soundcloud or other places so please do look for this podcast and share with your friends thank you for joining us today thanks for joining us this episode is available where you download podcasts including soundcloud stitcher tune in radio google podcasts and apple podcasts it is also available on the sei website at sei.cmu.edu podcasts and the seis youtube channel this copyrighted work is made available through the software engineering institute a federally funded research and development center sponsored by the us department of defense for more information about the sei and this work please visit www.sei.cmu.edu as always if you have any questions please don't hesitate to email us at info sei.cmu.edu thank you you

Channel: Software Engineering Institute | Carnegie Mellon University

Views: 946

Rating: 5 out of 5

Keywords:

Id: tdUm8uxRj14

Length: 40min 40sec (2440 seconds)

Published: Thu May 13 2021