SEI Research Review 2022 (Day 2)

foreign [Music] foreign [Music] [Music] [Music] [Music] foreign [Music] [Music] foreign [Music] [Music] foreign research review for 2022. uh hopefully I'm welcoming you back and you were with us yesterday if you weren't I'm going to cover a little bit about what we did and yesterday was really about our research work in bringing AI to Mission so the SEI does a lot of work in AI engineering and applying a new AI techniques to the dod and if you missed it you missed some great Talks by Dr Hein on AI engineering for uncertainty Dr McMillan on how to bring inference engines to Portable and high performance for the Tactical Edge Dr Lewis working on automating mismatch detection so that we can deploy AI systems more successfully Dr Gallagher who worked on deep fake detection and machine learning to how to do that and Dr Lau who worked on AI evaluation methodologies for defensive cyber Ops it's not enough to just have some good AI algorithms we also have to get those algorithms in a usable and Deployable way for the dod and so if you missed it I encourage you to go look back at the recordings when they become available I just want to give everyone a heads up that we're experiencing some network issues uh today in the Pittsburgh region we're doing our best to kind of overcome that but you may see a few bumps and hurdles along the way and hopefully we won't lose any of our panelists for our next panel that's coming up and I'm going to Pivot kind of right into that uh so the panel that's coming up is on sorry look at my notes it's transformative gamification I'm making an impact across the federal Workforce from cyber Readiness to AI and I want to be clear here that this is a panel this is not a uh kind of recorded talk as it was before so we're really looking to have this interactive so I want to encourage all the all the visitors out there that we have watching this to interact with us ask questions on Discord ask questions on YouTube we promise that we'll see them and you know take a look and try and work everything into the conversation uh I would encourage you as the MC since it's not my panel to uh you know see how much you can get the panelists to talk about your questions and engage that way instead of let them get through any kind of presentation that they might have uh prepped so I'm challenging all of you to kind of make that happen as you can see we have some great panelists here I'm not going to kind of read all of this you can um rotem is a research lead he's been involved in kind of uh various uh parts of research across the SEI for many years and really is um very very passionate about gamification uh Dr Hammer as you can kind of see she's a professor with the uh Carnegie Mellon and then the human computer interaction Institute and the entertainment Technology Center and so uh you know deep expertise in this area and Dominic Ross is someone who works across our media and uh kind of capture uh so you know how do we do gamification and make it real have the quality there um what are the the technology capabilities of kind of doing that uh really helps with that uh works on our kind of multimedia research and and also a long time collaborator of rotem so he's also very deeply involved in gamification and so these three are going to you know kind of really focus on what does it mean to have that transformative gamification and you're going to see all sorts of kind of Cutting Edge and uh techniques and what's you know kind of out there on the research side I would absolutely ask them you know kind of questions about readiness and you know when when do you think soldiers can really make use of this what about the kind of generating forces opposed to the Tactical force and so what that I'm going to pass it over there and by all means again please ask questions and excuse us if we have some network issues and you might see that some of the participants become kind of a headshot as opposed to kind of a live person that we're doing that to preserve the audio so we can still have a kind of a good audio stream for the conversation and with that I'm going to hand it off to rotem and Dom and uh and Jessica thanks everybody thank you Chris so uh yes thanks for that uh warm introduction uh the the three of us are going to be discussing focusing our discussion today on transformational games how can we create experiences that change our audience in a meaningful way change their behavior change the way that they uh interact with whatever the the topic at hand may be um I just want to very quickly make sure that you're familiar with all of our voices at the very least so uh Jessica if you could uh say hello yes hi looking forward to speaking with you today hi and uh Dom yeah great to be here loud Bergen to talk about gamification again here at the SEI research review and uh we have a lot of material to cover so I'm going to jump right into it that being said please give us your questions your comments we're more than happy to discuss what interests you so uh that being said uh let's jump right into uh we had a couple of different uh topics here that we wanted to cover one of which is Hands-On experimentation uh this is one of the areas where DOD has some of its longest history uh in uh applying techniques from gamification uh you know since the 1800s DOD has been conducting war games of various types one of the benefits of War Games is you get rapid feedback you get to iterate on experimental scenarios different techniques tactics techniques and procedures different hypothetical even weapons platforms whatever the case may be that you want to investigate it's a very uh low-cost very accessible way in order to try out new doctrine new techniques whatever the case may be that being said as we move towards the modern era with a lot of of the new capabilities that we've been bringing to Bear there's been somewhat of a divide forming in the in the community that we feel here at SCI is almost uh unnecessary there's essentially two broad categories of wargaming that have uh grown out one is a data generating heavily analytical activity where the uh the bounds are very tightly constrained there is a heavy analytical backing to each resolution to each step that's going going into it and you know a lot of computer modeling involved that you can generate data create analytics uh you know really determine what uh you think the outcome is gonna be and then you have this other uh almost uh the term bog sat gets the uh thrown around a lot of for a bunch of guys sitting around a table um we're obviously not uh advocating for that but that being said there is this explorational uh side of wargaming this interactive side of wargaming where you bring the humans into the loop you allow them to express their creativity you're able to actually try out new things and this is where real revolutionary changes can get discovered where there's a lot of value here however that doesn't mean that you have to give up that analytical rigor and the the techniques from gamification are exactly the way that you can bring that that analytical rigor with you and still allow the expression of that creativity and so that we feel very strongly that these two are not mutually exclusive so rotem essentially what you're saying is wargaming kind of provides that sandbox environment to better educate not only the participants but also leadership as they go through these examples yeah absolutely it's it's usable at every level and it's most useful when you're actually spreading it across the different levels of of the Enterprise so that you know the the war game exercises that you conduct at lower echelons can Bubble Up and provide the data backing the support to experimentation at the higher levels so that you're still talking about reality we don't want to divorce ourselves from you know the real world even if we're stepping into the near or even farther Horizons of the future we want to make sure that what were the the data that we're getting at the results we're getting out are realistic but at the same time we want to allow that room to explore to try something new because otherwise you're just going to get people that get together and then reinforce their previous prejudices and so you know I think one of the places where there's real opportunity um is not just sort of bubbling up the results of the games but actually thinking about the process of uh designing the games that when you're making a game you're making a hypothesis about reality you're modeling a future in some way and so um there's the opportunity not just to learn from the process of gameplay but actually to pair uh experienced game designers with people who have deep domain knowledge especially if you're trying to imagine some possible Futures to uh uh test hypotheses not just about how One Vision of the future might work out but actually to look at multiple possible visions of the future and compare across different models to understand directions that um uh sort of uh of scenarios that people might face at the same time developing the human capacity within the organization to to better and better articulate those models in the form of games and and growing it natively within DOD I think is one of the key things that you know we love to work with our partners in order to enable them to develop the games using the Frameworks and the tools that we create for the very reason that there is a lot of value lost when you go out to an external provider for a war game that they've created because you're essentially living in their mind in their in their preconceived notions of what that war game should entail instead of being able to allow your staff to go spend that mental effort to understand what they see the future of the fight is going to be how this should be conducted leveraging you know experienced designers so that the game can be conducted in a uh you know a useful manner but when you Outsource it to these commercial providers you're getting their prejudices baked into the system essentially and so you're limiting yourself on how you can explore well you and it can't be understated that there is a substantial Financial investment into war gaming from Staffing the running the running of the actual competition or game itself as well as any type of scenario that's created so one thing we want to make sure is there's reusability and that's where gamification really helps with that absolutely and that's another uh really useful tool for some of the the Frameworks that we've created where if you want to make modifications to it it's we you always want to ensure that those modifications are low cost if you want to try something new if you want to experiment it's great to provide that sandbox where that environment allows for it I see uh here you're you're showing one of the solutions that we've provided in the in the past which was the Cyber kinetic effects integration system where we connected our existing cyber range capability to a kinetic Battlefield simulator in order to allow for Joint Forces uh training and experimentation so that we could actually have our cyber operators you know can supporting a real world you know kinetic operation so that they and all the constraints that are involved in that and so that they could experiment okay yeah we could do a Cyber attack and take out the the power grid here or we could have a team drop a satchel charge you know on the on the um on the sub power substation you're still going to get a blackout either way but what does that affect what does that inform how much more you know uh uh prepared and alert or your Defender is going to be after something blows up versus the power just goes out right and so it really lets you iterate on those scenarios rapidly and see what works what doesn't what are the implications because for a lot of the the community sometimes those new techniques the those new capabilities that are coming on board aren't as trusted they know what a satchel charge is going to do they know that after that Central charge goes off that power will be out and it will stay out how reliable is the Cyber attack how you know how much attention is it going to draw these are factors that aren't as well or at least weren't as well understood and so you're able to do that experimentation and you're able to iterate and that quick iteration with that quick feedback is one of the key aspects of these uh simulations because I will tell you when we ran this particular scenario two two outcomes from from it that I think were very informative was first of all when we developed it entirely in-house the first piece of feedback we got when we showed a very very early iteration um of it to our DOD uh you know our experienced Partners um was you you guys need to stop watching movies nobody nobody's conducting an operation uh this way and that we were able to bring in that expertise and create a scenario that really reflects um the way that the Department of Defense would uh uh address a situation like this and then not only that once we put the real operators in that environment those teams every single team failed on the first attempt nobody succeeded but it was with that iteration with that uh experimentation that they were able to uh refine their plans refine uh their ability to go and conduct the mission this particular mission was a hostage Rescue Mission but they were able to rescue the hostage get everybody out alive yeah but we learned a lot by failure right right rotem I mean I'm sure Jess can talk about the practical application of failure within gamification techniques it's important things yeah change the way people fail uh the way I talk about it with my teams is we talk about failing gloriously or how do you fail in the way that helps you maximize your learning and your learning outcomes so with that actually we can maybe pop back up to the different approaches that uh uh we can use for gamification yeah so if you go to the uh top right one I think you're yeah you're yep yep so that's right increasing receptivity and so go ahead oh great so so you know we know that not everybody responds well to failure um for example when people fail in a training exercise even if there are no formal consequences for that people can become defensive right and they can blame their teammates or blame uh uh something about the situation instead of asking how can I learn and improve so one of the things that gamification can do is they can actually change the way that people respond to challenging situations like failure um they can uh so for example when someone fails in a game their responses often oh gosh let me try that again with a better strategy right let me see if I can figure out a way to solve this problem um and we see people doing things like asking for help expressing ignorance when it's about failure in a game when if it were for example failure in the workplace they might be much less likely to do so so to give an example from from from our experience you know when we have Personnel that come in and this was a while back before we started exploring the these techniques we were doing our best to recreate the exact copy of their uh uh of their environment make it you know as accurate as possible to replicate their office environment and then we every time we would get the same complaints this is different this is different and we and what it became apparent is the perception that they were having wasn't because this was Personnel that were already in the role not preparing for the role their perception when we said hey here's a a better way to do that what they didn't hear is here's an opportunity to improve what they heard is you're bad at your job and we're going to tell your boss and so instead of listening to it being receptive and improving their capabilities they would start getting defenses they would defensive they would latch on to any differences in the environment that they could find as excuses and the one that I always give because it's as close to as the sun was in my eyes as I've ever heard is oh the background on the desktop is a different color and the alert is the same color as the background on the desktop so I didn't notice it because it Blended in but they would have been different colors in the office yeah there's a certain point where you realize these aren't you know valid criticisms it's their defensiveness and that's when we switch to using semi-fantastic settings we put them in a role that isn't their real role but still exercises the exact same skills the exact same techniques the exact same systems but now you're not you your special agent whatever you're not fighting your normal day-to-day uh um engagements you are resisting an alien invasion you are stopping a mad scientist never the case maybe and as you talk about that right Tim actually we got our first question from Jason off of YouTube uh does gamification disconnect people from reality if so is that viewed as a negative or positive yeah I think I think go ahead I was just gonna say you know as with most things with games this is not a question with a yes or no answer that um gamification Can distance you to different degrees and in different ways so for example there's a body of research showing that just thinking about yourself in the third person versus the first person can help you um uh reflect on uh what you're doing and use change your strategies and be more distance and less sort of like caught up in your own emotions gamification can do that right just moving you to from first person to third person it can narratively disconnect you in the sense that rotem was saying right oh well you're not yourself you're uh you know a member of your super agent but you actually have the same job you currently have um and different strategies have different benefits so for example when we put people in semi-fantastic situations um we know that it helps them let go of their ego because the situation is giving them positive reinforcement of you're cool you're special you're valued you're part of a team and failure is normal in this context because everyone involved is failing right so um uh it can be a powerful tool if you execute it badly right it can not work very well but part of um the sort of uh vision and the reason why we wanted to have this panel is to make the case that um you know there's a there's a role for craft in aligning the benefits of whatever gamification techniques you're using with the goals that you have for your mission and now I'm going to point back to rotem because uh he's going to tell us about this video oh yes so um this particular video is of uh one of the first iterations that we actually made of these semi-fantastic settings and it's getting to exactly the point that you were making Jessica of you can't just apply gamification in a willy-nilly fashion it has to be done in a thoughtful manner I've seen far too many times when people slap a leaderboard or badges on something and call it gamified at the end of the day what you're trying to do is match your needs to the techniques that you're applying one of the key things that we look for in the way that we create our gamified environments is that the tools the techniques the things that you're actually going to do match what we are trying to train you to know how to do and in order to do that we we you know we ensure that you get positive training value and not have negative training value we try to replicate the experiences both on the social like who you're interoperating with and Technical levels so that you can take that experience that you've had in this semi-fantastic setting and natively transition it back into your work role yep so I say we have another question coming in from Discord this time um asking if gamification results cross over to updating ml models and the tne strategies especially considering modeling and Sim techniques well I'm glad you asked that uh I uh Dom can you jump again back I apologize for jumping all over this presentation but as we said uh we are uh absolutely looking to uh you know answer your questions uh primarily um and jump to the uh the Hades uh material I believe that's under uh the uh improved teamwork uh heading things like the last slide on that one so we've actually done some some work in that in that area um yeah one more slide down there you go so uh this is the human AI decision evaluation system this is a gamified environment that we use to actually test the output in this case rather than uh just creating trading data but test hypothetical output of uh AI enabled decision support systems so that we could actually run humans real humans through a gamified environment where they would you know they had a narrative they had decisions that they were making in context and then receiving different uh support from various uh possible AI decision support systems and the the key part is we designed the system such that you could after you've developed the system pull out the hypothetical part and put the real system in and still use the same game continue iterating through it and see how it affected people's behavior and one of the things that we you know we had some surprising results there when we found things that you know we as you know AI researchers we have a decent understanding of what's going on in a model behind the scenes when we see you know 98 confidence on a recommendation we think oh it's the model is pretty confident in this result it turns out when you take a general audience and they see 98 confidence they they don't think oh you know as opposed to no confidence measure provided you know if there's no confidence measure they're like oh it says yes so it's yes if they see 98 they see uh so it's not sure and you know it's it's a not the the result that we expected from that interface and so we're able to use these techniques to um collect the data that goes into developing the next generation of recommendation systems the same goes through for creating training data you can absolutely collect data from run-throughs of various uh gamified environments to get the data you need that might be either dangerous expensive or otherwise prohibitive to collect in other methods for training your data yep and uh actually I'll I'll add that um uh uh one method first for uh uh uh collecting Trinity that is actually two two slides earlier right we showed a screenshot we were talking about this in the sense of uh how do we get large teams to coordinate effectively there's uh what's called game sourcing techniques right where you have lots of people playing a game together and the total contribution is creating some kind of knowledge or some kind of data uh so next slide uh what we're looking at here is a game called uh fold it where uh I think they had 57 000 co-authors on a scientific paper that introduced new uh protein folds because so many people contributed online by playing the game and this is an approach that we can use to generate data sets as well right so um let's say we want to create a a machine learning model but we don't have an appropriate data set for training we can actually use these game-based techniques to get appropriate data from people who might not otherwise contribute their data to this kind of effort and so my research group one of the things we work on is actually new ways of using games for example looking at game streaming as ways to get people to contribute data to data sets that otherwise we might not have access to and then our models might be trained on a data set that right leads the model to to make mistakes because it has whole sort of groups of data missing and so one of the things that I wanted to note here is an excellent very relevant right now is you know GIS Arda right so this is a homegrown capability being used you know in Ukraine uh right now that uh is actually it's not I don't want to call it gamified of course but it is allowing us to coordinate these large teams and these distributed capabilities in a method that we can leverage it has direct applicability to DOD use cases um I do see that we do have a follow-up from was that YouTube that was from YouTube for Jason that's right yeah so uh Jason had a comment in response to our discussion that says that makes sense as a gamer myself people always try to explore exploit game rules instead of being present in the game sounds like these are curated experiences to avoid that scenario so uh again uh Dominic you could jump back to the the title slide this is how we're proving we're live right this is you well if you could go to align incentives because I want to make the point Jason that that the the the way that they always try to exploit the game rules isn't limited to the gamified experience that that we've created and actually the gamified experiences can fix that problem so the the the the comment I always like to make is you get the behavior you incentivize not the behavior you ask for you can ask for something until you're blue in the face if your incentive structure doesn't match it that's not what they're gonna do and the classic example here is actually with uh Rules of Engagement we can come back to the slide afterwards Don but I just want to jump ahead to uh the synthetic client help desk slide if that's okay yeah perfect and so we actually use cheating as a benefit so we had the case where we had repeated exercises with you know DOD Personnel that we would provide the Rules of Engagement and then score on something unrelated to The Rules of Engagement essentially we would say defend this network and here's all the things you're not allowed to do in defending this network and then we're going to score you on even though these things would be really effective for you but you're not allowed to do them because it would mess up the exercise and then what are we going to score you on whether or not you got compromised so what is the first thing that they they would end up doing whether intentionally or otherwise they'd end up doing some of those things that they weren't supposed to do because they're just trying to secure their Network and so you know we started thinking oh they just don't understand the The Rules of Engagement or they don't you know appreciate it so we're going to give them a quiz they have to answer correctly before they can even start the the exercise again what they do they violated The Rules of Engagement but what happened when we changed the evaluation criteria we aligned the incentives what we did is we created the system called the synthetic client help desk and we distributed thousands tens of thousands actually of automated fake users into the various enclaves that they were defending so that now it wasn't just some random Network it wasn't cyber for the sake of cyber it was an actual Mission they were supporting but the real Innovation there is that we then instrumented these fake users so that when something broke when they you know couldn't edit a document or an email they said they were waiting for didn't come in or whatever the case may be what did they do same thing a real user does they picked up the phone and they called the help desk and opened a ticket or at least what we would hope users would do but we created an automated ticketing system that would then triage these tickets dynamically and put them in the environment and now what happened in the key Insight is we made that part of the scoring keeping your number of Open tickets low is part of the scoring and now when they go and push a uh you know blank application white list so that literally nothing can run in the environment instead of getting a perfect score because you never got compromised your score starts plummeting because nothing can work and everybody's business has ground to a halt all of a sudden now they're super conscious about making sure that they're maintaining the ability to achieve the mission while still defending their Networks yeah we get a great comment here actually from Discord it kind of Echoes that rotem which is you know you get what you incentivize measure and reward 100. and I won't actually amplify so ample here is talking about Behavior that's fully within the game right fully within the simulation how did they change their behavior some of the work that my research group has done is looking at how people change their behavior outside of the game in order to get better at the game for this kind of like cheating as a benefit so for example we worked with a group of role players who played a game called ours Magicka it's set in the 13th century and I was really interested in uh their play because they would all with the exception of one person I interviewed them they said I hate history history is really boring I said okay well what did you do this weekend Oh I went to a local University to read about 30 13th century baptism theology I said is that history they said no that's playing the game okay that's that's how you play the game right and they were doing this on their own time because they wanted to get what they wanted inside of the game so I think that there's a lot of power here that isn't just about incentivizing measuring and rewarding within the frame of the game but also changing the way that people think about their learning activities and their commitment to practice outside of the game itself and getting them to do things that they might not otherwise do independently and take initiative yeah Dom Could you actually go back one slide to the one that we skipped because uh Jessica you bring up an excellent point is I talk a lot about incentives and all too often uh when people hear incentives they think payment and what we're really trying to get across here is that we're not talking about payment here we've actually we you know we've seen in our in our research that payment is actually one of the worst uh uh motivators once you pay somebody enough that their basic needs are met and they're comfortable additional payment isn't very motivating but the things you can get out of people when they're emotionally invested in a narrative or when you're speaking to really anything that's a core motivator to them be that competitiveness cooperativeness caring for the you know the success of the rest of their team getting recognition or status there's a lot of these different motivators that you can key in with your gamification techniques you can key into those motivators and get them to invest of themselves during their free time nights weekends this is really the way you can stretch your training Dollar in a in a in a manner where you'll get a lot more out of your audience when they're engaged when they want to do it you know the the example I think that you gave in one of those research papers I read was with civilization and so there's the there's the classic meme of like oh just one more turn and it's morning you know uh I'll be honest I've developed lots of cyber security training I'm certain I've never had one of my students in an un gamified training stay up until morning oh accidentally studying cyber security material but you can absolutely do that when it's a cyber security game that you've created and they just continue consuming that content because it's engaging because it's rewarding yeah so it's all about boosting that production through intrinsic motivation to to actually encourage that state of flow right so the Mastery of skills happens through the exploration of the content you know through somebody's own time outside of the normal training operation right and that can give them other kinds of rewards so again you know we uh you know I run a a research group that studies this kind of gamification and transformational gaming so we do studies we ask people to play our games in real life conditions we ran one study where we said okay we'd like you to play for 20 minutes and give us some feedback and the person actually uh played for five and a half hours and um we talked to them about why and they said well you know it was it was meeting a need that I had at the time I mean they didn't articulate it that way right but they were looking for something to do with their friends with their community and the game happened to sort of um uh fits right it's not like we're we're we tricked this person into playing for five and a half hours it's that the game met their needs for something that they couldn't get any other way and um that's one of the things that makes games so powerful is that it's a way of delivering rewards that are not just about getting paid so um we have another comment that came in from YouTube um from Claudine uh very interesting this could certainly change attitude of students towards assignments and keeping up with reading lists so actually on on that topic Dom could you go back to the hand the last slide on the Hands-On experimentation uh portion um and so this is actually speaking of uh students and getting them motivated and engaged the the last lie down um the uh there we go perfect uh this is actually a cyber security and risk board game that was developed here that um it requires no prior knowledge of networking programming risk assessment I'll be perfectly honest my risk assessment class when I was in grad school was the class I hated the most and I had the most trouble keeping up with and this is uh actually a game that was developed targeting 6th through 8th graders that ended up becoming ending up getting used across uh graduate programs uh trainings for c-suites uh you know uh all over in different areas because it didn't have that uh that prior knowledge requirement but at the same time it allowed you to recreate organizations from all sorts of different sectors be it you know critical infrastructure entertainment banking whatever or a school whatever the case may be and then actually have a mean meaningful discussion about how how do you balance risk how do you invest in your organization and securing it uh in that environment you don't need a computer you don't need any prior knowledge and you're able to actually have have that meaningful discussion there but the key part is what happened after is that the students would take these these cards that have these little snippets on them and they would independently go and research more about what is this technology what is this doing now I guarantee you that a group of sixth graders did not fully comprehend with all the uh nuances of the the difference between a um uh uh uh inventory management program and uh doing audits but having them have hearing them in class the next time having an argument over which one provides more value to their organization is incredibly informative when you're realizing that ninety percent of the argument they're having isn't on those cards they went home and they went and did additional research on on this they invested of themselves because why they wanted to win next time in class yeah and you know and and I said I think the thing I want to amplify is that's an approach that can create a really powerful sort of cyber security pipeline right but it's something that we've seen work with kids all the way up to you know adults lifelong Learners um and so uh this is something that can be used for for example ongoing professional training professional development um thinking about uh helping people meet their professional goals and I see we have a question in Discord right is like how much is too much right uh how much can you use gamification to modify people's behavior in their free time before you start running into ethical issues um that's a a huge question and uh the there's some research on this out of um uh University of Pennsylvania actually that game players know when you're trying to manipulate them and actually as people play games more they get better at knowing when they are sort of being exploited so the way that I would think about it is that if the game is providing a context that is giving value to the player and you are getting value back that that's a pure exchange then you're using games and gamification appropriately when people are only participating for example because their boss tells them to then you're you're you're not going to get the benefit of games right and so you may get the behavior modification but you've kind of left the sphere of gamification and gone more into the sphere of life and that's the key point is and just because they're introduced to it because their boss says hey you must go do this the key part is crafting in an environment that's engaging and meets the need that they have otherwise you're right back to you might as well just give them a lecture with a PowerPoint and give the the the the you know the training we all hate of sitting in a classroom and receiving it passively because they don't want to be there if you've designed your experience correctly you're meeting a need that they have you're providing them value um I do want to because we're running short on time I want to make sure that we get to all the comments we can there is a comment from Claudine on YouTube uh can we generalize this so that educators are able to instantiate with any kind of subject matter absolutely can I can I take that one I think Dom already has slides amazing yeah so um the the good news is that yes most subject matter is something that you can approach with this kind of gamification the bad news is that different kinds of subject matter and different kinds of learning contexts need different kinds of support So wrote uh Dom if you can I'm still seeing the transfer of skills slide if you can pop up to the uh CTP so one of the things that that sort of is my mission here is to is yeah go to the CTP slides is thinking about how we take research and design for a specific kind of problem and approach them with gamification so you can do uh uh next slide uh you can do good research but if you're not actually doing good game design with it right that you may have an impact on people it's sort of what we talked about that the it's it's um uh it's not going to be fun and you're not going to get the benefits of games right and at most you'll have an experience that you call a game that you're forcing people to play on the other hand if you just make something fun that isn't grounded in sort of what we know about research but is just a high quality game then you're probably not going to get the transformational effects but there's another piece which is why we can't really generalize this which is that we need to think about how we're deploying this that when you're putting this when you're having people learn for example I'm working with a group that's looking at physics and history those are two very different kinds of learning both in terms of who learns them uh how when do they get learned what kind of Assessments and outcomes do people expect how easy is it to measure computationally and so if you're not sort of taking this context into account then you end up with what we call deployment failures right that um maybe you have a great game that in theory is is making the kind of learning transformation that you want or the kind of behavior transformation you want but no one is willing to play it because you haven't thought about the way that it deploys in context so uh I actually I was really excited about the Synergy here that uh as of 9 A.M this morning we have launched a center at Carnegie Mellon called the center for transformational play where our job is to work with people who want to um who have a problem they want to address with games and gamification so and can an educator just do this on their own maybe maybe not but if you're an educator and you want to work on this or if you want to be designing uh you know uh cyber security games come talk to us and we're ready working with rotem and dominance I wanted to highlight that we are deeply ACI and the CTP are from their very birth uh very deeply integrated with one another and so if you are a you know in a government role in a DOD role anywhere in that space and you need help and you come to SCI to support you in that we're going to be working with the center for a transformational play we're going to be working together natively in order to ensure that we're bringing the best the most Cutting Edge research the most skilled game designers and CTP isn't just about game design we're bringing in artists from you know the art school we've brought in performers from the school of drama before we've done you know we will reach out anywhere in the sphere where we can get the best talent available to create the most engaging most motivating gameplay experiences that can affect that real transformational effect on your so we have about two minutes left I think in the presentation I figure why don't we jump over to measure and approve teamwork while we wait for any final questions to come in yeah so uh games let us really uh build trust in a rapid uh manner so if you have ad hoc teams that have to come together execute on a mission and then uh break apart you really it's hard to get people to start talking to have that willingness to share ideas to to uh feel that sense of flow that they're engaged in their work and especially for these short-term groupings that are fairly common in a developing uh environment by the time you hit you get that flow going together you're already done and so games can actually enable that we have a good body of research that we probably can't cover in two minutes uh uh about that so I just want to highlight one of the uh examples uh of where we've used this for measurement also for measuring teams and so Dom if you can uh Play that video um the so this is uh we had a quick turnaround a mandate from uh the uh an executive order where um we had to assist DHS in creating an environment we can identify Challenge and reward the US government's best cyber security practitioners and so we're looking at how do we actually get an entire team into an environment we can really challenge the best of the best and figure out who that who who's the most talented the most capable and so we created this immersive client where the teams would pre would uh be confronted with challenges in this case uh winter super storm hits a fictitious City and all sorts of things go wrong there's ransomware's attacks happening while there's power outages and they're trying to coordinate all of these activities fix everything they can and then we use their performance to determine who's the most capable team when a real crisis hits and we are just about a time so maybe we'll just close by saying that you know this is something that we would love to continue these are the kinds of things that we would love to work on with you um thanks so much for your questions all through this uh panel it's been really great to uh have this be a conversation and uh I hope that this can be to be continued thanks everybody I would first like to say an echo what uh Jessica just said and I really thank our audience for kind of interacting and and providing the questions second thanks to Jessica Dom and rotem for having this conversation it was great um it was nice to see how kind of gamification can really change the engagement and and really kind of motivate um you know kind of DOD and and honestly everyone to kind of really stretch and learn Beyond just kind of clicking through a pre-canned PowerPoint or you know kind of a you know HTTP you know web page animation um so thank you everyone for that we're now going to head into a 15-minute break and when we return we're going to come back to uh chain games powering autonomous threat hunting and I will say it's a completely different type of games we're talking about Game Theory at that point but uh stay tuned it's going to be uh really interesting and uh hopefully we'll see you back here in 15 minutes and keep the questions coming thanks everybody [Music] foreign [Music] thank you [Music] thank you [Music] hi I'm Phil gross a senior Network defense Analyst at the SEI I'm the principal investigator of the sei's chain games project for powering autonomous threat hunting threat hunting is a technique used by cyber security operations staff to find sophisticated attackers who have already evaded the Network's initial defenses and are either already committing malicious Acts or are preparing to threat hunting however is almost always done manually because human judgment is required to decide where how and how long to hunt so it proceeds at human scale while the network attack surface is already large and growing much faster despite threat Hunter's best efforts there are enormous volumes of data about numerous systems they never look at and that's where attackers can hide if we can find a way to conduct some level of threat hunting without requiring human investigators we can cover much more of the network forcing attackers to work harder to stay in our Networks so we've been researching ways to develop and evaluate autonomous threat hunting strategies using Game Theory a mathematical tool for modeling decision making among agents with different and often conflicting goals in my presentation I describe our team's work on chain games a set of games that build on previous work in cyber deception and camouflage our chain games develop the concept of the attack graph or kill chain a kill chain is already familiar to Security Professionals but we treat it as an abstract space where attackers and Defenders interact I'll talk about how this process can not only produce automatable threat hunting strategies but also make it possible to analyze and evaluate those strategies in an automated way reproducing at a basic level the Judgment process of the human threat Hunter if you're interested in learning more about these techniques and their potential watch my research review 2022 presentation or reach out at info sei.cmu.edu thank you [Music] foreign [Music] [Music] [Music] [Music] foreign the effectiveness of this assurance by implementing a model analyze build practice based on MBE to catch problems early and throughout the life cycle of the system however to be successful MBA approaches must use complex analysis that verify specific aspects of the system Behavior but shielding the designer from the complexities of the analysis examples of such Behavior include control stability timing safety and cyber security unfortunately the analysis make complex assumptions about the design of the system if such assumptions are not met the results of the analysis are invalid moreover assumptions from different analysis may even be incompatible with one another compounding the complexity even further in this project we develop a Nobel approach to handle a collection of complex analysis applied to CPS MBE models and their implementations in our approach we first verify their analysis assumptions and a seasonal designer in the resolution of problems and secondly integrate their verification results in an automatic formal argumentation structure that proves how these results guarantee Assurance claims that can be ultimately connected to the authority to operate [Music] foreign [Music] foreign [Music] [Music] foreign [Music] [Music] [Music] foreign [Music] [Music] [Music] foreign [Music] [Music] [Music] hello I'm James Ivers software provides the core capabilities of most systems that we build today stay competitive we have to evolve software quickly to face new challenges and use new technologies but too often the way our software structure prevents us from quickly changing it to meet our evolving needs large-scale refactoring is a way to correct significant structural challenges but it lacks dedicated tool support and takes months to years of development effort consequently many organizations forego needed refactoring and accept the limitations of existing software the SEI has created an automated refactoring assistant to dramatically reduce the time and cost of large-scale refactoring it recommends refactorings that modularize software to solve key challenges and activities like breaking software monoliths into independent services or extracting strategic capabilities for reuse our research uses multi-objective genetic algorithms to search for solutions to these problems and generates actionable recommendations for software written in Java or c-sharp check out our talk on refactoring for software isolation to learn more about how the refactoring assistant works and how it can work for you [Music] [Music] [Music] foreign [Music] foreign [Music] [Music] foreign [Music] [Music] foreign [Music] foreign [Music] welcome back everyone hopefully um welcome you back from having seen the transformation transformative gamification panel and now it's my pleasure to introduce Phil gross who's going to talk to us about chain games powering autonomous threat hunting uh Phil came to the Cirque division here at SEI in 2005. after working at several startups developing Network sensing and incident response tools at the sci Phelps worked on a number of different uh problems pretty much related to cerebral cyber security and situational awareness concerned with identifying and understanding security relevant information in the large volumes of data that computer networks generate which is especially problematic in government Networks in addition to his work on threat hunting Phil has designed data visualization tools and advised on building data pipelines for collection and Analysis of sensor data on large Networks and with that let's pivot over to Phil's talk thanks everybody hi I'm Phil gross I'm a senior Network defense analyst in the monitoring and response directorate of the cert division of the software engineering Institute and I'm going to talk to you a little bit about some research I've been doing into powering autonomous threat hunting using um something I call chain games this project that I'm discussing today grew out of conversations with people whose staff cyber security operations centers or socks as everyone does more online with information systems that generate more data than ever before and of course attracting more and more sophisticated attackers it's no surprise that the people who staff security operations centers are very busy as a researcher one of my major interests is making automation possible in cyber security where it could make a big difference and one of those areas in cyber is threat hunting the SEI is currently in the early stages of research into autonomous threat hunting modeled as games so that the costs and benefits of decisions can be Quantified simulated and evaluated hopefully sometime in the later stages of This research in a fully automated way we're not there yet but we've defined a type of game that we think forms a solid foundation for this later research I'm going to show you how that game works then discuss what we plan to do in the coming year to relate that game to real networks and threat hunting scenarios moving toward that goal of full autonomy the first what is cyber threat hunting why is it important and why is it important to automate well cyber threat hunting is not new Clifford stahl's book The Cuckoo's egg was about a threat hunt he conducted in 1986. however threat hunting as a formal practice in security operations centers is a relatively recent development as organizations start to appreciate how threat hunting complements to other common security activities intrusion detection and incident response intrusion detection tries to keep attackers from getting into the network and initiating an attack meanwhile incident response is the practice of mitigating damage usually done by an attacker when the attack is culminated threat hunting addresses the Gap in the attack life cycle when an attacker has evaded initial detection and is planning or in the initial stages of execution of their plan these attackers are in a position to do significant damage but that risk hasn't been fully realized yet threat hunting gives the defender another opportunity to find and neutralize attacks before that risk of materialize the main drawback to threat hunting is how much time and expertise is required to conduct a threat hunt individual hunts can take days or weeks to run meaning hunt staff have to make tough decisions about which data sets and systems to investigate and which to ignore every data set they don't cover is one that could contain evidence of compromise a faster hunt can cover more data and introduce at least the possibility that the evidence if it exists gets noticed and brought to the defender's attention at sufficient scale hunt campaigns could be coordinated to cover more data or to more intensively cover interesting data sets finally these faster hunts could serve as a scouting or reconnaissance process giving human threat Hunters information they can use to better direct their attention to achieve the speed and economy of scale necessary to begin thinking about hunt in this way requires Automation in fact we believe it requires autonomy the ability for automated processes to predicate conduct and conclude a threat hunt without any human intervention so that's the vision a fully autonomous threat hunting capability that can investigate cyber security data at a scale approaching the one at which it's created but we're a long way off from that goal to start down this path we needed to be able to model the problem in an abstract way that we and someday in the future an automated hunt system could analyze we wanted to build an abstract framework in which we could rapidly prototype and test threat hunting strategies possibly even programmatically using tools like machine learning and we felt that a successful approach would reflect that threat hunting concerns agents who wish to hide in a network and other agents who oppose them want to find them and evict them this led us naturally to Game Theory as a foundation for the work so we familiarized ourselves with some of the recent working Game Theory hoping we could find researchers already working in cyber security ideally doing work that we could immediately adapt to our purposes it turned out there was recent work we felt we could build on in this area which dealt with one of the fundamental aspects of adversary Behavior the use of deception somewhat to our surprise this body of work focuses on how Defenders could use deception and not attackers in 2018 for example a category of games was developed called cyber deception games these sought to investigate the effectiveness of deception in frustrating attacker reconnaissance I'll point out that this work was contextualized in terms of the Cyber kill chain about which more later and cyber deception games were zero-sum games meaning that the utility of the attacker and the defender exactly balance out in 2019 some additional work was published on Cyber camouflage games which were conceived of in similar ways but were General some meaning attacker and Defender utility aren't directly related and can vary independently the goal of this work was still defensive deception in this case in order to decide when to perform expensive deception tasks like OS fingerprinting more efficiently without losing effectiveness it was really helpful to find this work and seeing Game Theory applied to real cyber security problems gave us confidence that we could apply it to threat hunting deception was being analyzed as a defensive tool but at least initially we thought we might be able to take advantage of the analysis done in this work but invert the logic in the end we decided a different approach was needed for our work so while the work on Cyber deception games mainly used references to the Cyber kill chain to contextualize the work it struck us as a powerful formalism that we could Orient our model around and now I'd like to describe to you what that model is and walk you through some simple examples before talking about what we're doing now and what we plan to do in the near future our approach to modeling cyber threat hunting is a family of games we're currently calling chain games because they're oriented around this very abstract model of kill chains that we just saw we call this a state chain each state in a chain represents a position of advantage in a network or in a computer or in a cloud application or a number of other different contexts in an Enterprise information system infrastructure you can think of an attacker initially establishing themselves in one state say state zero perhaps someone clicked on a malicious link or an email attachment the attacker's first order of business is to establish persistence on the machine they've infected so they don't get accidentally evicted so they write a file to a disk and make sure it's executed when the machine starts up by doing that they've moved from initial infection to persistence and they're advancing into State One each additional step an attacker takes to further their goals advances them into another state now it isn't free for an attacker to take these actions for instance if they're not a privileged user maybe they can't set that file to execute and trying will get them noticed by an endpoint security solution so they'll need to try and Elevate their Privileges and become an admin user but that's also suspicious both activities entail some risk but they have a reward to model this there's a cost at any time an attacker wants to advance down the chain but there's also a benefit or a payout that they achieve when they successfully move to a given state the defender is not traveling along the chain like the attacker is the defender is somewhere in the network able to observe some of this and sometimes able to stop the attacker from advancing in the next few slides we're going to talk about different rules we can make about how the attacker advances through the chain and how the defender may try to stop them fundamentally all the games we're talking about here follow these rules their two-player games played between an attacker and a Defender the players and their actions will be color coded to help you keep track the games are all played currently over a fixed number of turns usually two or three in these examples chain games are mostly General some games so each player gains and loses utility independently there are a few exceptions to this which we'll see finally we conceive of these games as simultaneous turn games that just means that both players decide what to do at the same time and those actions are resolved simultaneously before we leave this slide I want to call two things to your attention the first is that the middle chain here isn't really a chain it's a graph from the attacker's standpoint this could represent a choice they make about exactly how to attack or exploit or otherwise operate within the network once that choice is made we can think of the path the attacker chose as a chain so even though the analysis is oriented around chains there are ways we can treat more complex graphs to think of them like chains the second thing I want to point out is that the payoff to enter a state which you can see here on the edges of the second and third picture here doesn't have to be the same for each state we'll be using uniform value chains for the first few examples but there's actually a lot more expressiveness in this cost assignment for instance in the chain at the bottom S3 may represent a valuable source of information but to access that the attacker may have to take on some net risk we'll talk more about this in our future work the first game here is a very simple game which we can call version zero in version zero the attacker and Defender both have two actions each the attacker can advance meaning they can go from whatever state they're in to the next one collecting the utility for entering the state and paying the cost to advance in this case the utility for each Advance is one which is fully offset by the cost which is also one however the defender receives negative one utility whenever the attacker advances so there's a zero-sum component this isn't meant to incentivize the attacker to advance so much as it is to motivate the defender to exercise their detect action a detect will stop and Advance meaning the attacker pays for the cost of the advance but doesn't change States and doesn't get any additional utility but it does cost the defender one utility to detect so receiving a penalty when the attacker advances gives the defender a reason to pay the cost for their detect action and avoid being punished for an attacker advance finally both the attacker and Defender can choose to wait waiting costs nothing and earns nothing the table on the right is called the payoff Matrix of the game it shows what the total net utility for each player is when they play the game for a set number of turns in this case two turns each row represents the defender choosing a single sequence of actions in the first row it shows what happens when the defender waits for two turns across all the other different sequences of actions the attacker can take each cell is a pair of numbers that shows how well that works out for the defender which is the left number and the attacker on the right this Matrix shows every strategy that the attacker or the defender can employ in this game over two turns technically it shows every pure strategy you'll see a mixed strategy in a few slides but for now you can just think of these as all the possible strategies so with that information we can do other kinds of analysis like identifying dominant strategies and in this case it turns out there's one dominant strategy each for the attacker and the defender the attacker's dominant strategy is to always try to advance that's understandable the defender's dominant strategy is never detect in other words always wait intuitively it seems that the negative one utility penalty that we're giving for an attacker to advance isn't enough to make it worthwhile for the defender to pay the cost to detect so think of this version of the game as a teaching tool a big part of making this approach work lies in choosing good values for these costs and payouts in this version We introduce some mechanics that let us think about when to deploy and detect attacker camouflage you'll recall that the previous work on Cyber camouflage games and cyber deception games modeled deception as defensive activities but here it's a property of the attacker this game is identical to version zero with the exception that each player's primary action has been split in two instead of a single Advance action the attacker has a noisy Advance action and a camouflaged Advanced action this reflects Tendencies we see in actual cyber attacks some attackers try to remove evidence of their activity or choose methods that may be less reliable but also harder to detect and others go boldly forward in this game that Dynamic is represented by making a camouflage Advance cost more than a noisy advance but it's harder to detect on the defender side the detect action is now a weak detect and a strong detect a weak detect can only stop noisy advances a strong detect can stop both types of attacker advances but of course it costs more in the payout Matrix we can strong detects are referred to as low and high detections this by the way is the full payout Matrix I don't expect you to be able to read it but I wanted to show it to give a sense of how quickly simple changes like additional actions can complicate analysis here's a detail of that previous payout Matrix showing all the defender strategies and three of the attacker strategies it also shows that despite the addition of a camouflaged action the game still produces one dominant strategy each for both the attacker and the defender now however we've tuned the game so that the attacker should never Advance I'd love it if we could find a way to do that in real life but unfortunately like the previous game this is an artifact of the way we've chosen to structure the costs and payouts incidentally for those who may be unfamiliar with the term a dominant strategy isn't one that always wins in Game Theory a strategy is dominant if it performs the best that you can expect to do more or less against a perfectly rational opponent so while we're getting these particular strategies because of the way the game is tuned we may find that attackers in real life play strategies other than the optimal rational strategy for some reason and if they do we may want to adjust our Behavior to optimize for that situation the most helpful tool for exploring situations in which one of the players is deviating from optimal play is simulation we'll talk more about our current and future simulation work shortly the last two games were played on chains with uniform advancement costs when we vary that assumption we start to get much more interesting results as I said earlier a three-state chain like this is a very reasonable characterization of certain types of attack an attacker gets a lot of utility out of an initial infection sees a lot of value in taking a particular action on objectives but getting into position to take that action May incur little no or even negative utility introducing chains with complex utilities yields much more complex strategies for both attackers and Defenders here we see the output of Gambit a game analysis tool describing the dominant strategies for a game played over the chain shown below the dominant strategies are now mixed strategies a mixed strategy means there's no right strategy for any single playthrough you can only Define optimal play in terms of probabilities for instance the attacker here should always Advance one turn and wait the other two turns however they should mix up when they make their Advance spreading them out equally among all three turns this payout structure May reflect for instance the implementation of a mitigation of some sort in front of a valuable asset the attacker is deterred from attacking the asset by the mitigation but they're also getting some utility for making that first Advance if that utility were smaller perhaps because the utility of compromising another part of the network were mitigated perhaps it would be rational for the attacker to either try and Advance all the way down the chain or never try to advance at all clearly more work is needed here to better understand what's going on but we're encouraged by seeing this more complex Behavior emerge from such a simple change I'd like to turn now from the chain games that we've defined and toward our near and medium term work most of this work concerns taking our abstractions and mapping them into more realistic environments the first thing we can do is change the game itself to look more like an actual threat hunt threat hunting usually happens as a set of data queries looking for data that shows evidence of compromise we can reflect this in our game by introducing an information Vector the information Vector changes when the attacker advances but not all the information in the vector is automatically available to the defender so some advances may be invisible to them for instance as the attacker advances from State 0 to State 1 here there's no change in the information the defender has access to however advancing from State 1 to State 2 changes some of the defender visible data enabling them to detect attacker activity the addition of the information Vector permits a number of interesting enhancements to our simple game deception can be modeled as multiple Advanced actions that differ in the parts of the information Vector that they modify similarly The Defenders detect actions can collect evidence from different parts of the vector or perhaps unlock parts of the vector to which the defender normally has no access this Behavior May reflect applying enhanced logging to processes or systems where compromise may be suspected for instance finally we can further Defenders actions by introducing actions to remediate attacker presence for example by suggesting a host be reinstalled or by ordering configuration changes to a resource that makes it difficult for the attacker to advance into another Technique we plan to do more work with in the next year is simulation as you've seen small complications can result in many more options for player Behavior which means a larger space in which to conduct analysis simulation can give us approximate results to questions that are computationally and feasible to answer exhaustively another benefit to simulation as we've discussed earlier is that we can model situations where theoretical assumptions are violated and see if some theoretically sub-optimal strategies have better performance in specific conditions the screenshot to the right is the definition of version zero of our game in openspeed a simulation framework from deepmind we've been getting familiar with it this year and plan to use it for more active experimentation in the coming year finally as the last game example showed we can use different Advanced costs on state chains to better reflect both patterns of protection in the network and patterns of attacker Behavior depending on how we choose to interpret the relationship of the state chain to the attacking player more complexity here results in a much richer set of strategies than the uniform value chains do there are other ways we can map Primitives in our games to more aspects of the real world threat hunting problem we can use simulation to model empirically observed strategies and we can map features in the information Vector to information elements present in Real World Systems that's the heart of the work we plan to do in the coming year I hope you found this talk entertaining and educational I find this work very exciting I hope you do too and I look forward to any questions you may have thanks for your time hi everyone and Phil thanks for that great talk we're now live here with Phil gross and looking to kind of continue this conversation uh hopefully you guys can submit some interesting and challenging questions for Phil on either the Discord or YouTube I'm going to kick it off with uh so Phil a lot of cyber attacks happened in seconds or even minutes would you would the approach you're talking uh to autonomous threat hunt be able to help with these kinds of attacks yeah I I think that speaks to um something that uh obviously I spent a lot of time in the work uh discussing where we are now which is in a very early stage uh and talking about the game itself but I think it's pretty fair to want to think a little more about what the applications are and where this tool can be applied and where it can't um and there are attacks sort of that occur on on lots of different sort of time scales right these these um uh very short attacks are probably uh some of the most common right so that you know automated systems someone clicks on a link and immediately stuff starts happening and people want to get uh from that initial infection to uh some sort of effect on the adversary as fast as they can and I think it's fair to to wonder threat hunting uh and an autonomous way could could come in and um and maybe close that that Loop uh and get inside sort of uh so to speak the ooda loop of the automated attacker by by responding with automation I am not so certain that this that this work can apply in that way uh although I'm not going to entirely rule it out because I think that once you start on a once you start automating uh a threat hunting process I'm not going to say the threat hunting process because I don't think you ever displace the human threat Hunter um but if you start applying a uh an autonomous threat hunting process ubiquitously um in a long-term Vision it might be possible to get down into those time scales but I don't think you're going to start there um but fortunately I think I think there's a very interesting uh space to work in with the attackers that function along those longer time scales because uh frequently the reason that uh these sorts of things are uh taking longer or or uh that the time scale is extended as much as it is is because there's essentially a multiple phase uh operation that's going on where the adversary will gain an initial level of persistence and then maybe um uh conduct multiple operations over time from uh one position uh or maybe uh do surveillance and try and identify a time which they can they can operate that is going to achieve maximum effect um and these sorts of adversaries are the most in a lot of ways the most important adversaries and one of the most difficult adversaries to uh to identify and to to sort of of mitigating defend against they're generally very targeted this is this is the kind of this is the kind of activity and behavior you get into when you have a highly resourced attacker when you have uh maybe a state actor or something like that so obviously something that is uh of critical importance to uh to the dod into other very high uh criticality sort of uh sort of Enterprises um and the sophistication that they need in order to get and stay persistent um is uh you know speaks to uh it's an investment it it from a game theoretic standpoint they're putting a lot of uh there's a lot of cost to that action so they're going to want to receive a very high payout for that reason I think it makes a lot of sense even if we're not operating on that really low quick time scale it makes a lot of sense to focus on these folks who are maybe doing their initial infection activity in a short time scale but now they're going to be surveilling the environment there might be some actual humans on keyboards so now we've entered sort of a human time scale um and uh and then eventually a long possible period of of quiescent persistence uh which indicates that they are trying to be judicious and careful uh about and careful for them being very not careful for us meaning that there might be a significant effect on us but we can use that to our advantage uh by aggressively uh scanning our own networks uh with with some autonomous threat hunting processes and uh trying to identify those signs of compromise and hopefully in in terms that that I'm using in in my work hopefully find them uh in the middle of the chain uh or maybe even earlier in the chain or at critical points in the chain as we start identifying uh uh and and you know mapping these chains down into actual attack patterns uh where um where they might not have already done any damage at all um they might have achieved persistence and been waiting for a moment and now we can interrupt their uh their attack activity because they're not expecting to be caught so that's the idea I hope that answers the question it's actually a great uh answer that leads into a bunch of other questions uh right so there's an interplay between kind of your work and how to potentially cause obfuscation or you know uh signaling that forces the attacker to do new activities which hopefully we have some other forms of detection on um so do you envision that there's some you know to the extent that you're successful in that regard maybe you know our other forms of Defense uh pick them up and and maybe they we find them before the real is that part of the goal then I absolutely want to one of the reasons that that we were drawn to Game Theory as an approach is because we want to be able to evaluate the utility of the adversaries choices as well as our own and be able to make them uh force them into double-blind situa or double bind situations right um so absolutely we want to force them into behaviors that are going to be hopefully easier to detect or we're going to force them to operate under uncertainty whereas the defender frequently has to operate under uncertainty as to what sort of attack is going to be made against them we want to sort of um we want to make it uh difficult for them even if they can attack a defender's activity to sort of understand the motivation of that activity so that they have to think if if now we have ubiquitous level of maybe not incredibly sophisticated scanning as I say in the presentation we're not trying to replicate uh the level of sophistication that you're going to see in a human threat hunt but even having a level of really sort of rudimentary algorithmic scanning that's going on at a at a very a much more intensive level than you have now will force their behavior to account for that and if we can make that uh incorporate additional aspects of uncertainty uh for instance by using the same activity in the service of multiple different operations such that um they may be forced to uh from an abstract standpoint maybe maybe the attacker has a choice of whether to cover up the evidence of their attack over here um or or leave it and there are reasons obviously to cover up the attack right that's at a high level we would call that a stealthy maneuver but it might also be a stealthy maneuver to leave it for instance if that data is going to be if they know that data is going to be observed before they can modify it and then copy it off onto a system somewhere um then perhaps a uh an investigator process has access to that information so maybe a subsequent version of this uh of this game to model that might have additional bits in the information Vector that describe the same piece of information observed from different vantages to see if those two different advantage is align with each other now when um when I as an investigator perhaps I'm making a request of an individual machine on which an attacker is resident for a piece of information and that piece of information is something that would have been modified uh as the attacker Advanced through their attack chain um if they have modified that but the original information was replicated off to another source I could see that discrepancy and I could know there's something wrong but if I if they leave it the same and I'm and I'm proceeding from a different set of assumptions where I'm looking for uh pieces of information that look a certain way right because I think that's evidence of attack sort of a signature based approach right then um then they're wrong then too so I think that's I think that's where I want to be uh uh in the long term and where we want to be in terms of of getting to the point where we have a a game theoretic framework for uh for conducting threat Hunt is being able to soon uh the parameters under which we operate such that we can look at the uh the different um possible interpretations of our actions uh from an adversary perspective and get it to the point where the utility that they have of taking one action versus another is uh is is no better one way or another and and ideally fairly small but uh but if we do that then we give them almost no choice right so yes I absolutely uh hope that we can can get into uh get to that point and this is the first year of two right uh just to confirm yeah okay that's right well thank you so much Phil um that's been great let me uh transition now we have our next talk coming up which is uh maturing Assurance contracts in model-based engineering it's going to be presented by Dr Denise he's a principal researcher and a technical director of the assuring cyber physical systems directorate at the software engineering Institute um his expertise and focus of his research includes cyber physical systems real-time systems and model based engineering and the security of cyber physical systems in the real-time Arena he has recently focused on multi-core processors and mixed criticality scheduling he's also co-authored and co-edited the book cyber physical systems where the authors discuss different application areas of cyber physical systems and the different foundational domains including real-time scheduling logical verification and security and with that thanks [Music] hello I'm deal Denise I'm the principal investigator of the mat during and shooting contracts in mother-based engineering project today I'm going to be talking to you about our first Year's results so our project aims added the challenge of the digital engineering that is basically called a model analyze build in the idea here is to be able to create model first analyze them for properties of interest and once we are satisfied with them build the system and the reason for this is that the late discovery of design errors in DOD are very costly for instance when we make introduced errors in early stages and we discovered them very early let's say at the Fielding stage then the calls can raise up to 300X or more so then for these we use models it is to use these models to be able to analyze them and we use this analysis as a black box because there are complex analysis but the challenges that we have with this analysis is that they make a lot of implicit assumptions and with these assumptions are not made then the results actually are useless are invalid right so if these assumptions are kind of not met we also have the problem that this complexity of the analysis kind of show up now the sudden needs to get into the details of the analysis try to find out what went wrong and this is actually a barrier for an adoption and it kind of defeats the whole purpose of this black box analysis so what can we do with this so not only this is one of the problems but also we have multiple claims and for this multiple claims we also have multiple analysis for instance we need to verify things about model checking control stability thermal dissipation schedulability and the like and all of these have multiple analysis that need to be a kind of combined together but also their assumptions need to be combined together we need to verify that they are not conflicting with each other and that they are properly satisfied so what we have done over the years is create a concept called analysis contract and the analysis contract is a module that defines what is the analysis that we are trying to execute what are the assumptions that we need to make for for this analysis to be correct and what is the guarantee that we we achieve with this right what is the property we verify but these assumptions actually can be validated at different stages in the development so for instance some of them we can make a very early determinations we can very early define whether we meet the assumptions and some of them we need to defer them until we have more data and even we can get to the case where we need to verify that these assumptions are made at the code level once we have an implementation and we need to enable all these with this analysis contracts in the end what we want to be able to do is to shift to the left and down to the implementation what it means is we want early analysis we want to enable this early analysis to evaluate these earliest and decisions even with partial information but he says we can evaluate whether we have we can meet a latency analysis and end-to-end literacy analysis even before before we have periods but at the same time we want to enable this refinement so as we add more information we need to keep track of the pending information we need to find out once we have enough information whether we can execute new verification procedures like schedulability and in the end as we have said before we want to go all the way down to the implementation in terms of conformance and for that we will need to test different verification techniques to really make sure that whatever we prove in the model is also present in the implementation so this analysis actually are used and these contracts are used to create what we call an assurance contract argumentation right so in the end a particular claim that we have at a high level uh can be supported by multiple analysis in this analysis the combination of this analysis will actually discharge the claim but the complexity of the assumptions of the analysis sometimes can be can be high and therefore it will require for instance a full analysis to be able to do Charter particular assumption but at the same time we can also have a very simple verification uh on data that we already have present in the model or we need to defer the the verification because we don't have enough data but we have enough information to say there is no contradiction in the data that we already have and this is what we call a proof of negation right so in particular if we go after just an example to say imagine that we have a reliable auto brake that is able to sense with the radar and see whether there is a need to break we obviously will need to have a radar to break a deadline we need to break before we crash obviously but we also need to have some fault resiliency some cyber security among other things that we need to add to this particular claim and when we focus on the meter radar to break that line then we do have multiple verifications such as whether the threats are periodic and we may not have enough information to prove that right away so we can defer that as a proof application even verifying if there are any contradictions at the moment whether the connections have been marked to be delayed but we can verify directly in the data that we already have order the threads are scheduleable and for this we will have multiple options to test whether the the threads are scheduleable in the particular contract and Rise that we will verify we'll create its own behavioral model of the scheduling to find out whether we can really discharge the claim that the threads are scheduled in the end when we talk about this contract argumentation the development life cycle the Integrity of the analysis based on the verification of assumptions we need to provide support to the designer and this support is about verifying these assumptions in terms of first detecting whether there are violations or not but secondly and more importantly suggest repairs and if there are ways to repair the violations we will suggest that that's part of what we want to do in the infrastructure but even sometimes we cannot really repair an assumption so in that case we can offer alternatives for different analyzes that do not need to solidify particular assumptions right at the same time when we refine the design and we need to verify where new data will enable new analysis we really need to make sure that this new analysis will be properly discharged and go after the the proof obligation that we also have right in one of the things that are important to uh for us in this project is this argument reusability so these analysis contexts allows us to have a self-contained module that we can reuse whenever we have different systems and we create different argumentations for planes we can connect the modules to other analysis and create new argumentations so if we talk about this Integrity of analysis how do we repair assumptions to prove a simple example in this particular case the harmonic schedule ability bound analysis we will have a three particular assumptions rate monotonic priorities harmonic periods and periodical to deadline and we can start testing the or verifying these assumptions based on the data that we have in this case the data that we have are periods so we need to find out in this particular case where the periods are harmonic what it means is the periods are multiple of each other and we can already see that period two is not a multiple of period one so we can suggest to say we can make it harmonic if you want if you can accept a changing Period 2 for instance from 15 to 20 then we can verify this this assumption and we can use this analysis but of course that may not be the case and if that is not the case we need to provide alternatives so we will able to create or kind of search for an alternative contract that will not have that requirement of harmonic periods that then the designer can use similarly when we go through refinement right the uh presence or absence of data in this particular case the only data that we have is utilizations it will allow us to prove or disprove certain conditions certain assumptions that we have in this particular case if we don't have enough data we just create this proof obligation as we have said in the past for instance that the utilization should be equal to the execution time over over the period but we don't have periods or execution time so we mark this of proof of obligation similarly with the example that we discussed before with harmonic periods is yet another propagation and finally period equal to the airlines we don't have deadlines or periods so therefore it becomes another proof obligation but as we gather more data for instance when we get periods now we are able to discharge the harmonic periods and once we charge that we can continue reusing this this analysis and as we get more data for instance deadlines now I can verify where the periods are equal to the deadlines and similarly when we get the execution times now we can test whether the utilization that we use in the past are equal to the execution time order periods so this is the way we keep discharging testing the different assumptions as new data becomes available in the refinement process obviously as we mentioned before the concept of the contracts allows us to have different modularities right so these different modules I will say that we can combine but we can also kind of uh preserve parts of this argumentation parts of these combinations and we use the combinations by themselves we can do this incrementally as we were saying before as we new data becomes available we can increase our arguments we can increase I would say the the kind of precision in the claims that we want to do we could have different Alternatives depending on the data that we have depending on the assumptions that we can verify uh different ways to discharge claims and assumptions in the end we implemented this in a what we call a symbolic contact argumentation we take advantage of the advances that are occurred in informal verification in particular with constrained solvers constraints or destruction solvers we use the C3 consistent function solver for for smt to encode the assumptions the guarantees and to be able to combine this argumentation improve this argumentation or discharge the argumentation that we built for a particular system this is implemented in an annexed language that is a sub language hosted in ADL and it's so sorry tool but we have a model query language that is adaptable for multiple languages even though it works initially in ADL is also adaptable for this system lb2 or any model that will have enough semantics to be able to reason about these constraints and and in context right and of course this contract verification plan that is how do we discharge the different claims is part of the infrastructure that we are building we of course is very critical for us to have this contact argumentation scalability so we take advantage of the advances in different scientific domains we exploit the efficiency of the algorithms that these domains have created sometimes they create greedy algorithms that are very fast for instance in real-time theory that can be implemented in imperative language we enable that we assume that these algorithms are correct in particular they have been proven to be correct in multiple papers but there are also some efforts to kind of formalize this correctness in some other Theory improvers like in particular for real-time Theory and we also for instance assume initially the correctness of the implementation leaving these verification these conformant tests for later on as part of efforts are for code generation or deferred called verification we expect to have a strong impact in certification because in the end we are creating this infrastructure for an automatic and sound verification of assurance claims through models which of course will impact the Fielding speed if this is adopted that will actually allows us to do an incremental validation uh through the design refinement that we have in of course we want to impact the community of the digital engineering and ideal ecosystem to be able to support the dod Air Force in modeling and Analysis the architectural Central integration practice that we have within the fbl projects DARPA programs using ADL and the like in the end we are creating basically a certification for the digital engineering era to support the model analyze built with automatic argumentation we are strongly shifting to the left starting very early to the to verify the design decisions but supporting all these designs through refinement all the way down to the implementation and drive all these properties and assumptions down to this implementation in a scalable way that exploits the efficiency of analysis in different domains but preserving soundness meaning that we exploit the advancing formal verification to do this argumentation in a formal way that can actually preserve the correctness all the way down this is our team we are very happy to have a very strong team to support this effort and we will be very happy to answer any questions uh either in the in the question session or by email in using this email thank you very much hey thanks for that great talk to you and uh right here we have Dr Denise here with us um and I you know I want to kind of get right into it but first I really want to ask audience members if there's anything you want to know about any of these topics modeling real-time systems uh kind of the formal analysis please post them into the YouTube questions or the Discord chat we'd love to kind of hear from you um you know pick Dio's brain ask them anything about these topics uh truly he's he's a wealth of knowledge on this stuff so um would be happy to kind of hear from you and uh you know kind of go there I have a handful of questions here uh Dio um so kind of first off uh how do you know that you've captured kind of all the data you need to check the assumptions uh to kind of validate these models yes let's say a great question actually uh so as you know I would I would put all she's integrating this multiple analysis in this model and this integration basically also captures the data that is needed and as we mentioned in the talk the data is captured in kind of two two Styles if you will one is the what is needed to run the model and what is needed to verify the assumptions so the Assumption modification can be deferred we say as soon as we do not have contradictions then we can Define but then of course at some point in time we need to make sure that those assumptions are verified and that data needs to be present to validate the assumptions so we are also using the the advances in constraint solving to really find out if we have specified of the data that is needed it means that if we are not uh if it's possible to find an assignment of values that will violate the constraints and therefore it means that we are still missing data so that is automatically checked by the approach that we have and so you're talking here kind of about um you know the model helps you kind of understand how much data and what to capture um and and you capture that holistically but can you verify the different analysis and the assumptions independently from each other or you know does that you kind of having that data together does it complicate matters yeah that that's actually a very a very interesting observation because a typically uh the different scientific disciplines different academic communities if you will develop their own uh techniques to do qualification in those techniques are based on their own abstractions and as attractions they erase data but the data that is that he raised in one particular Community for instance to purify timing they will erase for instance value transformations but if you want to and that allows you to do a fast verification of whether you will meet at the timing requirements however if you are trying to validate a value transformation like in logical verification then of course that is missing data right so then they in logical verification they would use this value transformation to really figure out whether the transformation is correct or not but they will erase timing right so then at some point in time we need to create this interface between the two abstractions and then verify them together to say what if a certain bug confirmation is actually produced but at the wrong time so how do you kind of put them two together and that's actually an interdomain bridge that we need to create and we that's kind of one of the objectives of our project to go after that that bridging if you will oh interesting I I I'm gonna pick on a little you know kind of one of your comments from from a moment ago one of your uh answers which was you talk about you know kind of satisfiability analysis and typically that is not kind of a fast process right it's a you know kind of a big search problem um and and historically we kind of know this right a lot of form of models that take a really long time to execute so help me understand you know how do we scale this how do we um do it you know uh you know your shortcuts tend to introduce errors what are we doing about this yeah that's a good point in what what we are trying to go after here there are kind of probably two two uh line of thoughts here one is where we can develop a single unifying model that will put all the domains as we talk about for instance value transformation timing all of them together and create a humongous model that will allow you to verify all the properties together but the problem with that is that of course if it's tough to go after these uh using these techniques for a single domain putting putting them all together is his grace and nightmare from this kind of point of view so what we are doing is uh actually a way to say we will reuse uh analysis uh algorithms that the specialized domains timing logic control have developed but then we develop only the integration language and that is where we use these formal techniques to say during the integration language where the description of the integration then is formal so that we can really verify that they will work together we will meet all the assumptions but only at the integration level and not going beyond that because otherwise it will not scale that's that's great so I'm feeling there's a you know almost a little bit of a trend here right so um I mean we talk about a couple of things about you know kind of scalability um you know the tools to kind of help understand what data a lot of this sounds kind of usability and you know there's a you know kind of the barrier entry historically on using formal methods so can you comment a little bit it feels like you know we're trying to head in that make make all of this a little more accessible a little more usable for users um can you kind of tell me um you comment on that where are we headed in this yeah that's a that's actually a very good point there are plenty of problems to tackle let me just start with that right but uh in general I will say what we are going after is uh we are trying to exploit what model based software engineering and system engineering is offering that is to say high level models are the architectural level that would use analysis it's a plugin to analyze particular properties you can say timing security safety all of that typically like they tend to be used and can be used as a block box so it's a black box where you have a description that is simple enough that you can run the analysis but the problem is that that black box actually is uh tricky because if you make the model that doesn't match the assumptions of the of the algorithm the analysis then you will run into problems right so it's like sure let's take advantage of those analysis there is some usability already in place but we need to tackle the The Next Step that is okay if I get an error because something was not matching then how do I correct it in the whole machine behind is what we are creating then just perhaps to connect it to another thing is sometimes with these complex models we're trying to bring the best explanations of what went wrong in doing at a bulk level but if we get deep enough to any particular analysis say value transformation to say you make an assignment that will lead us to an incorrect value then getting to that explanation will be tricky so there is a lot of explain explainability research to be done if you will for those deep analysis uh that need to be tackled and so I mean that's a laudable goal to to be able to bring these things uh you know kind of to a wider audience and the ability for DOD at large and and really the kind of larger community that needs to build highly assured systems you know even Commercial Air transport and and other needs um you know and so correct me if I'm wrong here but you're in the first year of kind of a three-year um uh you know kind of Journey here on this um what what are we looking forward to what um you know as we kind of progress um what do you see on kind of the next Horizon yeah so so the first year was actually to build up the the core Machinery of expressing these interfaces this interaction between analysis uh kind of uh executing and finding out when they are wrong what to do what is the the feedback that we we can provide uh do the trade-offs of some of them or doing um also variations and alternatives the second year we'll be talking about inter outside the interaction between the different domains as we described before in the third year is about making sure that the implementation actually matches the the model the assumptions and all that right so we're going all the way down to to the to the implementation as we said in during the talk and that's actually going uh for the third year that's what will happen in the third year if you will and of course uh during all this time we will be uh doing experiments that are representative uh for for the dod and our customers and so um and and I you know I'm sure I know the answer to this one right but so um we're looking for partners to kind of go on the journey with us right I mean so um what what's what's a great partner for you I know that we have some in in kind of um say Aviation kind of areas but you know what are other options what you know who who can benefit yeah so the good question actually anybody that needs these uh models that are working so I think because he students and of course the the big push on digital engineering and uh model analysis build is actually at the core of our uh our mission right so anybody working in that Arena will really be a great partner for us that sounds great and uh I hope there's some audience members out there that are kind of uh hearing that uh it's a great way to get you know kind of really Pro uh you know kind of you know high-end expertise and formal modeling and folks who can help you uh kind of develop your formal models um you know really kind of make sure that you know kind of the software and systems you specify and acquire Etc uh really are going to meet the the up to the needs of kind of all the different properties that we can kind of check we're heading into a break we're going to return back at 2 15 eastern time and the we'll be hearing a panel uh conversation and it's addressing devsecops challenges uh Dio thanks so much for for being here and I really look forward to kind of hearing as we progress and and hopefully uh some DOD folks will be reaching out and we can really kind of make an impact for their programs especially uh kind of providing the kind of expertise that we kind of really have on this team it's it's world class thank you thank you [Music] [Music] [Music] [Music] thank you [Music] hi my name is Tim chick I'm assert systems technical manager and adjunct faculty member cargo mountain university understanding and articulating cyber security risk is hard with the adoption of devsecups tools and techniques and the increased coupling between the product being built and the tools used to build them the attack surface of the product continues to grow by incorporating segments of the development environment thus many Enterprises are concerned that devsecups pipeline weaknesses may be abused to inject exploitable vulnerabilities into their products and services an insurance case can be used to reason about the degree of security for both the pipeline and the product the devsecos platform independent model can incorporate the elements needed to frame a software Assurance Case by showing how Gathering evidence can be combined into an argument demonstrating that the risk associated with a given pipeline instance has been adequately addressed this in turn provides the organization with the basis for making risk-based choices tied to assuring that the pipeline only functions as intended join me at my session to learn more about identifying threats and ensuring your desktops pipeline using model-based systems engineering [Music] [Music] foreign [Music] [Music] foreign [Music] foreign [Music] thank you [Music] foreign four the seis hosted flocon a conference focused on using data to defend Networks from its earliest days as a small gathering of analysts to discuss Network flow to its current focus on situational awareness beyond the network silicon has brought together researchers and practitioners to engage deeply with the latest tools methods and processes for using data to defend networked systems foreign this year's conference will be held from the 9th to the 12th of January in Santa Fe New Mexico our keynote speakers are rear Admiral William Chase Deputy commander of Joint Force Headquarters Department of Defense Information Network who will share his expertise and experience coordinating defense of diverse and volatile information systems and Mr Jay gasley acting associate director for vulnerability management for the Cyber and infrastructure Security Agency whose experience with managing and coordinating security and vulnerability management for large-scale systems provides something for all of our attendees to learn in addition to a slate of research presentations from the Department of Defense cyber crime Center Cisco the Mayo Clinic Chevron and Moore we are offering open source information security tools training and workshops on emerging web Technologies and Cloud deployment security I hope you'll join us in Santa Fe this January 9th to 12th and I look forward to meeting you in person at flocon 2023 thank you [Music] thank you [Music] foreign [Music] [Music] foreign [Music] project the integration of artificial intelligence and significantly enhance the capabilities of cyber physical systems indeed AI based functions extend server physical system capabilities by collecting and processing larger flows to provide Advanced control situational awareness and autonomic capabilities but at the same time these advances make it other treasure systems understanding the impact of AI functions on system safety is an ever increasing challenge that requires a fine understanding of the system architecture in the safer project our SCI team of researchers is investigating the impact of AI functions on the Assurance of safety critical systems so join us for sem research representation we will highlight all the SEI is advancing the safety analysis capabilities of ai-based cyber physical systems our work considers modern-based engineering and Architectural modelings as key enablers for system Assurance we expand water-bases damaging with mathematically grounded techniques to analyzing architecture of AI based graphical systems and derive an argument on their safety so join us to learn more about the safer projects and its contributions to improving system assurance [Music] thank you foreign [Music] thank you [Music] [Music] thank you foreign [Music] [Music] [Music] [Music] as your organization adopts the software Pathway to facilitate rapid and iterative delivery of capability co-evolving architecture and code becomes increasingly important one of the reasons that you invest in architecture is to ensure your system is extensible to provide for affordable future growth unfortunately developers sometimes Implement code that diverges from that architecture and puts extensibility at risk many times these discrepancies are only discovered much later when you're building a future release at this point remediation is more expensive and will likely delay delivery of new features I'm Dr Robert Nord principal researcher on automated design conformance during continuous integration a project that developed a prototype tool that automatically checks that each code commit conforms to its intended architecture the key to this work is new research in extracting design information from source code a challenging task because there are a few indications of intent in the code and because implementations show significant variations we exploited the use of software Frameworks to advance Automation in architecture analysis have a look at our presentation to see what conformance checking is feasible today and how continuous integration can be extended to improve the conformance of your implementations to your intended architectures [Music] foreign foreign foreign [Music] [Music] foreign [Music] [Music] foreign [Music] [Music] [Music] welcome back everyone I hope you got a quick break and something to drink uh we're gonna continue on with uh Tim chick he's going to deliver a talk on addressing devsecops challenges it's a short video um and we're going to follow that up with a panel that's going to talk about the cyber security challenges associated with devsecops um Tim chick is currently leads the software and systems engineering team focused on delivering trusted valued and relevant software engineering and cyber security approaches for software intensive systems it does a lot of work both in the Department of Defense as well as Department of Homeland Security he's also an adjunct faculty member at cmu's software and societal Systems Department and with that I'm going to pass off to the quick intro and then we'll head into the panel thanks everyone hi my name is Tim chick I am a certain systems technical manager I'm also an adjunct faculty uh for the Institute of software research I'm here to talk about addressing devs like Ops challenges using model based systems engineering so kind of the agenda I'll talk about what what is devsecops what does it mean uh to me and then talk about these basic two challenges connecting the process practices and tools and then also the cyber security aspects of devsecops and then finally how you can start addressing these issues with model bases of engineering so devsecops what is it right so here's a definition of devsecops the key here is it's a cultural and Engineering practice that breakdowns barriers and opens collaboration between development security and operations right a pipeline is really the in the connection of what some could describe as uh factions or traditional factions in a traditional you know waterfall software development organization right you have development which values features you have security which values uh defensibility and you have operations which value stability right and you really think about that you know the operations folks they're being measured by uptime how long is the system available how long is it you know is it is it reliable right where developers are always focused on the next feature the next capability and then you have security as how do I defend it how do I make sure that only the right people have access to it right and when you have that and you look at the diagram here right what you see is you have these factions trying to work together not always agreeing right but then they're being squeezed by this risk the benefits the quality you know the business aspects as well as time uh scope and cost right there's constantly pushing on and that actually creates more friction between uh the organizations so then you also need to take an Enterprise view right A lot of people when they talk about design Ops they focus on the product right that's just one section of it right that's that value that you're developing to a customer or that service that you're providing right the other two aspects is the vision the business Mission itself right the why and the who the Enterprise exists for and then also this thing called the capability delivery right this is actually the the people that process the technology that enable the organization to build and produce their products and services and at the bottom you have this common shared infrastructure that supports the entire Enterprise so what are the challenges uh well so the first one is this this interconnection between process practices and tools right um devsecops is not a static thing you don't set it up and then you just walk away right the tools are constantly changing the products and service that you're producing they're evolving and with it the capabilities that enable those products and services need to evolve sometimes your business uh Mission changes maybe your user base what they want from you your customer base change right so these things are constantly changing and that that impacts of your ability to uh produce right and build and operate effectively and efficiently the other part of that is a lot of organizations say Hey you know I'm doing Azure I'm doing devsecops problem with that is is you know how are you doing that right uh George uh box famous mathematician is creative saying all models are wrong but some are useful right while he was talking about mathematics it's true for most agile and devsecops techniques and methods as well right they were developed and they were designed to meet a specific need right they have their ideal you know team size their ideal technology stat there are their ideal architectural view right um and so the key is is it might depict the right methodology that meets my specific need right and and this is why it's really hard for one to say that I'm doing Azure I'm doing this you know correctly or if I'm not right uh so I really you know so what do I do right well there's three kind of fundamental factors right the ability organization or organization right um you know it's changed right change management takes time it takes resources um so do I have those things available to take to adopt this new methodology or this new technology right the other part is determining the suitability of the academic practice in the development of a given product or service right certain techniques are good at certain things right certain you know and so make sure you have the right environment for that right and then the sustainability of the adage under the projects and organizations right how do I maintain this how can I you know continue it to uh support it the Second Challenge is cyber security right I took this uh View and I put this web over top of it right the idea here is is that you know traditional uh cyber security threat assessments threat Now is really focus on just the product here right but with modern software development modern software engineering techniques and tools and practices the pipeline in the product have become so tightly coupled right that it is possible to attack the product through the pipeline right and so the attack vectors have really just grown almost exponentially right and so how do I secure that how do I do the analysis how do I determine that I've done a good enough job to address the the the attack surface you know the threat of of some type of cyber vulnerability taking advantage of my system right so you know to to address that cyber issue right you got to think about what a term called software Assurance software Assurance is really the software engineering aspect of cyber security right it looks at you know the love of confidence that software is free from vulnerabilities um and the second uh definition was actually better because it starts with the positive right it's application of Technologies and processes to achieve a required level of confidence that software systems and services function as intended right or in the intended manner right and then it's free of accidental right it keeps going on uh the reason I like this one better is because it starts with the assertion right and then from a from an assurance case and we'll talk about in a second yeah the rest of that definition are defeaters to my Assurance right you know my level of confidence is only as good as the way in which I have addressed those things that would defeat my ability to defend my system right so again so so how I determine that it really a few factors one is risk the perception of risk drives Assurance decisions right the other aspect is the integration right highly connected systems require alignment of risk across all stakeholders and the system right go back to the original diagram right you have the product the capability and the Enterprise all of those stakeholders are in play all other stakeholders impact your ability to secure your system right and then you have to trust trusted uh dependencies right your Assurance depends on other people's decisions and the level of trust you place in these depend dependencies right there's lots of players in your organization different tools different techniques entirely integrated tightly coupled in a devsecops pipeline right and then you have the attacker right and the key here is there is no Perfect protection against the attack against attacks right this idea that you can reach this 100 security where no one could possibly do anything bad with your system um it's not just a realistic it's not an economically achievable thing so how do I mitigate that right you understand first thing is understanding risk is hard right um and so what I mean is the ability to quantify or reason around the cyber security risk associated with my product and my debt setups pipeline right without that I'm not able to properly balance between features defensibility and stability right go back to that Triad at the beginning of the definition of devsecops right and I also need to make those trade-off choices to achieve my organization's Mission within a cost-effective way right I only have so many resources that I can apply to security they also need to build features and I also need to operate the system right um and so an insurance case is a way of doing that right it allows you to reason through it using Assurance cases and software isn't new um safety critical software uses those techniques um and and we also believe that it can very much apply to cyber security as well and helped you address these issues so what is an instructor what is an assurance case right it's really in a structured reasoning system right you start with the claim and I go back to that software Assurance definition that I provided right it's you know I only function as intended so in this case it's a devsecops pipeline right I only function as intended right and then I have sub claims right and make arguments to show you know and then evidence right to support my claims right and and having a structured way of doing that is really where model based system engineering comes in so addressing the model based system engineering right so how's it done today right well in the Department of Defense right a couple of you know guidelines there what they really do is they say you know I want you to do devsecops I want you to to do process Improvement but it's really up to the program office to make the final decision right program author shall ensure that software teams use iterative and incremental software development methodologies right and use modern Technologies right so it says to do it but there's a complete lack of sufficient capabilities to design build and implement the a devsecops pipeline um consistently right current guidance fails to prepare programs to address the social technical aspects District Ops right it does not define it does not provide a definitive uh reference of source that one can use to actually evaluate different uh alternatives to solving or building of your pipeline um so it really brings me into this platform independent model concept right so you know it's built off this reference architecture construct right reference architecture is authoritative source of information a Pim is really a general and reusable model for Solutions and for our place we created devsecops uh platform independent model and really where it fits in is if you just go out and do a internet search for devsecops usually find two things one being a high level usually the infinity diagram you know academic kind of description of what devsec devsecops is and some basic principles around it on the other hand what you will find is you know buy my tool buy my solution right some someone's you know a vendor trying to sell something right well how do I as from an engineering perspective rationalize which tools I should use which processes I should use right I really need that next level of abstraction and detail um beyond the academic piece to allow me to make it to be a good consumer and actually apply engineering to build my pipeline using the right tools and the right uh techniques that are available from various vendors or I could build my own if necessary right uh so that deficit goes platform independent right the idea here is it's an authoritative reference to fully design and execute an integrated agile and devsecup strategy right which really involves all the stakeholders right a lot of the material out there also really focuses on the software developer's view of the world right we also need to worry about the Securities view of the world and the operations aspect of the world right enabling an organization uh to really think through and do analysis of alternatives for you know pulling the pieces together and be really conscious and thought in terms of building their their pipelines a lot of organizations we go out and we work with we ask them like well how did you make that decision well we just kind of picked it right as well why is it not working for you well we didn't know we want to do X Y and Z right software engineering system engineering model business engineering allows you to think through I want to do x y and z now I have a criteria to figure out which is the right tools which are the right techniques uh to really increase my ability to be successful right so what is the models uh consist of right so we used uh uaf unified architecture framework um and so the Jessica's pin consists of basically five main categories right one is being the requirements right focuses on you know thou shal statements what are the different aspects one would want to build this pipeline the other spec is this the the capability and the strategy views right so if I understand my requirements what are the basic capabilities that I need to to fulfill those requirements right we came up with 10 of them uh the next part is it's operational views you know what's the process what are the interconnections between the product and the capability how do they come together um and how do they intertwine with the business aspects uh the other part is Personnel right um while we're platform independent we don't say you know this person does this job there are some critical roles within the desiccups pipeline uh this is just a snapshot of some examples right it's you got the business aspect you got the engineering aspect got the operational aspects as well and then you know you need to bring them together right the energy the intersection between these different aspects is really a key aspect right how do what roles map to which um activities and what is their role are they performing the activity are they observing the activity right are they approving the activity right those aspects also come into play because that is usually where uh issues fall is it's the interconnection between these techniques um another example operational activity Matrix again bringing the connections together right the other part is threats right how do I what are the threats to my pipeline where the areas of concern right um based on the Technology based on the requirements basically the interconnections right mod business has the ability to capture those effectively and then what you would expect is in a platform-specific solution they would um explain how they how they address that that threat how they mitigated that threat right and that goes back to your Assurance case of you know I've mitigated this to feeder sufficiently that I I can move on that I'm assured that my pipeline only functions as intended my pipeline is not going to infect my product uh or impact my customers right and here's kind of example of bringing it all together right the very center box is hard to see is the single activity there's larger boxes are all the are the threats right and then you have the the attack aspects as well as the the yellow oranges boxes where are the players right you have the attacker you have the the roles and responsibilities of the people who are responsible for mitigating that particular risk right this gives my obviously gives you the ability to actually capture all those constructs and see their interconnections and then begin rationalizing how one could mitigate or how one could address uh that uh that threat right and then another example of the views uh showing how all these things can come together you begin actually applying Software System engineering Concepts to do the analysis to really build the the Assurance case uh that your system only functions as intended so definitely got you know really enables organizations projects and teams and acquirers right to really specify what devcyclops is what they need what they want this is especially needed when you are hiring someone to help you build your pipeline or where it's not really the main focus of your business right it allows you to really do analysis Alternatives that say okay which features do I want how does changing this particular feature impact my organization before I implement it right and it provides a basis for threat and a tax service analysis it can help you build that Assurance case to really understand that my system only does what's intended and you know I've mitigated adequately you know the the unintended side effects so kind of in summary right with model-based systems engineering and the desk Ops platform independent model you begin doing the right analysis to understand and build you know that social technical system that is trustworthy predictable And Timely so this is this is our team that uh really helped enable and build this platform independent model uh for devsecops um we're very much looking for collaborate collaboration uh so please feel free to reach out to us uh info sei.cmu.edu um thank you uh for your time hi everyone I hope you enjoyed the video as much as I did I want to remind you this is a live panel and we're really looking for your kind of questions and interactivity here and the topic cyber security challenges associated with devsecops and I'm going to introduce you to our panelists uh we have Miss Natasha chevenko she's a member of the technical staff here she specializes in system engineering and model based system engineering and threat modeling methods uh I've already kind of mentioned uh Tim chick we have Joe yankel he's a senior engineer an initiative lead for the devsecops Innovation uh at SEI he's focused on analyzing and improving the devsecop's posture of organizations working with us and Dr Carol Woody who's an experienced researcher focusing on organizations to identify effective security Risk Management Solutions and developing approaches to improve their ability to identify security and survivability requirements and with that I'm going to hand back off to Tim go ahead all right thank you Chris um so I just want to talk start off with a few questions um the first one Carol let's start with you um why is he sharing the pipeline so hard you're dealing with third-party software and hardware and services um and their major factors for both the pipeline infrastructure and the developing products uh it's complex enough just to assemble all this stuff get it operating or get the uh the product working much less worrying about the cyber security risk for all this range um and the capabilities and it can be very daunting we've got to deal with um a mix of software at multiple levels uh each one of the products that you've got are components especially if you're dealing with a lot of third party have different unique outputs limits uh logging capabilities error handling so all this information has to be accumulated and monitored and managed somehow and these tools weren't designed to work together so when we're building a flow you're actually creating scripts so the pieces will work together that's more processes and as you automate all of these pieces over time uh you're creating a situation where it's more and more difficult to actually see anomalies you're removing the human in the loop and you have to actually create ways to Monitor and manage the parts and pieces that are moving so that you can identify potential risks and react in and adjust to them plus all the software has to be updated over time too I have a lot of things to get done ah okay uh Natasha do you have anything to add or a joke yeah I think I I can add a couple words uh to that is um on the point that we are removing the human from uh from a processes uh ornament uh people from a process that immediately involve with the process they do uh process information uh do the uh work by themselves but we still have them or even um have the humans on on people on as a different role they have to configure uh all of the tools and um this is another um surface of potential attack because you have you have human that actually um can uh process a configure a tool that do something wrong so it's not like we're removing the humans completely from we we change their uh role in a system and sometimes it's even um uh make the whole process more complicated because it's not obvious that humus is there human is some uh in some way is uh on a big stage actually coordinating and currently the processes and organize them as a pipelines so uh this is additional complexity and add additional complexity to the system and some of this complexity is not obvious complexity right but I think it's actually a good transition between the complexity that Carol talks about and the human factors you're talking about into kind of our next question is you know why do we need you know a model to support the software Assurance of a pipeline right and you know and my kind of take of that is is that you know while there are techniques out there for building um Assurance cases from you really from the safety uh Assurance aspects of practices um things are all really Guided by you know the laws of physics right and trying to apply there's some of the same techniques to you know you know assuring a system right cyber Assurance um you really start needing that model to bring that structure um to be able to to create some standardized approaches and really think through um the Assurance of the threats and how the impact and how they work through um and how do you mitigate those things um because ultimately you're trying to assure that the product in the pipeline only do what they're intended to do and nothing else right you know the concept that I can assure that the system is 100 secure is really a false Assurance right you can't really I mean the the the the adversaries are constantly come in with new techniques and new ways to try to circumvent your security measures so really you're trying to determine the resiliency of your system and ensuring that it does what it needs to do um and so I think a model helps provide that structure that system engineering approached to it um do any of you have any other thoughts on that I would add to that that the model helps you uh organize all of these many many pieces of uh uh details uh so that you can actually flow pieces together you can reassemble them with different views in the model so that you can put together multiple perspectives and compare them to each other and actually create pictures that you can bring multiple people together to analyze so that you can put shared eyes on the problem as well I think all of those add value so Joe what are your thoughts this whole model basis of engineering and devsecops bringing them together was kind of new with this research what what was your take on that and the lessons you've learned through this research you know it's a little bit the same as what everyone said it's really hard right so some of what we've been asked to do model based system engineering digital engineering devsecops it turns out it's it's very complex and what we discovered is uh there's a lot of guidance out there but as you dig in deep a single source of truth right a formal method for doing this is we found it to be very valuable right to understand details about these processes about the especially about the people involved right so helping organizations understand the roles necessary to do this was a very it's something a model strong in right it's a really good solid source of Truth for all the different expertise um so that that's what I'm seeing now is is that it's a it's a great way to collaborate among other experts at our own Institute bodies of knowledge that we can capture in the model as far as these being really good practices and processes that we should follow um one other thing that's a good model helps with is to um uh project you can do planning you can play what-if games with it but if I change this what's it going to change um and that's hard to do outside of that um to that question or you want to move the next question uh I just wanted like added one comment I absolutely agree with uh with Carol that model uh facilitate these games what if and uh especially it's it's easier to work with uh votive scenarios if you have a common picture in front of several people that can uh present theirs like separate uh points of view and that in the end you will have a solution you will have a decision uh that would be better if these people will think about it like on their own uh and actually model itself like pem gives you this uh this picture in the middle of a table that everybody is around and thinking about it uh about the same picture there is a language that shows you like helps you to uh facilitate this thinking and present this information in a common way for like for everybody so it's I think it's um Carol's points very important so that kind of leads the difference I mean you mentioned modeling what's the difference between you know apart from an independent model platform specific model and how do those contexts really help with threat identification um thread identification thread modeling actually start uh with collecting information rate the flows you need to identify the assets you need to identify the uh actors that participate and assist in the system and uh their roles the involvement or you need to identify threat boundaries there's the steps that like almost any thread modeling methods are employed but up and Pam actually gives you all of this information it's already there if you have uh the model it's a Pim that you used as a reference architecture or you are you created uh platform specific model for your system you already have you have a data flows there you have a data structure uh that needed for thread modeling you have potential actors external internal that involved in this processes or actually have access to to your assets and essential assets it's part of your architecture or its critical processes that your business need to implement and they can be at the asset by themselves or pieces of information then flow between them so it it's almost there and um you can um if you use the like for example pem you already have kind of set pres uh set of threads scenarios and threads that are related to the spam on like on a higher level so you can look at that and see how it's related to you to your specific system is it enough already for you to analyze uh uh your system and identify the thread that's specific for you and PM contains the list of types of attack developed by miter so we provide it in a pen where you can use to for your thread modeling processes if you if you want to so it's in some way for thread modeling it's one store deal you if you have your Pim or like a PSM platform specific system for your system you have already everything needed to start your thread modeling and thread identification yeah so I mean what I see is it kind of the lead on to that is is having that platform specific model where the platform independent model excuse me really provides that reference architecture that allows really large organizations who might have multiple pipelines supporting multiple customers multiple capabilities multiple value add propositions to really have some structured approach to bring consistency across their Enterprise for starting to build those Assurance cases to do their threat identification workshops their their vulnerabilities discussions as well as the mitigations right because you have you know when you build an insurance case I want to prove that it functions only as intended and really there's threats and there's cyber concerns really become defeaters to my Assurance case right and so the the Pim provides the structure to be able to build a consistent Assurance case argument and then your platform independent model when you map it to that reference architecture provides the evidence one needs to make those claims in your Assurance case and I think that's really key and how they kind of can work together to really complement that third identification but then that follow-on uh threat mitigation um uh Joe do you have anything else any other thoughts on that I think like you said it's a it's a great One-Stop shop as Natasha said to capture all these things it really is it's difficult right as we really try to plan Assurance we realize it's it's planning there's hardly a better planning tool than than a platform independent model to plan what you and do right you have a chance then if you understand the scope of what you have to assure to possibly achieve that we've seen a lot of organizations right fail to close the deal so to speak right begin to build this environment up without having done that planning and they fall short well it forces yeah well it forces you to take into account uh a range of um expectations that you might not have immediately come up with uh because basically you've got um a baseline to work from uh and if you're just assembling tools and putting together your own Pipeline and not really looking at what is a standard base of consideration you're going to miss things yeah so I think because they're planning and that consistency but that's really key right you if you don't have that complete list of items you can't really have a complete plan and I think that's kind of General you're talking about is we've seen people go down this path of building this capability these pipelines but they didn't think it through they didn't really understand how difficult it was and how all the elements you need to interplay with each other to be successful right and and they've they've suffered because of that through technical debt through a lot of rework um a lot of missteps um having that pimp having that holistic view how things work together you can begin thinking about and mapping out you know what is my implementation plan how am I going to build my model um how is it all going to work together um that's really key um just from a basic you know project planning project tracking and building you know a good capability for your organization um let alone the cyber security implications that now I can also make sure it's defensible um and you know I'm using good cyber hygiene especially a world where finding people who have these skills you have the experience in devsecops um sufficient to build something from scratch is really hard to find having that reference architecture having that model with that information there that checklist um so that people who are more novel at it you know have that guidance of those experiences um I think you know Joe you kind of mentioned you know it's also a great way of free lots of different experts together capturing their that tacit knowledge that's in that's in between their ears and helping make it a little bit more explicit for the rest of the world to benefit from right uh so so Joe I got I got one last question we'll see how it goes maybe we'll get a few more uh in chat um how do you envision the model being used in the future so this is what I'm excited about right we just we've really just started down this path of right now as we've captured a lot of SEI expertise right and in the past what we've found is uh different experts approach something in their own way it wasn't very repeatable you know one of our objectives is to uh you know teach others how to do this and so a model provides that platform so going down that line right now we we do lots of devsecops Assessments right let's go look at an organization let's see how they're doing things um and often what it looks like is we have a particular expert that asks questions based off of their own expertise um and it looks different every time someone does it a dude from person does an assessment might get a different result so one way we're going to use the model is to be a foundation for describing the the activities the people um the things that happen to do good devsecops right we have requirements we have capabilities that now is a very consistent platform to assess the current state right so step one we're learning how to improve the model as from the Viewpoint of an organization self-assessing where they are to understand here's what I I do well at here's things I can improve upon it's really strong in saying these are the organizational people I need the Personnel I need to do these tests that's often very enlightening when we find out there's a lot of work here are we sure everyone this person doing this much work it really show it really allows us to see that it takes an army to potentially build software devsecops methodologies so when we can catch that very early we can we can plan to hire those those folks early uh so that's that's one place we see it I'm sure some others can add some other ways we're already using it today well I'm particularly enthused about creating a place that's organizing cyber security requirements for a pipeline um too frequently um a laundry list of all of the controls and what needs to be done is provided as the acquisition requirement and then you try and figure out where what does that mean for a pipeline because you're dealing with a different um way of structuring product instruction and you also have to be looking at it from multiple perspectives and the requirements allow you to start to divide up and assign responsibilities so I think all of those are a good starting points for not only creating a pipeline but evaluating ones that are already existing to say where are the gaps how can we improve uh how can we move ahead so that we're creating a better more efficient environment at the same time create creating a better result Natasha where do you see the model going you have any thoughts yes ah I would like to see that uh Pam will mature as a tool a bit more tool of MBC four deaths of cops um I would say that one of the probably Innovation with this tool is that we approach devsecops as a system from system engineering point of view not pipeline not that not just tools and it's mean that we can now apply system engineering engineering methods to analyze the uh deficit cops as a system uh use uh Gap analysis use impact analysis and so on so forth so um Pam can provide these tools uh um right there so we can add some analytical views that will help organizations not only Implement their pipelines to do decisions consciously if they decide that something should they should not do or they need to go to different level of maturity they will have a tools to make this decision um after careful analysis of all information uh they need it for that so tool can provide this Pim can provide these tools this is one way I can see uh payment goes another is to provide organization guidance how to create a platform-specific models for themselves because it's not trivial so you um right now it's numbers of reference architecture uh everywhere that actually talk in specific tools um yeah platform uh specific tools but we tried we will try to uh create a transition but I won't say that stress-free but at least it will be easier transition from pem to PSM so the organizations can create uh models for themselves create specific architecture for themselves and have the tools again system engineering tools right there to do uh conscious decisions not not just looking what is fancy tool right now there or what is like fashionable to do it right now right so this is a couple of uh areas that I think the pimp can uh be developed and be more helpful uh to uh to Industry and to to any organization that uh want to do the deficit cops a bit more structured and formal way okay so I do have a couple ideas too that I don't think any of you really brought up our trust and one is um the idea of Jessica's measurements right I think that's really a key weakness in terms of the dips that got practices and and the research out there and so how do you really articulate you know where those right decision points are within the the various aspects of your pipeline and then what data can help inform those types of decisions and really think that through from a real holistic view of you know based on the roles and responsibilities what questions should you be asking what type of data throughout the pipeline can answer those questions from like a gold driven measurement uh construct as well as you know through the Automation and through the tools what data can be collected um and and correlated across your different you know types of tools and data collection aspects you actually begin quantifying answering those questions right I think that's I think a great place that the model can go to help support um and then secondary to that would be um with that additional data how do I begin to describe residual risk right I think that's one another area that cyber security as a whole struggles you've not just devastate house pipeline but just in general is how do I move from this compliance checklist type of cyber security approach to really being able to articulate and defend my decisions on what controls I included or even more importantly what controls I didn't include and why I did that um and really be able to quantify and not just you know gut feeling you know might you know some basic arguments to really build you know quantitative Assurance case um around have been a defensible system so I think that's another area that the model can lead and then I think even further going forward um is really thinking of machine machine learning operations right I think there's a lot of automation there how do I trust that you know when I put you know artificial intelligence and machine learning constructs into my pipeline where does that fit in what is that what does that look like right so I think those are a few additional areas I think model can go towards into supporting that um I do have a question here um we have I think we have like two more minutes left um you know what level of maturity is practical to achieve today any thoughts on that maybe Joe you do a lot of Assessments what do you think about that level of maturity construct I mean we see various levels of maturity right we folks can develop software today and systems that don't you know maybe don't they're not all the latest best practices um I think we understand what the future can hold and we we think that's a very mature system um eventually that's what we strive for right we strive for high mature systems where we have uh Automation and AI to help us make intelligent choices um and so we'll I guess we'll we'll see we'll be happy just if if most organizations right now can be level one we can develop software and we can describe what we've done and how we've uh how have we secured it and how it has high quality right to be able to do that with intent is a pretty good goal to start with yeah and I think you kind of mentioned that that artificial intelligence you know automated decision making I think the AI machine learning you know research needs to figure out how do I trust the AI and machine learning uh constructs and once they figure that out we can figure out how to get that training or how to get that decision making automated into our pipelines I think that's kind of the ultimate High maturity for devsecops um so with that is there any uh last closing remarks anyone has be sure to turn it back over to Chris no so I think it's back to you Chris thanks Tim and thanks uh you know Joe Carroll and Natasha that was you know a pretty interesting panel I know that there's a big push across the department um to kind of adopt devsecops uh and and we have a big Focus both on kind of the dev side and the security sides and so uh really interesting panel and thank you guys uh our last presentation of the day is by uh Dr Jerome Yuga um it's safety analysis and fault detection isolation and Recovery synthesis for time-sensitive cyber physical systems uh Dr Yuga is a expert in systems modeling and um kind of the verification and Analysis of those systems including uh work in aadl and I'm really excited to kind of uh hear this uh future work of his so thanks everybody thank you thank you [Music] hi my name is John ERG and I'm the pi of the safer project in safer our team of ACI researchers seconded by a group of researchers from Georgia Tech is investigating the topic of increasingly autonomable cerebral physical systems for those system AI engineering delivers the premises of fresh breeds bringing new functions new capabilities for advanced controls situational awareness or decision making capabilities but at the same time our fear is that this fresh Breeze may be coming from an iceberg and indeed those AI functions make it harder to demonstrate the safety of the system to assure it if we cannot address this issue we may be forced to perform longer VNV processes or we may experience a distrust in the system by its user and ultimately the capability may not be deployed so in safer we aim to address this issue by delivering Advance section analysis techniques to implement software breaks phone detection isolation and Recovery policies that will make sure that any error in the system will be properly detected and mitigated before becoming a threat to the system and indeed if we consider a typical scenario for an autonomous psychophysical system like a UAV Patrol at the very top level this system will need to find and detect Intruders if any will be able to recognize threads but for that we have to understand the concept of operation in the system let's say factory wheels slow moving parts that system will have to operate in both close and open areas with different environmental conditions like wind lighting conditions if not rain it will have to execute type Maneuvers to enter and exit buildings and of course we need safety margins to avoid damage else this system is also autonomous in the way it will make decisions and also in which it will interact potentially with human operators the key research question behind this type of system is how can we guarantee that the system is safe to operate and will operate safely in order to do that as we are building and delivering the system not only do we deliver the UAV ready to fly but we also deliver an argumentation that this system is safe to fly and will fly safely this argument is if we express express it in plain English an argument that claim that the system is safe because it does such and such Mission as collection of requirements that are implemented by an architecture and collection of software code and that VNV activity demonstrates Street conformance to uh to the element above furthermore that this system can be operated safely and that in the event of Hazard of threads then those events are properly can be properly monitored and ultimately mitigated to build this argument we need a convergence between multiple team system Engineers cyber physical system Engineers software Architects and embedded system Architects that they will build together a collection of models concept of operations requirements model they will execute other analysis and derive an architecture an architectural description of a system and then implement it as code in safer we are building a comprehensive approach that will combine modern-based system measuring and safety assessment and for that we'll work on three tracks Define architectural patterns for fault detection iteration and recovery tool support in order to understand the impact of this fault detection resolution and Recovery policies on the behavioral system and finally extensions of existing tool sets so that we can produce automatically an argument that will articulate all those artifacts that are produced by those various engineering processes but we don't want to generate another boilerplate pipeline what we want is to bring additional engineer rigor assume guarantee type of reasoning let's say for instance that assume that we know some something about operational hazards that we know thoughts or vulnerabilities on sensor actuators timing and anomalies or AI function misbehavior basically all the elements that may contribute to an unsafe scenario then we want to be able to grant you that a candidate architecture will properly mitigate faults boss at the design time but also at implementation time and finally at runtime and for that we developed three contributions first off for taxonomy and guidelines for selecting fault detector so that Engineers will know precisely which policy are relevant for addressing a particular threat in the scenario a mechanized semantics of architectural descriptions of a cyber physical system so that we can simulate model checking or even prove some properties of a system and finally a way to derive a sexy argument from all those models so that they can be reviewed by certification authorities let's start with the first contribution which is how can we select the right Mouse for detection iteration and Recovery or fdir policy for a particular system from an architectural perspective the Cyber physical system will interact with an environment through sensor that will bring information so that the system can build a representation an abstraction of another environment a AI or ml functions will operate on those elements and execute algorithms for predicting inferring or optimizing some parameter of the system and from this make a decision and perform basic action that will again impact ultimately the system again so there's a dual challenge here in that first off we want to make sure that the architecture can mitigate any earnings inputs that will field um those learning enabled components in terms of timing loss of timing reference in terms of inconsistent value and conversely we want to make sure that AI ml functions will not misbehave that will not trigger wrong scenarios in the system and for that we need again runtime Assurance capabilities for detection capabilities in order to capture all those elements we need a baseline to model our system and we selecting the ADL language hey there is the architecture analysis and design language it is an sa International standard that provide a mean first off to capture a design and then to execute analysis or even code generation and to produce evidences for specific questions you may have on your design like for instance performance analysis timing analysis safety analysis but also to perform modern review for specific quality attributes and ultimately generate an assurance report out of those elements foundation for a lot of elements and for safer we are interested in safety for that what we need in order to assure our increasingly autonomous cyber physical system is a way to capture the fault scenarios in a system nationally this AI function may receive may receive data from the environment take a decision and act on it pragmatically what it means is that some data is processed by this platform and may be subject to specific rules like for instance data tampering the lows the loss of a reference point due to a sense of fault or attack a loss of a timing reference because of latency and Jitter or timing relation in system and of course all type of other typical force that may happen like a defect in a propeller defect in a sensor all of that can be properly characterized using the ADL EMB to full text enemy the emv24 taxonomy provides a dictionary of all the fault that may arise in a cs10 with their mathematical description and a complete understanding of what it means for instance to have a value error versus having a timing errors from those premises we were interested by the foreign question can we first off capture a typical for scenario for AI systems and then can we select the right Mouse detection mechanism there is a huge literature on those topics and so we need to organize it we started by defining a decision procedure using the mv2fold taxonomy as a pivot the basic idea of the following for a specific scenario we made this derive what is a set of errors that it produces and isolate them and Define them properly similarly for a set of existing detection algorithms we may Define the set of era classes that are sensitive to timing value Omission etc etc and from that we make the claim that the detector is efficient if the set of errors that you may detect contains a set of errors that can be produced beyond that what we have developed with Georgia Tech is a surveyor for detectors a set of mappings for all for a collection of phone attack scenarios two detection mechanisms and this way we bring to designers a rich set of guidance so that from their description of thought or attacks and I use a major arrive the right modes for detection algorithm furthermore as we are investigating the topics of the impact of faults on system as I said we have to make sure that an architecture does not impact too much AI functions a typical example is reference modern controllers that will control the trajectory of vehicle the Arctic the architecture May induce delays or tutors as clock offsets because of the timing takes for data to flow from sensors to CPUs what we demonstrated with Georgia Tech is that if clock offsets are bonded then the reference point learning controller may still converge it may be less efficient than in the pure theoretical case of no clock offset at all but we demonstrate that if those clockoff sets are bonded which is something ADL can help demonstrating then the system May converge simulation results also shows the limits in outsets that ensure convergence if we go beyond those limits the system is basically impossible to control with those two elements we contributed to folitation resolution and Recovery as foundation for safety we have defined guidelines to go from a fault scenario from an AI system to the selection of the follow detection isolation and Recovery principles also we have demonstrated that there exists safety margins an architecture should demonstrate in order for AI functions to be efficient but that's ideal considerations we need to go further and understand more precisely more problematically what is the impact of AI functions on the architectural system and for that we need to revisit the way we analyze the system if we look at ADF for instance but it's the same for any standard be it ADL CML uml Etc you will find in a document a collection of definitions what to use your component model consistency rules that will make a model value collection of property sets basically attributes that will help you to configure system a tool we're generating internal representation of those elements and Define for instance a declarative model an instance model that will help you to go further and extract information so that you can perform a scheduling analysis static security analysis Etc and if you're interested in behavioral properties in order to perform simulations on model checkings you can add those elements are as additional definition for instance what user State machines attach to your classes of components what is a broad-based communication 760 system and again you can derive your real model that will be suitable for model checking on simulation however there is a paradox here in modern-based techniques in that we claim that product-based approaches are better than document based but at the same time the semantics of the model itself is described as a document therefore it is prone to interpretation errors to enter this issue we move it a bit further and say okay can we Define answer model the semantics of a there in our case it will act as a reference for other tools but also it will improve the standard Itself by eliminating Corner K sales our contribution here is a mechanization of ADL using the code in interactive or improver through this mechanization released as a software artifact we capture the static and dynamic semantics of aadl along with configuration through attributes defined as property sets how does it work first off we import ADM models analyzing Json as a collection of data types those data types are variants of our interpretation if you will of the idea meta model this data type is properly typed by um typing rules that will define basically what is legal from a typing perspective what is legal from a consistency perspective so as by defining it as well from this rules we can assert that a model is correct with respects to um to The adap Standard furthermore we have defined and implemented an operational semantics of ADR but can use a set of rules that will tell you how to execute a model from those elements you can perform proof or model checking or simulation of an aldr model and at the end of the second year of safer we delivered three illustrations of those capabilities first off a complete definition of Alia as a set of COG data types and legality rules simulation capabilities so that you can simulate an ADL model by mapping it to the depth from Amazon we can perform scheduling analysis using the proza library and currently under development full propagation and Analysis of Ideal model again mechanizing calc so as we conclude those two contributions basically our goal was to provide a way to build evidence sales that our system is safe first off at the system engineering level how can we select the rightmost for detection iteration and Recovery policy and second how can we validate that the integration of this area component in the architectural system is safe and for that we mechanized the idea language so that we can understand very precisely how the system will behave those elements are providing evidence sales that must be combined together as we are building the final argument for the safety of our system and for this I will leave the flow to my colleague kitten Anna that will tell you how we extended on that day to generate a argument in the gold structuring notation stored out thank you shiram so with this project we took a requirement specification language and mapped it to a popular safety case argument format so our hope is that this will give developers the freedom to create architecture models using ADL and also benefit from Elisa's requirement verification and osate's built-in analysis we're going to take all these existing inputs to generate a clear and concise goal structuring notation argument so goal structuring notation is a graphical safety case argument format it has six common or six core elements goal solution strategy context assumption and justification a goal element is essentially just the claim these claims are broken down using strategy nodes and then once that claim is successfully broken down you can use the solution node the context assumption justification are used to give more background for agencies that review your safety case so we chose to use respect because it is a highly capable requirement specification language so as we can see here we have the description for the requirement OE is sr2 this just gives a quick breakdown of what that requirement is doing then we have the category keyword this allows for you to group requirements together based on similar characteristics now if we take a look at eahs1 we can see that it also has a refines keyword this allows for you to break down requirements until they are verifiable now to verify these requirements we use the verified notation now this is a separate notation from the Rex back language but the the connection between the two is easy to understand as we can see that this claim is named eahs1 which is the same as the second requirement that we took a look at So within the verify language we have the rationale keyword which is just used as kind of a description of what the verification activities are and then we have the activities keyword this activities keyword allows for you to call verification methods which verifies that a requirement is met against the ADL model so now we'll take a look at the mapping so for each requirement we use their description as a description for either goal or strategy nodes each requirement is denoted as either a goal or strategy node based on its category uh as I mentioned earlier the refines keyword is used to break down goals which then is able to be used as a sub goal for the rationale we are able to connect that as a context node to its given strategy or goal node and then for verify we use its rationale as the description for solution node so we try to keep this as simple as possible but what we our goal is to allow developers to use ADL and Elisa just as they normally would and just be able to click one button to jumpstart their safety case creation now I'm going to send it back to Jerome to conclude the presentation as we are finishing our two out of three of the safer project our team of Asia researchers can claim already a couple of contributions that may be relevant to your own project safer objective is to extend virtual integration capabilities of model-based system engineering AKA shifting to the left for the case of increasingly autonomous or AI based cyber physical systems first off we derived a set of guidelines so that you can go from a fault scenario to the right mode for detection reservation and Recovery mechanism for your system furthermore we are improving the VNV and simulation capabilities of cyber physical systems by mechanizing the architecture analysis and design language in calc as a way to provide stronger results in the way you can analyze your system for instance for performance for safety finally we extended the repo generation capabilities of the other two chain so that you can generate GSN reports in addition to typical report as part of the execution of your Azure process if you are interested by any of those contributions please reach out to our team first off in the QA session that we follow or directly through this addresses at info sei.cmu.edu thank you thanks everybody for that interesting talk we're back with uh Jerome live who's here for your kind of questions I'll just kind of remind everybody you know make this a challenge for him he loves it uh you post your questions to the kind of YouTube chap the Discord chat um you know it it'll be great he's he's uh he's game for anything and we already have one so uh thank you uh and I'll read it for you Jerome I know you've kind of already uh taken a look for it uh can you offer an opinion on what the human role regarding safety in an increasingly autonomous system should be is it monitor analyze confirm direct other uh you know what what should that look like uh in your opinion well it's it's it's a very broad question actually what we have to remember is that uh increasingly autonomous system is not just one monolithic uh class of system it's a broad spect spectrum and so for for me it's a human uh in the loop means that first we have to understand what is the mission and basically what is and also what is the pace of the system um we for this project we are considering all type of scenarios if we consider for instance uavs systems and we are building one scenario for UAV patrolling the factory for instance this system will be highly autonomous very few human in the place and therefore the human will be there mostly for monitoring and eventually for confirming that there is something suspicious or not but it will be probably every minute every 10 minutes or whenever it's required we may also Imagine fully autonomous systems uh for instance long run UAV patrolling for instance monitoring some electricity grid as you may imagine for those we may expect fully autonomous behavior and very few human interactions so I guess it really depends on the type of missions and the I would say the balance between the capability to implement autonomous function versus the need for human to supervise or monitor them it's it's really dependent on the type of system I would say it's it's how to provide any form of a definite answer so I'm going to pick at that a little bit um right and so and I kind of want to bring it back to the modeling a little bit and what we're capable of kind of making assertions on and so can you help me kind of bridge those so is there a way to you know what level of modeling or Assurance capabilities kind of lead you to say um uh you know um you just check it when something goes wrong type of scenario very light monitoring right um you know and assume that you we've you know it's correct by construction uh almost versus kind of that and other end of the Spectrum which is direct can do you have a feel uh for what level of modeling gets us there well there's a level of modeling it really depends on the type of uh properties you have to demonstrate on your on your system um and actually the reason why we were interested in this particular questions is research question is that if we start with the foundations the architecture of the system and we want to hook autonomous functions if we don't trust the architecture it's it means that we have to supervise it much more often we have to check vertically without the system is still working and operating as expected or not and so it is a reason why we are interested in this correctness by construction of um adl-like models and the way to simulate them and also the reason why we mechanize its semantics in Coke so that we can have a better control a better understanding of the way it will behave and on top of that we will plug autonomous functions and we worked with Georgia Tech to evaluate um the level of I will say Fidelity or the capability to detect misbehavior and what we discovered is that this part is much more difficult because you need to quantify them in the abstract um using some confidence metrics and from that what we may say is that the system May Fail after one hour May Fail after 30 seconds depending on the running scenario and ultimately it's a question between the system designer and the system operator and it's very difficult to to provide a strong option because we have to understand the operational context and we need some expertise which are outside what we do um I will say typically when we do a mblc well in my group we are much more familiar with cyber physical systems for which we can make definite answers but as we bring autonomy we're also bringing operational context and so we need really to be uh to be able to engage with the final users to understand what level of risk he is willing to accept what level of confidence it needs in order to operate the system and that is opening all type of new challenges for us uh are playing model systems engineering for those systems well that's interesting I'm gonna pivot a little bit and kind of kind of go back to some other topics you mentioned in the talk which is can you tell us a little bit why are the fault Tech detection mechanism so hard to implement what what confounds there well it truly relates to the national race because that was I mentioning just before uh if you want to implement a detection mechanism first off you have to to determine what are the other scenarios that may lead to those uh to those faults and usually it's a collection of um low level signals it may be a sense of failure or some additional latency in the system that will corrupt information flowing down from sensors to CPUs and so it is difficult to implement those full detection because to some extent we have to process an additional stream of data so that we can say oh this looks suspicious or this looks completely wrong and to make this decision basically you have to know what will be the ideal solution so it means that in parallel ideally you would have to run a digital twin of system or at least some prediction algorithms to make sure that yeah the system is behaving as expected and the trick here of the fault scenario the more complex is um simulation that you have to run in parallel has to be and so it's it's creating all type of new challenges because we have to implement very clever fault detection for the system and as they become more sophisticated their test and evaluation is becoming increasingly complex and so we are looking with Georgia Tech to uh to um or to say that or to qualify those for detection in such a way that we can reduce those testing elements that's great I've had some other questions but unfortunately we're out of time um and so Jerome this was great work and and really thank you for your time and uh you know continued work on this and I'm just gonna poke one last question even though I'm I'm really out of time um are you this project has another year of effort in it or is this right yeah okay that sounds great thank you well everyone uh that wraps it up for today I just kind of want to cap off the day and and talk about what we've heard you know uh I've struggled a little bit kind of you know bring all of the topics together but you know in talking with the presenters offline and kind of hearing about it that uh I will say uh I'll credit Jerome for this one of the things we can say is it's really interesting if you look across this it's a combination of modeling but in underlying that modeling to make the impacts we want to make you know it's really a bunch of Applied Math and how we're kind of doing that math and you know as a software engineering Institute as the as a DOD ffrdc we're really focused on bringing value to the department and so we're applying this in areas that make you know how do we make uh secure reliable um you know uh fit for purpose cyber physical systems uh how do we make the software development process have security properties as well as deliver you know software on time in an affordable way in devsecops you know how do we use Game Theory to kind of understand cyber risk uh you know how do we make this modeling capability more available to people and lower the barrier to entry to use these and then you know kind of the last one which was really the first talk we heard today right you know kind of transformative gamification one of the takeaways there is while we you know are using gamification to make it more engaging for users in order to not just fall into the kind of you know fun failure where it's fun but it doesn't serve the purpose we really need to understand human motivation uh have a model for why people want to engage and understand what they want to accomplish and so what you're seeing across all of these efforts and whether it be trying to improve the security of cyber systems where it's trying to improve our ability to deliver kind of high quality software that really delivers improvements that are measurable and and you know important for the department whether it's really can we get kind of soldiers and and you know DOD Personnel more engaged in training and in you know kind of doing exercises it's really modeling and math that's enabling us to get there and make those improvements for the Department um and it's kind of impacting across the board um I hope everybody has a chance to kind of think about that give me your thoughts and opinions um would love to engage with you kind of all the presenters all the projects would love to have more engagement with DOD we would love to be able to kind of uh you know kind of try these techniques out for you um and we're really excited about that and I can only tell you you know we've had the AI day we're going to we did some kind of uh defense tomorrow's going to be more focused on kind of software systems uh proper and we look forward to kind of seeing you there again tomorrow for the third day of our research review uh thanks everybody for staying tuned and I hope you join us tomorrow thanks everybody [Music] thank you [Music] foreign [Music] foreign [Music] [Music] foreign [Music]

Channel: Software Engineering Institute | Carnegie Mellon University

Views: 36,218

Keywords: Cybersecurity, AI, Software Engineering, AI Engineering

Id: A1o9Vfj7ii8

Length: 216min 30sec (12990 seconds)

Published: Tue Nov 15 2022