AI Engineering: Ask Us Anything About Building AI Better

[Music] [Music] [Music] [Music] [Music] hello and welcome to today's sei webcast ai engineering ask us anything about building ai better my name is shane mcgraw outreach team lead here at the software engineering institute and i'd like to thank you for attending we want to make our q a as interactive as possible today so you can submit questions in the youtube chat area now or if you're on linkedin or twitter use the hashtag askusai and we will get to as many as we can our featured speakers today are dr rachel dan de zomback and dr matt gaston rachel rachel leads our digital transformation team from the sei ai division in her role she works with organizations to realize the capability of artificial intelligence for mission outcomes prior to joining cmu dr de zomback was an innovation fellow and professional faculty member at the university of california berkeley matt is the director of the seiai division and the founding director of the sei emerging technology center he's also an adjunct professor at the cmu institute for software research welcome matt and rachel thanks jane good morning chain uh glad to be here today yeah it's great to have you guys we have we have lots of great questions already coming in over the last couple weeks uh in in preparing for the webcast so we're gonna get uh we're gonna get right to them so i think the first one we kind of gotta ask you guys is to kind of level set is what's the difference between ai and ai engineering so i'm gonna point that one to matt first well thanks shane that's a great question to to start with um so yeah we're here today to talk about ai engineering and it's important to set that in the context of of the field of artificial intelligence so ai has been around for a long time um usually considered to be the the field of study uh where uh we're looking to figure out how to make uh machines intelligent for some definition of intelligence and ai is a very broad field it's been around for a long time i think 1956 at dartmouth the term was coined um and and ai includes a lot of technologies and has been through a lot of waves over the decades today at cmu at carnegie mellon university we often refer to what we call the the cmu ai stack to look at what ai includes and it includes everything from the low-level computational resources necessary to realize ai capabilities up through sensing and processing of data and information to machine learning and i'll come back to that machine learning concept in just a second and then on up to reasoning and planning human machine interaction autonomy and includes a rapper of policy and ethical concerns and ai is really about the science of all of those things and how to incorporate ideas across that whole stack into building intelligent systems so let's talk just a little bit more about machine learning so machine learning is one piece of ai and really i like to think of machine learning as a way to create ai capabilities traditional software systems you know software developers are programming specific rules in a sense micromanaging the functionality of a system in machine learning systems uh algorithms that learn from observation or learn from data are used to create a model and that model can do some sort of reasoning or inference on its own the important distinction between machine learning and traditional software engineering is that there isn't that micro management process right there's an algorithm that learns from the training data and then the model itself just sort of behaves based on what it learns from that those observations then inside of machine learning and really a big focus of of modern modern ai modern machine learning is is this concept called deep learning where we've taken an old concept a concept that was created in the 80s maybe even the 70s called neural networks which require a lot of data a lot of computation and so in the last two decades we've realized that the internet has provided an opportunity to produce lots and lots of data available for these types of algorithms and we now have the compute available in a very democratized way with the cloud that allows us to train very very large neural networks to do some amazing things like recognizing the damage to buildings after a natural disaster in from satellite imagery and so ai is really all of that stuff right remember that full cmu ai stack and ai engineering is really an emerging field it's not it's been around for a while in different incarnations but it's really an emerging field that recognizes that there's this huge power and promise of these ai technologies there's great applications there's great things that can be done with these technologies but we need to improve the discipline around how we do that in a reliable and responsible way the best way i've heard this describes ai engineering as a concept is from michael mcquade the former vice president of research at cmu and member of the defense innovation board michael describes ai engineering as doing ai as well as it can be done and so it's not so focused on individual capabilities or particular algorithms but around the qualities that we want these ai components and these ai systems to exhibit how to make them trustworthy reliable responsible uh etc so i think that's a to maybe a gentle introduction to ai and and the concept of ai engineering and rachel you might have things to add yeah thanks ma and thanks again shane i'm excited to be here today really excited to answer questions from the audience so please do keep those coming the only thing i wanted to add on was just that there are so many people looking at individual components of this across the ai research community and they're focused on specializing in those specific aspects of focusing on model development or on data management processes and i think a key distinction for ai engineering is we're taking this whole systems approach trying to understand how do these elements intersect and how do they come together to drive towards the outcomes we're looking to see that's the challenge right now with ai is the implementation particularly in context where ambiguity is higher you know it's one thing if you are trying to be recommended a restaurant and the stakes are low if you get that recommendation wrong but in higher stake scenarios so you know that's example of seeing a building after a disaster and understanding the damage to that building there's a lot of factors there a lot of variables that come into play and so figuring out what does it mean to achieve that to get towards that outcome and use these systems in the way that they're intended it requires us not just to look at the model that is the basis of the system but all of these individual component parts and where they come together so we just had a question came in from twitter and i think it's relevant to this section so i'm going to leave it open to either to matter rachel this was from martin asking what's needed to build an ai engineering mindset in the team so i'm happy to kick that one off because matt knows it's one of my favorite subjects i think that one of the biggest elements is some humility and a curiosity for learning because the field is evolving so quickly that i think there's a tendency for people to think oh well person x they have all the answers they have all of the expertise that's needed and it's an impossible goal right now it's an impossible goal for someone to know absolutely everything because the field is too big and the techniques are changing all the time the leading practices there's new ones coming out every single day from different parts of industry and so you need a team that can flex and adapt to where you are in your process you also need a diverse team that can see the problem spaces differently to pay attention to things like potential unintended consequences or different impacts that a potential system could have with ai especially in removing human decision-making it opens you up to all sorts of potential consequences that can be really detrimental to human lives as well as human prosperity so i think that that mindset of we are learning how to learn together that we are thinking about this and are constantly going to need to keep updating evolving our mental models we're never done that part is critical but also understanding that one person doesn't have all those answers and you need to work amidst your diverse team which is much easier said than done um and that is a piece that i think is critical moving forward rachel let's stay with you for just for one more here while we're kind of setting up the stage here how does ai engineering relate to other engineering engineering disciplines and that was from heidi yeah absolutely so you know i'm an engineer by training and work with a lot of computer scientists and i found it interesting in this in this field that there's a lot of competition between the two and people kind of try drawing lines between them in my mind as an engineer engineering was engineering disciplines evolved as an applied science how do we leverage technology how do we leverage math and science to solve problems that matter that could look like developing products processes systems that's what engineers do we try to approach the house side how can we reach this place we want to get to how can we reach this desired outcome that we have our customer has and i think that's the mindset worth taking with ai engineering it would be we can't look at the system development without also understanding the context in which it's going to be implemented in we're learning from a lot of other engineering disciplines uh around what does it mean to build rigor what does it mean to understand systems there's great examples to be taken from civil engineering from mechanical engineering about how do you start to problem solve in in a way that's context specific so we are learning from all those disciplines and trying to build towards that and you know when we define this field of engineering we say it's the integration of systems engineering human-centered design computer science we're pulling those elements together because it's what's needed you can't just view ai from a single discipline and expect implementation to work perfectly you need this multi-disciplinary approach and we view our roles not as having answers either but as being curators and helping to weave threads between those different disciplines and bring them together great before we go to matt with the next one i love keeping an eye on the chat just you know seeing that we got people from uganda kenya mexico uh joining us so we've got a truly worldwide whites it's great to see so keep letting us know where you're from and questions you have so matt let's go to this question for you from joseph asking how do engineers navigate trade-offs among ai system qualities such as between robustness and scalability what would a person in that decision-making role use to determine trade-offs yeah this is a great question and and i think still a largely uh open open question in as far as the engineering discipline is concerned um but what i'm excited about related to this question is this idea that we call beyond accuracy uh let me start by by talking about you know traditional ml ops and traditional machine learning operations and building of machine learning models the the focus is usually on what we call model evaluation right so i've got a very precise very specific task like let's use the simple example of classifying cats or dogs and images and i can measure accuracy of that of that task right so i have data that has cats and data that has dogs and i run it through my model that i've trained and i can measure the accuracy of that model this is the large the large focus of the work that gets done in in building of machine learning capabilities uh a worry about this model evaluation uh concept and how well it does on the specific task to get to the actual question and this idea of beyond accuracy we're keenly interested in methods that rather than looking at model evaluation and the precise you know classification task it's really important and rachel just said this it's really important to think about the business or mission or operational context where the system where this model will be deployed inside inside of a larger system and and that context can actually drive metrics and those metrics are that balance between robustness and scalability and accuracy uh in a really great example that i'll credit martial a bear the dean of computer science at cmu with um there's a there's a a very common machine learning task called image segmentation and so what image some image segmentation is is given an image uh how do i identify which pixels belong to which object in that image so there might be a road and a car and a tree and a mountain in the background pixel by pixel in that image can i classify what object that that pixel is part of and if if you go into the academic literature and look at image segmentation papers the metric they use is is overall accuracy and so how accurate am i on a pixel by pixel basis now if you think about that in most real world applications this probably isn't required it's probably not required to have every single pixel perfectly accurate and so this is going beyond that accuracy measure and thinking about can i trade off computational resources can i trade off robustness can i trade off accuracy to build the mission capability or the operational capability that i'm most interested in i can add one thing onto that which is i think this is where our work in ai engineering has a lot to learn from other engineering disciplines as well because it's about how do you make smart design decisions and how do you track the impact of those decisions over time right now a colleague of ours jay palette he likes to use mary shaw's definition of discipline growth and says that right now a lot of ai's in the craft phase we are trying things out we're doing a lot of customization to get to that mission outcome that business outcome but we're not at the place of standardized repeatable processes you know a developer can run their their program multiple times and get different answers and it's a question of why does that happen and i think to get to that place where we are understanding what the trade-offs are and how to navigate them it's going to take us as a discipline you know some continued reflection around those those intentions of what are we trying to do why are we trying to do it what's the context in which we are making this trade-off when does it make sense to prioritize one design consideration over another and starting to share across those so there's lessons to be learned from the discipline you know i think that's a focus of a lot of our work at the sei is learning from others and trying to bring together those lessons learned and help people unpack the the choices they made whether implicitly or explicitly and i think especially in this craft phase a lot of times the choices we're making are more unconscious we're just moving through or hacking it together to get it to work and it's a it's a matter of refining that process and thinking about what was my motivation there why did that technique work why did i would make this decision and getting clearer on that so then it can be shared and repeated by others in the future great rachel let's stay with you for this one william sent in how are your founder foundational efforts at ai engineering leveraging leading edge research in the overarching field of socio-technical systems yeah i love that question thanks shane but i think you know historically so many engineering disciplines and computer science have been deeply technical and have wanted to wish away the the context parts those social parts of systems but you know i think matt and i are both big believers that when when you're trying to solve a problem it's typically a problem that a person has you know people have problems they are trying to make things happen in the world for a reason and so our technical systems need to be built with those end users in mind so we are leveraging a lot of uh previous work in socio-technical systems a lot on behavioral insights so why would someone how do we reduce the inertia for someone to make a decision how do we reduce friction for people to navigate and how to gain understanding what does explainability look like in this context what parts need to be explained you know we've talked a lot about how if i may not know how every single part of my car works but i trust that it will drive me from here to the grocery store and so what is enabling that trust and how do we build systems in a way that are are allowing people to have a similar level of trust that cannot if we just built the technology and didn't think about the people side of it we wouldn't be successful especially when working in high stakes applications like the work we do in national security so instead we we have to be drawing on the psychological elements the behavioral elements all of those pieces and going back to the early question on education that means we also need to be training our engineers to have what i call social technical fluency to be equally at home on the social side and the context side as they are on the deep technology side okay let's go from one from our live chat from nermeen asking i suppose and i'll leave this to either one of you so you can uh jump in she asked i suppose the ai architect is part of the ai engineering group so what's the difference between an ai architect and a business analyst i'm happy to take a a quick um uh attempt at this question um i i think the the ai architect serves as a translation uh between that business analyst right the business analyst presumably has deep knowledge of the domain uh where where a particular application might be operating a lot brings a lot of that context and and um knowledge about the the domain and the business operations uh and i think the business analyst and the ai architect work hand in hand um to understand how ai is going to affect the business or can affect the business and what are the right things to do and the right questions to be asking about the ai technologies that they might be trying to incorporate and so uh ai architect maybe brings a little bit more understanding of what's possible with ai business analyst brings a little bit more understanding of the business and and what is needed and and that combination right that that teaming of those two roles um i think gets us to the right questions uh that need to be asked in the designing and deployment of these types of technologies and the only thing i would push back on is that it could be two roles it could be a person who has purview into both spaces you know in this notion that no one has all the answers finding people who can live in both worlds who understand a bit on the architecture side also understand the business side that finding that that ground finding people who can cut across super important for teams right now especially because oftentimes people in those two roles speak different languages they have different terminology different nuanced terms and you need folks in the team who kind of can not necessarily that are expert at either but are comfortable flexing into the roles as as projects need them and i think increasingly we're going to see more of that and i heard someone coin the phrase multi-potent multi-potentialite of people who have these kind of multiple pathways or diverse backgrounds to be able to live into both roles so i really think about it as what are the jobs to be done to understand understand the system that you're trying to build what all needs go into it versus kind of saying we need pers a person to fit specifically into that one role great thank you for that we're gonna go to matt on this one john had sent in since most deployments fail however what can then work to define the a.i engineering discipline offer now or in the near term for ai systems now in deployment that can prevent them from failing in deployment let me know if you need me to re-read that matt no i i've got a chain this is this is a critically important question um i presume i think you said the the question asker is his name john i presume john is familiar with this gartner study uh that's been um publicized and and and shared and through various venues recently uh about 85 percent of machine learning projects in industry uh never make it into production i don't i don't know that i would say fail but never make it into production uh so the important first part of trying to answer this question is is to talk about why why we think uh they might fail and and so i'll point to um some work uh that's based on the incident the ai incidence database which is a database of of ai failures from i think it's run by the partnership for ai or at least sponsored by the partnership for ai and the center for security and emerging technologies at georgetown just recently did a study of that database and the conclusion of that study was a categorization of the failures and why these systems fail and there were three core failures uh core failure classes uh the first is specification so um the the the desired intent uh the desired mission application or business application wasn't specified uh correctly or specified in in the appropriate context and so maybe the the ai system that was built didn't actually solve the problem that the teams were trying to solve the second is robustness and that gets at testing um maybe the system worked in the in in the lab environment in the development environment where it was being built but as it gets integrated into a larger system and then put out into the wild to operate the testing processes weren't robust enough to capture all of the use cases and edge cases that the system might be faced with and then the last one is assurance um and and and the assurance there is once a system is deployed how do we know that it's continuing to operate the way we want it to be operating and it turns out that environments shift and data sets shift and district you know to use a statistical term distribution shift over time and that affects how these systems behave and so once you deploy a system the context in which it's operating can actually change like um think weather conditions right if i've got a a machine learning system that can classes that can see cars on the road um and weather changes right rainy conditions snowy conditions the road goes from usually a darker color to a lighter color when it's snowing that's an environmental shift that can have an effect on these types of systems so specification robustness and assurance and then i think those are the three things that there are recent developments on how to better capture uh the the application domain the actual intent uh that we want these systems to operate and that gets to what i've already talked about and beyond accuracy thinking beyond just how accurate can i build a model but how am i going to deploy this model what are the types of questions that are most important that it does well on where can it not do it as well and where where is that trade-off space so that's for specification there's lots of work going on in testing and there's an increasing understanding of how these systems can fail on their own without intervention but also how these systems might be manipulated which is an interesting topic all in itself that would maybe require an entire uh another webinar like this um uh and then insurance assurance uh there there are there's there's a growing focus both in the in sort of the machine learning science community uh as well as the machine learning practitioner community about monitoring these systems and and how to calibrate these systems appropriately so that the systems themselves can actually know when they don't know something or know when they've stopped being able to perform as well as they can and so there's a any number of a variety of techniques and and emerging tools that can be applied to those types of problems one thing i would would add there is just that you know matt and i are big fans of learning from failures not just casting them off of ah this didn't work but being intentional to look at well why why did they fail what was at the root of that and of course the different areas that matt just laid out give lenses into it but i think oftentimes it's super nuanced you know i think there's a lot in the news right now about people pulling investments out from ai applications in healthcare and i think the the articles leave a generic answer at times saying well they just didn't work doctors didn't adopt them but there's probably a lot more to it and i think it's a place where i would love to see the community keep doing work to understand that understand what actually happened in those situations and unpack it a little bit you know we uh have a just a quick plug we are doing a symposium for triple ai which is one of the leading ai conferences and we would love to see people submit some case studies on failures trying to understand what went wrong and why did those things go wrong and the flip side of that is finding awkward case studies where successes happened and understanding what enabled that success in the spirit of amplifying positive deviance you know was it just that the model was so fantastic was it that you had achieved that notion of a very specific use case you had assurance and you had robustness or what else actually allowed that to happen was it the team was it the problem was it the context was it some combination i think it's a space where you know our focus on implementation we need that we need more people looking at what actually goes on and what are those enabling circumstances enabling factors because to the earlier points on socio-technical systems the answer is going to lie in both of those categories and i think that's one of the main drivers that will move this forward is being able to articulate those and make them more bring them more front of mind okay another question for our chat and i think it's related and really the reason why we're here today and we'll stay with you for this one rachel peter wants to know ai has brought a dynamic black box to software how can we guarantee quality security and safety sure so you know i think uh one i would challenge the notion that you need opaque or black box models we're trying to move away from that language but that you need your system has to include an opaque model there's a great article from mit that looked at um when there are more transparent models being used how they have become more explainable more adoptable um and more understandable in that context but i think guaranteeing safety is a false paradigm perhaps to aspire to and this is just my opinion we we want to have trust but you know my example of driving to the grocery store there's no guarantee that i will make it to the grocery store in my car my brother always says when i say fly safe he reminds me the most dangerous part is getting in the lift to get to the airport and because there's just there's risk involved and i think that's the part where we have to better understand what's our tolerance of risk when will we use it and i think challenge the assumptions about how systems are being built today because there's a reason huge uptake hasn't happened in all use cases it means there's more to work on with the technology and how we're developing it how we're designing it it is certainly true that there's more that a lot of what happens in ai is opaque we don't understand why exactly it works the engineer in me views that as a challenge of how do we get better how do we grow our understanding when is that true when is that not true to kind of explore it a little bit more matt i don't know if you have anything to add there well rachel i think i think your focus on risk and and understanding the the risk framework uh in which and the risk context in which these systems are being deployed and used is critically important and like the engineering qualities right trade-offs between scalability and robustness there are trade-offs that you can you can take on in in the risk context as well and so i think risk risk thinking risk analysis is a big piece of understanding how these systems can be deployed um it's also important to note um that uh so i wanted to mention that there is work in in formal methods right so if presumed some of the folks out there because we're the software engineering institute are very familiar with software engineering ideas i don't want to get into explaining really what formal methods is but formal methods is a method for making guarantees about systems there is ongoing research and recent breakthroughs on applying formal methods thinking formal verification uh two deep learning systems two neural networks to make some guarantees about the boundaries of where those systems might uh go out of their operational domain and so that that's pretty promising research still on the science side of things from an engineering perspective and again channeling our software engineering routes um in traditional software engineering you have a lot of tools for introspecting analyzing unpacking the behavior of a software system these are tools like static analyzers that actually go through the code and analyze it for failure modes for bugs for violations of secure coding principles those sorts of things we do a lot of work in in those areas at the software engineering institute there's also dynamic analyzers which are the sort of runtime analysis of systems and then even reverse engineering tools and i think there's a great need and this is an area we've identified as critical to ai engineering and some work from the explainable ai and interpretable ai communities on how do we build those same sorts of tools for these models and like rachel said i think it's important maybe to get away from this idea of black box or even that language but just there's a model there and how do i understand what it's doing and how do i analyze it and how do i test it on its boundary conditions and how do i find those those failures or bug modes uh and so there's there's again interesting developments in this field there's some tools out there already that you can find um in in open source or in the public domain for starting to get at the analysis and understanding of these types of models the other thing i would add is just that i think in this day and age we have to continue planning for uncertainty you know certainly covet 19 through everybody a curveball and helping us you know and making force this to look at the uncertainty the unmodded phenomena that exists in the world and that we can't anticipate everything that's going to happen and build them into our models i think it's hard you know there's a tendency for us to want to plan for and control all possible scenarios and stepping back to say things are definitely going to emerge that we don't know what to deal with there are definitely factors that will emerge that will compromise their systems that will make them less robust how do we start to plan for that how do we build that resilience into the design so that we can tolerate these system shocks in a different way that versus hoping and crossing their fingers that we've thought of everything that could potentially happen okay graham i'm going to try to combine two questions from our our live chat i'll start with you matt but feel free to kick it over to rachel um first one was do you have recommendations around the strategy to formulate an ai center of excellence for a large enterprise that's the first question and then the second somewhat related is there a template that exists for doing ai maturity assessments uh yeah so i'll make just a few comments here um we're being a little bit careful about maturity models and maturity assessments at this point because the space is uh large and complex um but we do have some sense of technology readiness and and things that you can be doing to to assure those things uh to assure these systems uh and then i think the first part of the question which is more about organizations rachel was just just published an article or a blog post on on ai readiness for organizations so i think i think rachel's best to talk about that absolutely yeah i think there's a lot to be learned from other centers of excellence you know i think over the past decade we saw innovation centers within enterprises emerge quite strongly and a lot of those weren't successful where people weren't satisfied with their investment because they didn't have a clear intent of what they were trying to do with it or how it was going to connect to their organization they didn't have the reporting structures in place to allow idea flow to emerge oftentimes those innovation centers became separate from organizations there was kind of the real business and then the people who were exploring things over here and i think there's that's something to pay attention to you know when you're thinking about ai centers of excellence you know is it that you have a separate center in the organization or do you have one person from each team that is kind of both has a foothold in a specific function of the organization but also is exploring where and how ai can connect having those different dialogues and ensuring that there's communication flow across the organization and starting small you know i think that's the part of right now there's the failure rate is is high and it's kind of your classic challenge with prototyping where you put a huge investment in and then it doesn't pan out in the way that you think it was it would and it affects the bottom line and it affects people's perception of this technology that you're trying to bring into the system and so starting to think about what areas in my organization are are ripe for ai where do problems exist that we're trying to solve is there an experiment that we can start to do in the next six months with a small team versus you know and also i the the first question of being where do we have data in the organization that we currently can use in new ways that's a critical first question and i think one that people say oh we'll find the data but it's the start the main starting point that you have to have and then it's a non-trivial lift to pull together that data to you know really kick start everything else that you're trying to do so some level of experimentation some level of thinking about how does this fit into the broader organizational strategy what behaviors am i wanting to incentivize is it get ai rolled out at all costs probably not it's really about how are you leveraging ai smartly to drive outcomes so how does it affect connect to the organization's strategic goals uh if forever whoever in the chat asks that question we also have public office hours that shane can link to at some point or we'll put into the chat and i'm happy to to bounce those around more and different ideas you're thinking about if you want to utilize those office hours they're open to anyone we're happy to talk about partnership opportunities or work that's going on in your small business or large business or whatever that or your research ideas your case studies it's been an experiment that we've been running for i think six months now and has been really interesting to see who we connect with everyone from educators who are starting to think about what does it mean to develop ai engineering curriculum to small businesses that are struggling with resource constraints to large organizations that don't know where to start or have some great leads and want research to support what they're doing okay great answer thank you for that rachel let's go back to matt for this one kim has sent in what a.i engineering efforts do you find particularly promising uh well we've talked about um we've talked about some of them uh you know i think i think the the expansion of testing beyond just evaluating model accuracy is really important and and we're seeing more and more of that um even from a security perspective right we've i mentioned earlier that uh machine learning modern machine learning systems in particular um have an interesting set of security concerns around how they might be manipulated in certain ways and this can be you know they can learn the wrong thing at training they can do the wrong thing at inference or they can possibly reveal information that you don't want them to reveal and there are tools now for understanding those types of security policies in a particular context and testing for those and so i think the focus on testing not just that model evaluation but at system integration and then into into operations and monitoring is is one of those those big areas um and then i i'll i'll sort of make the meta observation uh i've noticed and it not just because of you know the software engineering institute and our perspective on the importance of ai engineering but out there in the wild right as i go to different venues and conferences and um have conversations with folks uh the shift from the the maybe you would call it hype around ai right the the craze around the ai and what ai can do to to transform business and uh and and and all kinds of applications across a a broad variety of of spaces and domains the shift in the narrative is now well how do we do this right how do we do this in a responsible and reliable way and i think that as a development right starting to shift that narrative and think about you know there is there is this amazing power here but what are the right questions we need to ask how do we build this in a way that we can know it's operating the way we want it to and how do we monitor it and and have it give information to its user community about how it's making decisions and how it's how it's continuing to operate that narrative shift from a meta perspective is is is quite promising uh from uh from my personal perspective i want to agree with matt and amplify just the nature of conversations that are evolving right now i think we're seeing a lot of conversations around the power dynamics that ai systems create the values that are embedded explicitly or implicitly into systems who is involved in making these systems and who does it affect and what is the disconnect between those two things i certainly have a lot of excitement that that those conversations are are growing rapidly and i hope that they are you know it's not just our echo chamber that those conversations are happening in that they're getting across but i have a strong level of belief that that's that's the case and we're going to see that more and more in in the coming years the other thing i find really promising are the tools that have gained a lot of traction that have helped with these questions of transparency and explainability so the classic one is data sets or data sheets for data sets and model cards from model reporting those are small behavioral interventions of having folks document where are you getting your data from what are the attributes of this data source why did you choose it for these applications and instantiating those processes making them establishing them as part of the development pipeline i think is super critical for transparency and repeatability all of those aspects that we are seeing i think more and more those types of small different you know they're high leverage point opportunities it's a small change that can lead to a lot of impact i think that more and more is what we're going to see as companies figure out how do you navigate uh these different situations that we're in and how do we build towards more rigorous process on the creation of these systems there's a there's a really cool example that i want to mention here because i think it ties all of this part of the conversation together just yesterday i learned about about this idea um uh in in traditional software engineering and this has been done all over industry and government um there are these things called bug bounties right can you find a bug in my software capability my software system uh and companies will pay will pay you lots of money to find a bug in in their in their technology it's a way of sort of open sourcing um the testing uh of their systems just yesterday i learned about a challenge that was put on by twitter um they called a bias bounty challenge and what they so twitter has a capability and i don't know a lot of the details i'm not an expert on how these systems work within twitter um but they have an algorithm a machine learning algorithm that identifies the the key part of an image that's put into a tweet and they auto crop the image based on that and they put that technology out on the web and said hey researchers people of people that are interested in this problem anyone out there that can that can help us find bias in in how this algorithm is working and how this machine learning algorithm is is doing what it's doing and they got a lot of great results uh and many results that they didn't expect to see many types of bias that they didn't they couldn't think of as themselves and you can search for that out there i don't have a link ready for you shane to put into the chat but um just a really great example of bringing engineering concepts like bug bounties and traditional software systems to this machine learning world uh and bringing you know taking a sort of large tent approach to understanding how to um explore analyze test uh these types of systems and also that's a great example of planning for unintended consequences as it would have been easy for kind of twitter to feel confident say we don't have any bias but they'd be lying to themselves and instead they said we're sure there's bias in here even after all the work that we've done internally to make these systems why don't we put it out there and allow ourselves that vulnerability of sharing and putting that out there because ultimately it'll make their work stronger on the other side of allowing that exploration to happen from their user community and i think the other the piece that matt talking about that made me think of is we're hearing also more and more of organizations that are being proactive to have the hold kind of unintended consequences workshops to engage their kind of user community to think through potential scenarios potential use cases and all of that work is quite powerful and quite important even if it means that at this stage hey we need to pump the brakes on this project because there's too much risk involved or there's way too much potential for uh harm to be caused by the work that we're doing i think companies are getting more comfortable with that with that action step or at least i hope so maybe it's just the idealist in me but i think that it's a place where overall organizations are doing more critical thinking about about all of these topics and that's super necessary and i hope it continues with time great we'll work in a couple questions from our chat here another question from martin asking for tech companies that practice scrum parentheses agile methodologies how do the research aspects of ai engineering fit into this kind of workflows yeah i'm happy to start with that one and then that you can jump in you know i think oftentimes when you are in any sort of process flow especially you know when you're in design build test loops it's great that they're iterative and from that's point testing is completely necessary and something that we want to see more of testing being done early being done often focused on what are the riskiest assumptions embedded in this idea and how do we test for them and it requires that these teams also step back out of that process from time to time and think about the broader system in which their work exists i think oftentimes that's the step that agile specifically leaves out of popping back up to say wait are we is our work even still aligned with the mission outcome how has that mission outcome shifted in the past two weeks two months six months and what does that mean for our process and i think that there's a hope at times on development sides that somebody else will figure out that problem that they i just need to be told what to do but i think more and more we're seeing that there's an individual responsibility to pop your head up and ask those broader questions and be comfortable with hearing the answers of hey things have changed and i need to shift my design and development process at a given time and i'll say that you know in my world of of spanning different engineering disciplines this is not unique to software there was a long-standing analogy in the manufacturing space that you know the manufacturers would be working on the line and it was up to the design team to throw over the wall the plans and the manufacturers would just make sure it got built and over time that field saw that that breakdown and recognized there needed to be more integration from you know it's really the s specificity part that matt had mentioned earlier of specifying what are we trying to do with how are we going to build it and creating a tighter connection between those two things and so i think more and more we're going to see communication across boundaries process adjustments to ensure that we are doing that behavior of popping our heads up asking why and ensuring that we're aligned with the mission outcomes and doing that frequently so that you're not in this space of building towards an outcome that's not going to work out i might just add a few things i certainly agree with uh rachel's perspective on this um and and maybe i'll come down into the the details of of the processes right so in agile and scrum those are oftentimes associated with devops um activities and i think when when organizations are considering ai and specifically machine learning or deep learning capabilities inside of their development process agile and scrum it does introduce new um challenges uh to the overall process right um some people talk about data ops and how to create the data sets that drive the capability of these systems and that is an entire process on itself how do i how do i accumulate the data i need um how do i know it's it's the right data how do i know that it's not riddled with all kinds of challenges uh and things like bias um how do i version it over time how do i age off data i don't want there's also ml ops which is sort of a sub component of a larger devops cycle if you're building a system that includes machine learning and and sometimes model training can take days or weeks which which has an impact on your development cycle time and so there are interesting ways that i think these ai technologies are pressing on the processes um that exist within organizations and will require them to adapt and evolve but i think to rachel's earlier points these incremental iterative approaches where we experiment with things and try them out and make sure we're asking the right questions again in an iterative fashion are really really important and the last thing i'll add to that just because it's something that has come up a lot in our work is also that this is the maintenance of these systems is a critical part that we often find organizations are underestimating what maintenance looks like and especially what does it mean when there's context shift when there's data shifts how do you track those things it's both that you know the architecture is evolving at the same time the context is evolving and that's a difference from traditional software development so it's a place that there needs to be thought organizationally about how are you guiding these processes guiding the behaviors you want to see across teams and i think we're going to continue to see evolution there's all sorts of combinations of words right now of devops devsecops devsec f-ops all sorts of permutations and i think we're going to keep seeing new processes evolve as organizations try them out and figure out how do they support this these development processes over time okay we'll get a question from dallas in the chat matt we'll start with you on this one are you aware of any projects uh any are there any projects that work for poverty alleviation or pensions through ai day trading i'm not aware of anything specifically in in on that problem um but i i often think if you can imagine it there's probably someone someone out there working on on something like that so i i don't have anything off the top of my head but um searching around you'll probably find something similar and there's a path to follow there okay and then we'll go back to john john sent in another question earlier asking um ai will impact on software design will have impact on software design development and evolution now what ways do you see the role of the software engineer changing as a result we can start with you on this one matt well there's there's two interesting sides to this question there's sort of the what does the software engineer do about ai components in in their systems but there's also how does ai help and enable the software engineer so the first one um i'll i'll leave it with an easy answer which is software engineers most likely given the proliferation of these types of technologies and the commoditization of machine learning capabilities and other ai techniques software engineers are i think necessarily going to be forced to learn about them understand how to use use these types of tools and technologies in their process but maybe the more interesting side for me uh and an interesting question that the software engineering institute is is is doing a lot of research on is how can ai how does ai impact the the role of the software engineer uh and how can it really help or augment what software engineers do and so people out there might be familiar with a a new capability called codex it was jointly developed by openai and microsoft and it's one of these very large uh so-called transformer machine learning models right uh i won't get into the details of how they work but they essentially read lots and lots and lots of data lots and lots and lots of text and in the case of codex all of i think all of the open source code that's available on github and what codecs can do is from a natural language description generate small snippets of code and and it's really interesting to watch the demo you can search for this out there and and watch openai put on a demo of codex by the way i'm not endorsing the product i just think it's an interesting use case out there in the world um but you can add essentially as a software developer what you might put as a comment and then codex will automatically fill in the code that's there and it's not perfect by any means today but it's pretty powerful the demo includes the creation of a web server and a website it includes the creation of a game uh yesterday i heard someone talk about you know i i could describe uh write write me some code that shows snow falling on a back on a black background or whatever it is and and codex is capable of creating that um automatically and so i think um even in software engineering right it's sort of software engineers building capability that then software engineers can use there are real interesting questions about human machine interaction here and how software developers can rely on but maybe not totally trust or 100 trust these capabilities and really interesting system design concepts and code design concepts that are at this intersection of what can be auto-generated is that easier or harder for than a software developer to go and fix and analyze and test and integrate into into other capabilities and so there are amazing ai capabilities out there today um that are already working on sort of automatic generation of code right today it's a small scale you know five ten twenty lines of code um but i think there's a there's potential for these things to go much much further rachel anything to add there just that i think it speaks to other conversations we've had earlier in this that there the role is going to change and it's why that that learning is is needed and that curiosity is needed about how things are changing and i think with any change we as humans we get scared and we worry that this is going to be detrimental for us it's going to change all organizations and and certainly that we're going to see roles eliminated but we're also going to see a lot of new types of roles being created you know the the tools that matt is talking about that's effectively lowering the barrier for who can become a software engineer and then allow software engineers to ask interesting questions to expand their skills to apply them to new areas i think that there's certainly two ways to look at things i tend to be more of an optimist but a realistic optimist at times but i think it's a part of paying attention and just being willing and open to how these systems will change how that will affect roles and i think we're going to see organizations grapple with that over the coming years and i for one am excited to see how it emerges okay we're down to about five minutes left so i'm gonna combine two questions from our chat um let you guys give a response to that and then we'll go to closing thoughts for each of you before we wrap up and then lastly i'll just mention lots of great questions to chat if we didn't get to your questions in the chat or that came in via email we'll make sure we give them the matt rachel to and there's maybe we have other ways to respond uh post event but uh the questions we're gonna combine are from annie and phillip annie asked is there any effort at cmu underway to establish a degree or certification for ai engineering and then phillips question was what is the path to become an ai engineer what skills do we need to have so we can can we start with you on that one rachel yeah i love this question so there is already a degree at cmu or a couple degrees that are in this flavor so there's one that is actually at cmu's africa campus in rwanda kigali rwanda and so that program is a masters in engineering of ai systems it is in its first iteration right now and so it is about really tying this core curriculum in engineering systems engineering ai systems with domain knowledge of what does it mean to apply ai in uh agriculture in manufacturing and how do you balance context knowledge with this kind of deep technical knowledge there are plans in the works for other programs especially at multiple levels at the undergraduate level at the graduate level as well as for executive education it's a place that we're definitely interested in helping to evolve and are working with campus right now to think about what those options look like so it's a long way of saying stay tuned and things are already happening at the same same time in terms of skill sets you know i think there's a lot of debate around that right now so this is not an sei viewpoint but it's mine i think that being able to be you know have this technical grounding have a strong foundation across the ai stack that matt described earlier is going to be critical as well as being able to hold the complexity of the problem spaces i think more and more that's the skill that's going to be needed you know as an educator my least favorite question is when students come to me and say tell me exactly what i need to do to get an a on this test and that mindset is not going to fly as we think about how to implement these systems instead there's a lot of uncertainty a lot of unknowns being willing able to wade into that complexity to break it down understand what are the component parts what does that mean for me and my design decisions that that is going to be key and it's it's really comes down to this social technical fluency that i talked about earlier i think more and more as the tools continue to evolve the technical barriers are lowered we're going to see an increasing importance of that that fluency or you're closing part yeah i'll do both shane i'll add i'll add that there is there is sort of um interesting thinking along the narrow path of how to become the ai engineer and i think i think rachel talked a little bit about that but i think there's also an interesting um phenomena that i'm observing which is there's there's a great opportunity um to to add ai education uh into the core curriculum across a broad variety of of disciplines of majors of courses of study much like you know 10 15 years ago we a lot of people started talking or maybe 20 years ago started talking about computational thinking um with the rise of of software systems and the influence of software in the world i think we're seeing the same thing with ai now right no matter what you're studying it it's likely to be of interest to understand how ai works from a science and theory perspective but then also how to practically apply ai in whatever domain you you might be working in civil engineering medical healthcare public policy all of these places have opportunities for these technologies to to have a very positive effect and and that brings us back to where rachel started with a diversity of perspectives is critically important to get this right lots of different people with different experiences and different training need to be a part of this my final thought shane um obviously we're really excited about ai engineering here at the software engineering institute we're excited that the narrative is shifting uh not just around the excitement around ai itself and what ai can do but how to build these systems in a responsible and reliable way and our approach is um we want to be a part of it we want to be a part of creating an ai engineering discipline and an ai engineering movement but of course we can't do it alone we don't pretend to have all the answers we're very much taking a community-based approach and so we we look forward to engaging you know folks on this on this webinar today in any number of ways into the future whether it's education or understanding what things have failed or worked in in your particular application space in any number of ways so just wanted to emphasize community-based approach we're going to be doing a lot of things like this and a lot of other ways to engage with that broader community and excited to see people participating rachel final word for you yeah just for me one just can't help myself on the pathway to become an ml engineer i'll just say that no one has this solved yet we did a really interesting study talking to different organizations about how they were hiring for ai engineers ml engineers and there's a lot of open questions and a lot of it comes down to how do we train internal people to develop these skills and hold them in parallel with kind of the work they were doing previously so if anybody has more questions on that happy to have a follow-up on it i want to echo everything matt said about our community-based approach we feel very strongly that we can't do this alone and so if there are things you want to contribute resources you know of tools you use successfully case studies that would be worth us to know positive or negative please do reach out i know shane put a couple of ways in the chat to connect with us we'd love to hear from you and are continuing to think about how we intentionally move forward and bring a diversity of partners into this effort in the same spirit of having many perspectives on what does it mean to actually engineer these systems and how do we do them in a way that is responsible great rachel matt great discussion today thank you very much for sharing your expertise the last machine yes and lastly we'd like to thank each and every one of you for attending today uh just a quick note the sei is a strategic research partner this year for the 2021 ai world government the show takes place october 18th and 19th in this two-day forum educates federal agency leaders on proven strategies and tactics to deploy ai cognitive technologies for more information see www.aiworldgov.com [Music] upon exiting today please hit the like button below and share the archive if you found value today also you can subscribe to our youtube channel by clicking on the sei seal in the lower right corner of the video window lastly you can join us for our next live stream which will be monday september 27th and the topic will be the future of ai scaling ai through engineering uh registration information is available in the chat now it's also on the sei website and we'll follow up with an email as well any questions from today's event you can send to info sei.cmu.edu [Music] thanks everyone have a great day you

Channel: Software Engineering Institute | Carnegie Mellon University

Views: 791

Rating: 5 out of 5

Keywords: AI, Artificial Intelligence, Machine Learning, AI Engineering, ML

Id: onq5aUcDI-E

Length: 64min 46sec (3886 seconds)

Published: Wed Sep 22 2021