Move fast without breaking the bank: AI model risk management

[Music] hi everyone again to our evening session uh and i'm happy and proud to uh welcome cern nice busy have you here thank you very much thanks it's just actually near christmas time so that you spend the evening with us that's just awesome and you have a family as you said with four kids waiting for you yeah exactly fantastic just have to be patient that's good that's good so i i think we have an interesting subject and uh but uh let me let me help you to introduce you a little bit uh to the audience of course sir you are located in denmark yes copenhagen yes in copenhagen i would say i would say nightlife because nightlife okay we have a little bit of a problem with a crisis at the moment exactly let's say beautiful city what are you doing in your daily daily job so data job is um directive in uh ernst young um so i'm heading our ai risk team in copenhagen where we are helping clients uh with navigating the treacherously uh dangerous world of ai uh the usual world of ai okay i think you hear something like a thriller story this evening right yeah yeah so it's um yeah with applications in the financial sector so um that's my main line of business and uh something i've been doing for for quite some time um but i've only recently joined um estonia i used to work in full bank doing the same kind of activities um but now i have moved into consulting to help a broader range of banks and other financial sector companies yeah so uh why do you think risk is so sexy that you spend the whole working time on it so i just uh try to do what i can to help and i think it's topical and relevant and actually a um a particular perspective from the financial industry um which might be very relevant to to other parties um so we have a long history a pedigree of of managing risks in the financial sector it might be a benefit to other people um yeah well what are your from a background what's your background i'm um actually i used to be a molecular and evolutionary biologist a long time ago um so i moved into computational biology um did my phd in ai and machine learning with applications to bioinformatics i've been a researcher at university um studying models um [Music] disease interactions uh various types of interference analysis um i've done a brief stint as a algo trader so i rode programs that traded in um in fx so billions not buildings maybe just millions but still a lot of money on my laptop for a small startup um and then i've i've worked in in the financial sector um for for bank doing uh model validation um to start off with primarily for pricing models so various types of stochastic differential equations used for exotic derivatives um but then also over time more generally model risk governance writing policies are rolling out the principles of model risk management out to new areas and then for the past three years establishing dedicated ai model validation team looking at model risk of ai and machine learning applications so that's quite quite interesting so a nice nice collection of expertise to be between us yeah did you ever experience looking at the topic of today risk and looking at your background did you have ever had uh sleepless nights or uh nightmares because of risk and ai a few of you okay but but that's that's that's how it is um i think you learn to live with it over time um but there are there are a few potential calamities out there um i've also had um sort of been sharing roundtables with leading banks in europe and interacted with regulators and supervisors on the topic um and sort of collaborated across the globe actually with people and now that i'm in in in a very big uh company uh consulting um other companies across the globe um that has brought a sort of broader perspective to that but there are some concerns um for sure but uh as you will see when uh when when i go through my my presentations awesome i mean uh it's it's actually a perfect timing for for me uh to be honest because i have to read a lot about risk management at the moment i have to no it's more like i do that because it's quite interesting at the moment i'm concerned about the black swan i'm sure you probably heard of that nothing stuff and just uh nothing is predictable so that's one of the messages i got yeah yeah forget about that those things which are just yeah with the big big events and cause much trouble that they can't be ticked from um so i'm really curious about to listen to what you have to tell us today and so say take the floor as yours yeah thank you i'll just um share my presentation here so one thing before we start you can prepare your presentation uh for everyone who would just join um remember how that works here so we have a q and a section if you look in the navigation bar you can find the cool day you can always ask questions during the presentation uh siren can absolutely focus on what he wants to tell us and share his expertise here i'm gonna take care about your questions and i will ask all the questions when we come up and it makes sense or at the end but usually i ask you questions during the presentation sorry so that we can really put up so there are no silly questions this is the one thing i can say don't fear punishment or something like that or laughter nothing happens right uh just ask the questions here you think are just painful for you and say i i need to know that i'm curious and yes so we are here to answer seven no it's up to you thank you so much um so my the title of my my presentation is move fast without breaking the bank which is of course a little bit of double play on uh cost reductions and uh potential um sort of issues for uh for the financial sector and and banks um in general um i do apologize uh up front for puns um even though this is a relatively serious uh matter um of course um and also the the concerns are sometimes a great sort of concern um there is a little bit of humor so i i i hope you also humor me and don't take that to um take that out badly um so the main definitely not so definitely because honestly if you've got more serious ideas for better humors right so go ahead so so the main main topic is ai risk um and particularly with the outset of model risk management is used in the financial sector um as applied to ai so perhaps before i start i need to stress so when people say ai risk management there's two meanings of it one is the use of ai4 risk management and the other one is risk management of ai and of course we use ai to do risk management of ai um but the main subject here is risk management of ai not as much the use of ai for risk management so um the first thing for some reason it doesn't change so here we go there you go first thing i i would like to talk about is the perspective so of course managing risks is a crucial essential part of developing ai machine learning applications and using ai machine learning applications and there's a lot of focus on that most of the heat currently in the general field of ai and machine learning is on the development side and the use side finding new applications finding new techniques um getting sort of accelerated uh activities uh across the line in terms of developing platforms that can help people out with ml ups and all those things but also just finding new applications the other perspective to risk management of ai is the control component the financial sector is and especially banking is all about control so governance um audit trails making sure that uh the money is there now and also uh tomorrow so people can buy houses get loans pay for food those kind of things go on with their daily lives that's the main sort of at the core of uh banking in particular so the control component is very important to the financial sector which is also the main line of um [Music] work that i'm going to present for for the rest of the presentation is going to focus on the control perspective on ai and that also is just this for development you can go from sort of frameworks platforms um sort of full enterprise solutions down to individual application the control elements also run the full spectrum from frameworks how you set up the governance to what you need to do for the individual applications most of what i'm going to talk about is not going to be the itty gritty details the gory details of particular applications of aeon machine learning techniques to particular sort of problems but are going to be of a more general nature there will be details and also a little bit of call but but perhaps not as much as uh as some people might want to know about where how is this being used in the um in the financial sector but just more kind of overview also thinking about risk management in that specific industry what does really mean so what what kind of things we need to consider so something like that yeah yeah so just to give you a temperature reading on sort of the current status of a on machine learning um in the financial sector at large but also for banks is that there is a lot of activity going on right now so this rapid uptake of um of ai it's still early days um both in terms of the use of of these techniques and also the underlying technologies the applications are quite right ranging so from various fin crime prevention tasks to customer satisfaction analysis chatbots data masking texture and analysis model validation as well and also the techniques being used are also quite varied um so from symbol ish regression type um techniques to uh more sophisticated complicated setups like deep neural networks and reinforcement learning or causal inference so so the full spectrum and of course there's a lot of variation across the industry in terms of how many individual organizations are using which techniques for what applications and the maturity generally in the field but but we are seeing across the industry uptake and use of these these techniques and of course that will bring across the globe significant benefits so a lot of efficiency gains a lot of improved processes improved performance um but also naturally uh the flip side of the coin is that this also brings risk and um new types of risk yeah you have a quick question just questions are all out of curiosity because you have ai use cases yeah techniques do you have uh any any kind of insights about the trends so what is is there kind of some part of it specifically of interest at the moment in the industry um there's a lot of talk about the use of these techniques in credit and i'll i'll get into that a little bit later but um i think sort of overall trends is are really hard to to spot except that there is more widespread application than uh so so it's a the trend is that things are growing um and it's becoming more varied there's still a tendency to not use the most sophisticated and complicated techniques for high-risk applications yeah thankfully yeah um but otherwise uh no it's hard to spot a particular trend in this except growth and applications um across the board so it's more likely we need to do the things we have to but now it shouldn't be too risky right um there's some element of that but still also room for improvement on [Music] on that aspect okay so um in terms of risks um there are what we call model risks in the financial sector which is typically something which has been used as a particular risk discipline for managing complicated math dangerous math associated with pricing models so exotic derivatives pricing for market risk and counterparty credit risk and for credit risk which is also of course of paramount um importance to banks um [Music] these new types of techniques and users do bring a lot of operational uh risk in addition to the model risk as well um sort of other more manual types of errors and problems that can that can rise scores also compliance risk are we compliant with the regulation are we compliant with our internal rules are things done by the book um even in in these sort of complicated settings security risks as well associated with the use of sort of broader open data sources and the possibility of also due to complications from using sophisticated techniques also that there might be vulnerabilities in this that can be utilized by um by ill-minded adversaries risk related to broad regulation uh that might lead to fines to the to the institutions and of course um if you do dodgy things with your data um without the consent of your of your customers and clients rep severe uh reputational risks um of using these techniques and then also societal impact of uh of using these techniques in terms of uh making your workforce redundant um using a lot of uh you know co2 emissions from uh heavy training uh or using your server farms right um and these risks can if they materialize um so become losses of course uh the range from first impact of course is is loss of your investment in your ai and machine learning project because you fail um you might end up paying for your investment three times first from building it secondly from uh realizing that it doesn't work and then cleaning up after it and then thirdly for re-implementing a proper solution that actually works so it can be a costly um so even even putting return on investment into the negative territory so basically losing money on this of course depending on the application you can also suffer direct financial loss by these models or applications if they do not function as as intended you can lose revenue there is for grave severe mistakes the possibility that um supervisors will impose uh additional capital requirements um or fines even massive fines ultimately issues related to reputational risks if it becomes sufficiently severe you do run the risk of bank runs which is a complete loss of trust in the bank [Music] getting all the customers to withdraw their funds simultaneously and then in the event that the bank cannot fund this short term in the market that leads to capital and uh liquidity issues which is why we do you know have regulation in place in supervisors to make sure that this doesn't happen because that's a unfortunate sort of um property of money um as clever as it is and and as easy as it is from your day-to-day work that you don't need to go to some pharma and and barter your stochastic differential questions for corn um that money can disappear if you lose trust in money in banks then it evaporates basically you can look for your job yeah not only that but people cannot then pay for their houses cannot pay for food um and also jesus systemic risks this can spread across the globe and cause financial sort of system uh issues as we've seen in in the last financial crisis that was not triggered by ai machine learning but there's still a possibility that these kind of uses if they are important um and do not work as intended can potentially trigger the same kind of sort of worst case scenario and it's definitely something that people need to be aware of and that means of of course also that the license might be at stake for um for running a business as a bank if you do not act responsibly um and and mitigate and manage the risks associated with using sophisticated techniques for important tasks in terms of what causes these problems our experience is from interacting with people across the industry um and also generally yeah yes of course uh if you just go back um so one thing is because there was a question i would like to address also as uh immediately no one would ask um regarding the slide before the two slides before we're slightly showing everyone here um nlp has a natural language processing used as well i guess so yeah because this is i think one of the main application areas is just using it and for fine and for finance industry there's a lot of different applications of national language processing [Music] document tech information retrieval processing of financial records multiple uh use in chat bots as well of course uh responding to that so so multiple uses of natural language processing thanks that was the second question from laurent as well so you're asking for use cases now you named them right they go to the next slide that was the question i i would just ask as well this is a huge long list of risks right because it's just like probably yeah shall we really start to build a model that would be my next question so uh looking at a team and the teams actually working um operationally on building models like this is you should be a huge team right you have to have people um so a huge set of skills from the beginning yes um and experience um but this um but but this is not too different from what is the case for other types of dangerous math um so you have the exact same problem with um other types of models i think the widespread use and this sort of distributed use for aer machine learning does put you know um makes the problem worse in in many respects but i'll come back to that on the risk drivers actually uh for for my next slide but you know the the worst case scenario i thought just to go through these uh these risks is that you have a um you use an ai machine learning application that is fundamentally flawed it doesn't do what it says on the tin it breaks down constantly it's non-complied with your rules uh it's vulnerable to attack um and you get a fine um and you're also using customer information and acting in ways that aren't you know to the detriment of your of your customers and you have you have laid off quite a substantial amount of your workforce uh for this broken machine um that and and if that leads to not only loss of the investment uh but potentially fines or issues related to your license that's the kind of territory that we're trying to avoid and get out of um by uh by any means uh or by particular means actually so which i'm gonna go go through uh later on how we can not do those things yeah interesting but the problem is um so this is not particular sort of for necessarily restricted to the financial sector or only my experience um or of those of my colleagues generally there's a very high failure rate associated with most ai projects of course it depends on what service you're using and how they they ask for that but surprisingly many ai projects fail because it's not easy um i think people tend to underestimate how much effort you need to put into this just like any other thing that's complicated and important you need to put in the effort um and there is a tendency to underestimate it one of the problems is that i again across the board um i can't remember exactly where these statistics are from but um one survey showed that i think it was from keckle um that 40 of ar developers had less than two years experience which is quite substantial um and then when you go to people who are not developers so all the control functions your compliance functions your operational risk your legal departments your senior management your supervisors they have practically no experience with their machine learning um for a second um because this is quite interesting so forty percent of ai developers with less than two years of experience uh i found um i think it was from andrews and to nick's um deep learning newsletter uh just recently an interesting discussion about um my experience and lack of free experience on the point is and that was an interesting point that some of the techniques we use today are just so young but their experience is not a matter of issue because it's just young and so it's probably more that we have to label with projects more as an rnd project unless and just like you do something and know already what's happening yeah so so so it's very much the novelty um also just it's even if the techniques are some of them are quite old um that they actually work to the extent that they do if you do it properly is also known back when i did my phd um you know uh neural networks didn't of course they did work uh but they didn't work to the same extent that they do now and we've seen massive improvements very you know super compelling um massive breakthroughs in terms of using these techniques for playing poker folding proteins driving cars doing all sorts of wonderful things wonderful things that that weren't basically weren't just possible just a few years ago but there are some you know issues apart from that related to this which i think is is leads to these um potentials for using broken prototypes um and that that are poorly documented as well because the junior level of developers and not necessarily uh also people who like writing down what they do uh especially from the beginning um and and that's driven also by this you know data-driven approach um data science is a lot of data um perhaps not enough science um but but but still available of open source software of data that it doesn't take long for you know very sort of broadly speaking to to just take a course on coursera or similar sort of urban university something um get some open source software put it on your laptop get some data sets try out some different things and you know it compiles it might work you have your you know prediction performance being humped somehow and then you just go to production which is not how things should be done in a uh in a bank um [Music] so the ease of prototyping is driving some of this early uptake of uh immature undocumented models and there are also other issues related to inappropriate use of metrics building up technical debt and also this fear of missing out that your competitors are also driving the use of these techniques and that you're going to miss out on the opportunity of of not doing this and also a half you know a a [Music] severe cost focus because a lot of these you know that's the artificial intelligence is that to a lot of the applications are intelligent in the um in the form that they are used for tasks that was otherwise done by human people you know but by humans by ordinary people right uh not in artificial people um so so that means that you can save a lot by using machines rather than people it also means that managers think they know what people are doing because they're people themselves so they they might think they understand the tasks that are being conducted they might also think that they understand what people are doing even if they don't have actually the data proving that you know that that is actually what they're doing and the statistics don't necessarily work um to their advantage and then this general inexperience across the board is also a problem combined with a um you know the the general fear of missing out also of course is that these techniques are good they are powerful you can use your data in for things that you couldn't do before you can meet or exceed human performance on a lot of tasks and of course we need to do that [Music] but we also run the risk of putting things too early into uh production and underestimating seriously the investments that are needed to do this in a secure and safe way and especially in a way that is you know trusted ai but also trustworthy ai or even to you know to put it to the extreme you know fractional reserve banking trustworthy ai so uh it needs to be something that you can put into use in a bank with the consequences uh and managing the consequences that it might have um if it goes wrong just like a lot of the other complicated things that also happen um in a bank yeah and pretty much our term as well like so you said trustworthy or just trusted ai it a responsibility yeah yeah yeah yeah so it has many names but but the bank the worst case scenario would be that it's not fractional reserve uh banking worthy ai that it's it such undermines the trust in your institution that it's it poses an existential risk yeah sure of course that's the extreme end of you know fear-mongering a little bit here but uh but but that's the perspective that i think um this needs to be seen as well yeah because there's also discrepancy um there's a little bit of a discrepancy between the core risk functions and the awareness of what's going on and then uh from the developer side the awareness that they are working for a bank and it's not a startup company you can't just start over uh it has real consequences for uh for a lot of people okay if you pretend yes it has more consequences for more people but to be honest it's working in the startup uh it's uh it's also worth everything's happening yeah not the way expected uh what i like about that slice specifically is it's uh for me it's like a checklist i can use like a checklist list let's walk provo steps and see but how does it what does it mean in my context what is about technical depth what is the human task how how did we actually organize everything uh what experience do we have what kind of lack of experience do we have i think this is pretty helpful yeah um i'm a little bit worried about uh checklists in general i'll i'll i'll come back to that but of course there are some good elements of it but there's also tendency to when people are using checklists that they are not you know considering necessarily um you know the fundamental issues related to these things which is another thing that inexperience also brings the the overall perspective is sometimes lost um on people who don't have that mature uh view to what it takes and what we're doing and and it's not an easy task so it's not something that comes naturally from just a few years uh work uh in these areas it's typically something that that takes a lot of effort and a lot of mistakes that you need to learn from your own mistakes and other people's mistakes over over a longer period of time and that those people are just not very available it's really hard to come up to find someone who has many years of experience with ai and machine learning and also practical experience and then expert matter you know experience for the particular applications and the many nuances that are in as the previous slide showed that there are many uh sort of aspects that you need to cater for um for doing this maturely and responsibly yeah and that's that's tricky so um [Music] so so to summarize a little bit the types of um of risk here so so this i've just tried to to do a traffic light right in the green um the red one of course is that the ai or machine learning application just doesn't work which is um i think where we are for a lot of applications um and of course the the worst ones are those that don't fail quickly but that you use over a long period of time building up problems um and then you have to to clean up afterwards which is a problem and even in the event that ai and machine learning works there are still potential consequences to societal consequences of you know the impact on your workforce on uh employment opportunities on you know impact on the environment um and then lastly when a on machine learning works a little bit too well um you also have problems um and that's actually a lot of the concerns for um in the proposed eu ai act is of that last um so a lot of the prohibited items is ai machine learning that works a little bit too well so the the evil overlords that that knows you better than you do yourself and can exploit and utilize that um information against europe against your will so so there's risks in all elements of um sort of in the full spectrum of how well aeon machine learning works is the key take home so there's of course there's a lot of regulatory um now with the upcoming ai act which i think is very welcome um that uh someone else then myself are concerned about these things um so for a financial institution of course you need to be complied with the basic law regulation stuff the basel the crd the crr your gl11 supervisory uh stuff in your your internal models so your credit risk market risk pricing models um that they're compliant with these things but you also have gtbr concerns uh related to your use of of aei machine learning models of course in terms of generating data sets um potentially having those in the cloud or by third-party people so you're concentrating um privacy uh information for uh training data in ways that they that they aren't necessarily in the legacy systems in banks you have special regulatory requirements related to anti-money laundering and preventing financial crime which also might be impacted by the use of a and machine learning and then just recently eba the european banking authority released a discussion paper on the use of machine learning techniques for internal rating based so for credit risk um capital calculation of regulatory capital where the supervisors in the past have been very restrictive in permitting people to use these types of techniques so the mod credit risk models are more or less set in stone there's very little uh room for for maneuvering um compared to uh when you're building pricing models um and there's a lot of constraints in terms of how you need to benchmark and test and you know build your models and and also validate and test them um so i think what them what they're trying to do um ebay is to open up a little bit um for the use of um of these techniques within these capital calculations uh for for credit risk which is sort of softening up a little bit their their view on that but on the other side you know putting some i i expect some limits as well uh for the general use of a machine learning financial sector so not only this the the things coming out of the eu act uh which is a general um for all users not only for the financial sector um of of users of ai and machine learning where they have this risk-based approach of of course as well you know if you're doing complicated things for important science you need to to put the right efforts in um and then have some things which are just unacceptable which was the green uh sort of societal big brother type um activities that are outright banned but then you also have high risk um applications um where i think for now at least it's only explicitly mentioned the only reference to crd in the ai act is for credit scoring um financial institutions can also be subject to some of the other ones um sort of for recruitment um but but the the overall um what can you say the the main driver of the ai act is the individual rights of the citizens of eu of the european union and not as much these fundamental uh financial stability problems so as as relates to banks and the eu act as it is now is mostly concerned about whether you can get a loan or not and not as much concerned about whether you can buy food tomorrow or not because your money's gone i think that will change over time um so the definition of ai um is allocated to an appendix um in the act so potentially subject to change i do like the um their definition even the you know having to try to define what a machine learning is in in a financial setting it's it's super hard to do and there's a lot of grace zones and um of course we have problems with defining just what natural intelligence is it's got to be difficult to to define what artificial intelligence is but it's going to run the gamut from [Music] statistics rule-based and then something in the middle uh which is covered quite nicely by i think the definition used here especially when combined with the view on how are these techniques used so if they're used in a way that puts the citizens [Music] in danger or their rights in danger or might have a severe impact on them that will then be treated by this um by this age so i i think the list of uh naughty applications uh so to say that that is also in the appendix will grow over time and i also do expect um ebay to to issue guidelines as well um specifically for the financial sector on on the more broader use of of ai and machine learning outside of uh of the credit risk space um because i just see um questions at the moment regarding um for regulation if you go back one slide yeah yeah sure um who is managing such regulations in companies in general is a role for that so so for making sure that we're comply that the banks and insurance companies and pension companies are complied with with the regulation there's a horde of people doing just that so so that's typically done by the compliance functions or your legal team but but everyone else as well of course needs to abide by little yeah so so everyone is quite aware of of what the regulatory requirements are especially in the risk function uh but but throughout these organizations um so so yeah that's that's the short answer is everyone but of course concentrated in in what we refer to as the second line of defense um but but also uh of course there's uh audit functions internal audit functions uh making sure that all the controls are in place and um and and will make sure that the second line uh compliance risk functions um operational risk um are conducting their work as they should and also that the first line business um more direct business line is is doing what they need to do yeah i mean that's just also from a practical point of view just um for for you guys who are just joined us today thanks again for being part of our episode today um uh since i have been attending ebay startup academy recently right uh i was i was in the lucky situation to find out so much regulation but it was like scary it scared the hell out of me yes like okay when you just said okay we do we do something yeah usually instead we do something test something we try it and then you exactly find out but for something very you need to take care of because there are legal threats which you need to take care of and just coming back to the question again and you i think you partly answered it um but some companies do have the resources of bigger ones to build up stuff in that area uh i will start i wouldn't have a capacity to do well so we need to actually ask outside consultants to do the job right and to say can you help me can you check if you are the kind of legal um pitfalls might be on our way which we need to um know of which we are not aware even if you have domain expertise because you have been working with industry for some time you need an expert in legal issues as well right yeah yeah i i must say though that you know of course the impact of error or fault or problems are much bigger for large financial institutions than they are for most startups so if a startup fails people can buy food again tomorrow it doesn't spread across the globe to the same extent so of course the regulatory scrutiny and the requirements set on those institutions are very sort of uh you know severe for good reasons um but that doesn't mean that everyone needs to do all these checks and balances that everyone needs to live you know live in a bank vault for the entire life it'd be constrained to that but of course there's a lot of things that that other industries can learn from that uh perspective of managing risks of being i think he is careful and but it's also of course expensive it's interesting that you say that and you say that we're talking about the financial industry at the moment but of course this is just like there's some some general um general truth in it which can be used anytime and to see as you mentioned yeah yeah yeah we have the healthcare startup so healthcare is a similar uh difficult field because yeah or people's life this is about living on this life yeah you have of course some health care problems can also potentially propagate out to millions of people um but for most of them they have a more direct impact on individuals yeah than um than in the financial sector there's more of a statistic sort of a big game issue for the financial sector but but but that also does have some quite potential severe consequences for individuals and for a lot of individuals if it does go bad um but of course and and also that in terms of investing in risk management um it is expensive and also it's expensive to get people with experience and um and to invest appropriately um but it's also expensive to uh to not do it it's it's even more expensive actually to under invest in this area yeah and then suffer the consequences down the line so actually on that um this is sort of meant to illustrate where you know financial institutions are when you compare the level of ai adoption with the amount of ai control of course you have um some optimistic people with not too much ai adoption but a lot of ai control in in your top left corner to my experience they would definitely be outliers might even be liars because because i don't think that that is the case um basically anywhere um the the feedback i get across the industry is that there is actually much more use of ai than people think and unfortunately there's not enough ai control so there might be some especially big institutions they can afford and have done this potentially in in good time so so where you want to be is is in your top right corner right um so that your ai adoption the control that you have is commensurate with your ai adoption and you also don't want to be down in your bottom left corner and not use ai and miss out of all the good things um so so most people would want to be in in your top right corner but what we do see is that there's this variation and there's you know some distribution but a tendency to be perhaps down in your in your bottom right corner so of course what needs to happen um especially now with the uh with the upcoming regulation and and the increased scrutiny that will follow from that is that you can basically do two things you can move up which is sort of the ideal um or you can move left so basically scale down the use of ai and i think in reality what will happen is a little bit you know up until the left so so be less ambitious about rolling out this but to be more committed to to building high quality applications and also making sure that you have all the controls in place so as a key element and this is where the financial stuff comes comes into play so we actually have quite a lot of experience with controlling complicated sophisticated modeling techniques um for things where there's a great degree of uncertainty related to it so uh various so of course we don't have there were some issues in the last financial crisis related to exotic derivatives um but but but there's a whole discipline which is you know has has been used for decades in the financial sector especially the large financial institutions related to the use of sophisticated uh math and sophisticated computer models for important tasks which is this model risk element and a lot of this is sort of the discipline is based on this sr-1177 which is law in the u.s but is sort of industry best practice for the rest of the world before it became law it was also industry best practice in the u.s it was called the occ guidance on on model risk that the underpinning of that is in essence some element of peer review as known from uh universities and um sort of the field of science more generally or that you do something you write it down and you submit it to scrutiny by your peers before you have it accepted for publication the same fundamental approach here is is sort of set out in sr117 that you should have independent challenge of your models by suitably qualified people who can do this extended type of peer review where they take your model to pieces they you know hit the tires bump it uh subjected to stress look at all the uncertainties all the assumptions all the bits and pieces going in and write up um detailed reports that are then submitted to committees to prove that this is actually um fit for the you know fit for purpose is the essence of it and this use a definition of model risk which is basically just adverse consequences from decisions based on misused model outputs and reports which is fairly broad um and a definition of model which is basically just a quantitative method system approach that applies statistical blah blah blah blah so that takes some input and and turns it into output right and that can be anything but of course if you take a risk-based approach there must be some element of is it sufficiently sophisticated and complicated is it used for something sufficiently important to merit that you do this costly exercise of getting someone to spend months taking it to pieces analyzing it from various angles and making sure that it actually does what it says on the team so that's the underlying principle that i think to my mind is something which is somewhat unique to the financial sector that you actually have a dedicated team of rocket scientists in one corner of your organization to make sure that your other rocket scientists are not falling out of line and uh you know misleading management or even misleading themselves in terms of what it is the uh their math is doing um and of course the central um sort of category or label that we put on this when we do model validation is is to look at things on a very fundamental level uh whether the application is fit for purpose to make sure that there's alignment between the reality that's being modeled and the model itself so and and that can be you know of course something that happens over time that there's drift either in reality changing in ways that the model does not cater for or that the model drifts in ways that the reality cannot substantiate or that they are misaligned even from the beginning due to sort of fundamental misunderstandings of course all models are wrong they are not reality um and that's why we have them because they also have some uh you know features that reality don't have you can use them to forecast you can explain stuff with them uh but but ultimately it's a simplification of whatever reality you're trying to describe and you don't want them to be too far apart especially on on the important issues that's super complicated to find out and what to do so so i think that the simplest recipe for that is basically to get someone who is sufficiently qualified to have an opinion sort of second opinion on what is going on and make a statement on is this fit for purpose and write up some level of report that mirrors the model documentation that you require from a sort of a you know publisher perish type regimen just as as you have in universities that you require people who want to use or build these models that they need to write down what they're doing because you cannot assess computer code alone or just data alone you need to subject it to tests and it's much easier to assess this if you ask someone to write down what they're doing and also to provide that in a way that the other people can understand um and and to sort of propagate this knowledge about what is being done so so there's transparency uh involved in it um and as part of that process of course not of course but what we do when we do these model validations is that we compare the reality that's being modeled the data um to the mathematical description the overall abstract description of the model what it's intended to do what it's you know what is it it's its main driving elements and then comparing that model the mathematical specification of what's going on or the statistics with what is actually then coded implemented in in software and then compare the use of that software in real life with the reality that it's uh that's being used and this is actually um just this aside now this triangle here is is um is inspired by a paper from by thak adele from 2004 out of los alamos used for model validation and verification of nuclear weapons testing so when they when they banned uh you know actual tests of nuclear weapons and did computational tests of nuclear weapons these kind of this was sort of a recipe for how to manage that so it's sort of bringing the loop back of these weapons of math destruction to to principles underlying uh you know actual weapons of mass destruction and and how he can monitor and use them and and make sure that they uh in the event that you do need them that they actually work so so just just taking that sort of full um full circle uh thanks to this um and then in terms of actually so so when we conduct the model validation work we require people who own the models and develop the models to write down uh in excruciating details such that um such that the outputs can given the same inputs be replicated in essence um and and and and that enables us to to do an assessment of of the quality of the model but also to some extent the quality of uh and the experience of the people uh building the model sort of the quality of the model documentation to have access to the code um and to have access to the data which includes input output and also parameters and then we subject this to months of scrutiny and running tests on you know what is the use is it suitable for this particular setup are there any fundamental flaws what is the training the retraining calibration what's the feature stuff what are the biases dependencies testing various components looking at the uncertainties and assumptions and all this of course depending on what the particular technique is and what the particular application is just just a quick question between um do you work with synthetic data or do you work with simulations depending on what the particular application and the particular technique is so of course there's a there's a broad range of various types of tools that you can use to probe the validity of any model or any particular ai or machine learning application but it depends on what it is so it's very hard to give a generic recipe of what needs to be done um also in terms of explainability or bias or some of these other sort of hot topics which of course are important elements of this but i think to my mind the main problems um with ai and machine learning applications in the financial sector is sort of more the build quality and the maturity of the setting and not as much the explainability or the buyers even though those are of course uh depending on the application very important aspects and also something that you would test for the validation but then we also now if if we go too much into detail with the particulars of this we also of course end up in some kind of client privilege um a competitive issue on what are the gory details and how can it go wrong in ways that are perhaps a little bit too gory uh for for general public consumption um but but it's something that we need to make sure that we have um controls for and the way that you set up those controls is by subjecting it to this type of extended peer review and then in the end of course validators model validators write up their own reports their own documentation with nice executive summaries for senior management to to digest and understand the implications of it and all the relevant background and all the results and and recommendations for what needs to be done in terms of outstanding assessments of the risks and also um suggestions for how these can be mitigated or potentially accepted as well if if they're unmitigatable um but it's still you know paramount to use uh these techniques and you're willing to live with the risk that that might be the case sometimes and then lots of tables data uh references uh equations um and the lights and then from a broader sort of scale picture these are sort of that's what you do for individual applications then for all your applications of course you need to identify your models or your aei machine learning if you sit sort of on top of your organization and need to be aware of are we are we on top of our risks here you need to identify your models which can be tricky especially concerned you know related to how you would otherwise identify your legacy models your pricing models your market risk and greater risk it's easier to backtrack from your balance sheet where those calculations are where they fit in the magnitude of them than than a lot of these applications which can be used throughout the organization and also might impact you more indirectly on your balance sheet via fines or reputational risks so that's a problem and also it can be you know purchased by you know vendor models purchased by someone that are not centrally registered necessarily everything of course is centrally registered if it's a bank but people might not be aware that they're buying something with a sophisticated ai technique or machine learning um in in the box so to say and also by someone who is not necessarily accustomed to statistics or quantitative meshes so if it's someone in hr who've bought a vendor solution for assisting in recruitment they might not be aware of the regulatory expectations and general sort of risk management expectations on dangerous math to the same degree as your credit risk models so that's one element and then of course gather the documentation track the governance decision make sure that when you're looking at these applications that you have some element of tiering so you can still do a risk-based approach where your mitigating and risk management activities are commensurate with the risks that there are in these applications conduct validations do use recommendations and restrictions potentially and approvals for use to make sure that you do things by the book and this there's a trail on on how that's done monitoring um your models and applications in production looking at the retraining and adjustments if they're necessary and then of course also updating your uh reporting um as to how these models are are performing and then associated with that there's a governor's process with roles and responsibilities ranging from you know people responsible for the model uh people developing the model um various types of control functions in addition to the you know the model validators and the model users and and owners um that that also needs to be set up and defined uh typically in policies um that are applicable for the entire institution um but that needs to be refined for these ai and machine learning applications as well um and then specifically for model risk management and ai validation to update model risk management broad guidelines to make sure that they cater for these types of models as well to set up specific validation guidelines for these types of models um so somewhat tweak the existing rules and guidelines for how you manage your your legacy models to also cater for the the characteristics of in particular risks of these new types of models consider whether you have the adequate validation teams and updated charters for those if if you want to to set up that route and that you have appropriate um representation in your governance bodies so you can so they can take the right decisions and understand uh the the extend and the level and the characteristics of these types of models as well update reporting get the identification done from the validation team which are most likely to be able to spot and identify and tier these models set up validation plans so make sure that you've seen things in a timely manner and identify stakeholders across your organization not only users and developers and and make sure that things aren't good in management but also across second line that you have compliance and operational risk and security risk your legal and your audit teams aligned on the types of of issues that can arise these new applications and new techniques and then of course from a practical point of view from validation you you also need infrastructure you need compute resources you need access to data uh unique data scientists uh validators of course but you also need the data um and and to build the tools and be able to to conduct the tests that you need to do in a timely manner as well yeah and then potentially um lastly also support the development side by by making development standards that the developers and users need to apply to [Music] so it's important to also from a resource perspective because this sounds terribly expensive right to keep in mind that your model the fundamental abstract thing that that you that you typically would assess as a model validator is only the tip of the iceberg there's there's always more stuff underneath in terms of governance and you know getting the appropriate staff in place making sure you have controls and and things related to your data ethical aspects the basic operations making sure that it doesn't break down constantly and that you have the appropriate infrastructure for running this um that's the vast majority of your resources are not going to be put into these you know fundamental elements of does it do what it says on the tin but regardless of that your your your model the bit that sticks um above the water still fully capable of syncing your titanic project uh if it does hit the ice back so so of course you need to be mindful of uh these more general um generic aspects of what it is you're doing and preferentially before you start spending all your money on the operations and the infrastructure and all that stuff yeah so is it fit for purpose is should be the first thing that uh that comes into mind when you want to run any of these projects and then of course also on the resource consumption related to to the control side it doesn't take that long time to write up rules of course consequences of bad rules are you can be severe and significant but just providing the documents doesn't necessarily take up that much effort it takes up a lot more effort to do the controls to conduct continuous model validation and and and monitoring and making sure that you do that but the vast majority of your consumption is going to be in your development so being compliant with the rules um [Music] that's where the real resource constraint actually is yeah even though it looks a little bit perhaps like all these controls are uh are sucking everything out of of of your business that you need to comply to all these requirements i to my mind still the main task is building and using these models and that's also where you're going to spend the most of your irs and also where the impact is going to be biggest if you do make mistakes and then lastly of course um all this is sort of ideally if you're a large you know globally significant financial institution then you need to to have an ai risk framework in place and also potentially to have a dedicated air model validation team and if you know sense of excellence and control across second line but if you are not a global significant financial institution you still need to have an ai risk framework in place that's the easy part those are the documents so if you're a medium-sized bank um you can upgrade your existing validation function that you have if you're in a immediate size bank to potentially cater for these types of models as well i think most places actually do have a sense of excellence even if they're not necessarily these massive international institutions of course there's some level of variation of how much um you know the size of your center of excellence also how centered it is um and of course also how excellent it is um so so some variation there um but most people actually setting up some uh some you know shining city on the hill for the rest of the developers that they can mimic and they can go to and ask how do we do things in a compliant in a good way and how do we make sure that we don't set out in overly optimistic uh projects and be surprised by the level of governance uh related to to using these types of applications in a banking sector and then lastly um sort of in in this gold silver bronx type of approach to ai model risk management and ai risk management is that for smaller banks uh insurance companies and hedge funds and other sort of not as heavily regulated entities i think making the documents and setting up some kind of center of excellence and alignment across your second mind is is still needed but you might get away with only not doing model validation per se but more sort of risk assessment in a more broader sense um related to these models so a little bit like what we discussed on uh on the difference between uh being a bank and being a startup company so so you you can utilize some of the same principles but you don't necessarily have to do it with all the bells and whistles it comes with having a dedicated air modulation team luca is asking why you put insurance here and referred here for that or is it just like it happened so so in insurance companies are generally not as heavily regulated as banks are so it's it's actually uh related to the regulation and the intensity yeah but of course regulation is also related to the underlying risk so it's no it's not just uh out of being complied with someone because we need to be compliant there's good reasons for the extra level of scrutiny uh on banks and credit risk uh sort of credit institutions uh generally uh than there are um on insurance companies or hedge funds or small banks yeah from your experience and that was one of the questions earlier which perfectly fits now um are all ei machine learning or machine learning project subject to regulations depending on the use i would say um so if if if you use it for something which is really important yes if you don't really use it or whatever you're using it for is uh sort of you know relatively benign or unimportant then in in a risk-based approach you would not need to do um anything but just assure that that is actually what's going on so if you say used is a kind of rule of thumb to say like hey look if if your ai model uh effects for people's life in the sense of money aspects of people over the area this is getting more and more and more something like that or yeah yeah and and for um for financial institutions there's typically um especially for operational risks there's typically some sort of framework that are set up or scoring schemes that can enlighten you in terms of is this important or not important oh yeah is it is it gonna cause uh you know instant calamity uh or is it something with a you know low probability of uh something bad happening or and if the impact it's only going to be so so for financial institutions most risk is you know measured in money yeah um so it's not measured in lives or lost limbs but but quantified in terms of monetary loss and i think that's a that's a good guiding principle for in in this industry at least is to translate these uh these impacts into potential loss um but it can be difficult i i think and also that's one of the main challenges here is to find out you know what is the actual impact uh and how does it impact when it's this of this more indirect nature but i think sort of basic sanity checks that you know what would happen if you just pull the block or you say that it you know you think it's been doing what it's been doing but in reality it hasn't been doing this for this period of time so what if whatever you know obligations you had that you thought your machine was doing um what if you actually haven't fulfilled those what what would be the consequences of that in terms of fines or reputational impact or or other types of financial loss yeah an interesting um aspect of just raising because i was thinking of um a video i i saw about uh the city of palantir uh i guess you know volunteer i i suppose that most of the audience have heard of poland here as well so the interesting part of it because this seems to be high tech really like modeling our supply chain effects and military operations and what happens in terms of prices and whatever um it seems to have heavy stuff right uh the interesting part was when he said actually uh it's more like an augmented ai so it's not that this whole thing is doing something uh without a human in the end it's more like adding value to someone where still to make decisions and i think it's the same like healthcare again so where we have augmented ai in many cases as well where it's just like an invented physician we have a doctor who decides what to do and there's a suggestion thing like that yeah you know of course it um so so what's typically done for model risk is that for the tiering is that that you consider the complexity of your application and also of the problem that you're trying to solve yeah the exposure in terms of what is the potential monetary loss or financial loss and then some degree of reliance so do you have a human in the loop have you replaced a human completely so there's uh full autonomy of the system do you have alternatives to your model that can also you know is is uh does it rely on you know is there a high degree of uncertainty related to it so it depends on input that you cannot observe so unobservable parameters and all that stuff that relates to this element of how much do you trust it and how much do you put put your life in in the hands of this particular setup but also of course if if you do claim that you have a human in the loop as a and you're only augmenting and and that this human is actually mitigating some of the risk you still need to test that that is an adequate mitigate of the risk yeah so you put reliance on the human and that they can actually safely do that so i think and and you're still interested in both of course um identifying and managing your um sort of inherent risk of your ai machine learning application or model um but also the residual risk that you have after so i think my um being a biologist my sort of uh the best damage i have is from if you if you visit us soon um so the most dangerous animals are i don't know tiger something um inherently most dangerous animal is the tiger but so the residual risk that you're most subjected to is probably you know in the petting zoo um but of course you need to you know an actual damage most likely would be from you know a goat uh or something like that in the pedals but you need of course to make sure that you know the bars are in place there's a lock on the door in the tiger den as well yeah um so so so that the risk is identified and mitigated and that those mitigations are adequate um for the risk it is still vital for for managing your risks um and that goes for these applications and and uh augmented setups uh as well i think um that is building [Music] so so that's basically my presentation for tonight uh thank you very much uh sir and yeah thanks for for sharing all the insights uh uh if we took of course a little bit longer maybe in your evening hours i'll bring you back for instance private time in some ways obvious one uh one uh let me check one question open um look but i think you need to explain that a little bit because i i'm not really sure what you mean with spiders insects and bacteria or maybe in terms of risk risk factors but anyway different ends of the spectrum i think i was the one thing to say is uh you uh you're reachable on linkedin so this is basically where by introducing uh from from francesco uh to you uh we reached out to each other as well on linkedin so please for everyone who is still open questions i'm looking at the time a little bit girls because i know everyone has also some private life here um um i think it's probably open to uh to answer questions i think it was very intense i i liked it very much i lied or enjoyed personally i enjoyed her discussion here with the talk with you and it was also awesome just to see your deep level of expertise in the field which is like cool man thanks for that so uh for anyone else because the question was all the race yes the session was recorded and it's still recorded uh we will make it available on our youtube channel again uh um thanks again uh oh yeah here look what was a question what are the most interest but often overlooked risks maybe the last question do you have an idea what are the most overlooked risks from ux it's difficult to say is the as a general sort of across the industry yeah uh problems are very much sort of individual um i think that the the key risk drivers though are some that are shared across the industry and that's that's what uh sort of as ey that we've seen um examples of uh of course some people are on top of their risks and manage it and don't make the same mistakes but but these are sort of general characteristics so why those are the one that that i've tried to highlight and and those are also the ones that i think if you want me to to provide a short shortlist um those are the ones i think it's also a bit problematic because uh it's still a young area right yeah and so so maybe uh so so so one perhaps word is is sort of inexperience or immaturity um as an underlying theme is is is the major problem associated with uh with these things and also one of the most tricky things to mitigate and compensate for so what because where do we find uh experience uh to to the degree that we needed yes yes fully understand well thanks again for this last question again if thank you everyone who joined us today and whatever time it is on your side i know it's sometimes very difficult because it's morning hours it's evening in late night hours it could be everything because we are spread all over the place thanks again for joining us uh please stay healthy uh we still have the same stuff happening since two years and i just hope that everyone just goes through a crisis without any issues thanks again sarah for taking the time and have a good time with your family thank you thank you bye [Music] you

Channel: AI Suisse

Views: 3,310

Keywords: artificial intelligence, natural language processing, causality

Id: m5l3jO4LAZY

Length: 85min 6sec (5106 seconds)

Published: Thu Dec 16 2021