Four Key Tools for Robust Enterprise NLP with Yunyao Li - #537

[Music] all right everyone i am here with yun yao lee yun yao is a senior research manager with ibm research working on natural language processing yunya welcome to the tuemo ai podcast thank you so much for having me here i'm looking forward to digging into our chat we'll be talking of course about nlp and some of your experience on the research side as well as making it useful for ibm's enterprise customers to get us started i'd love to have you share a little bit about your background and how you came to work in the field sure um so as you mentioned right right now i'm a senior research manager at ibm research before that how they are getting there it's actually a long story so i have to say i grew up in my small town in china right before i went to college i didn't not even see a computer but when i picked my major i picked the major of automation because i thought if i automated everything i don't have to do anything it's part of really think about ai helping people okay so when i went to college i also did a second degree i do that due degree i did automation and economics because i want to understand how technology impacting people impacting the society then i went down to come to us to pursue my graduate studies so i i did two master's degree again i did one in computer science another one in information science because again i want to understand the technology itself but i also want to understand how technology impacting people so my information sounds more focused on hci side and information economics then i pursue my pid in database however it's not a traditional database uh phd so what i did was to enable people to query database using natural language so that's basically one of the hottest topic today in natural language processing so i i would say my connection with ai basically started from the very beginning of when i started my higher education nice nice and uh now as a senior research manager talk a little bit about your research interests and more broadly the the role and it sounds like you you're focused on kind of traditional research but you also have these customer engagement element to your role can you talk a little bit about that sure so my general research interest is quite a broad but in general i'm very passionate about building systems and the tools to enable people wide range of users to be able to use technology in particular since i work in the field of natural language processing my passion is really to empower people to harvest information from text to build the next generation of ai applications um so within ibm you know we have a lot of product based on natural language processing capability like watson and nlu was in um discovery and so on so behind this technology i i'm very proud to see a lot of the products are part of our technology develop developed by my team so really empower the product themselves meanwhile you know when we have things in the lab they may not always mature enough to put in a product yet so then we also often engage with customers and also our technical sales team to say okay for this particular problem we don't have a product yet what can we do for the customer and that really helped me to understand what are the key challenges that we need to address from research point of view i think that in my mind you know as someone working in industry research lab for over 10 years right i really enjoyed that because when i was in school kind of you always need to think about what's the motivation behind the work here in industry research i don't have that problem the problem comes to me and then the interesting part is how do i generalize one person's specific problem into a research problem that can be solved in a staged fashion like both very ambitious from scientific point of view but the same time we deliver things as we go yeah well i want to dig into some of the specific challenges that you've identified in the nlp domain but projecting from an individual customer's challenge to kind of a broader interesting research problem is one that i'm also interested in can we maybe start start there and i'd love to get your take on that um so maybe i can share my story right when i join ibm i work on my project called the system t so system t is a declarative system for natural language processing and i have to see to a certain extent i'm still working on this system the reason is that when we have a larger research agenda we cannot solve the problem at once right so for example how systems they come into bin that's because at the beginning we want to build a better enterprise search system so when we're trying to support enterprise search internally within ibm we found the traditional io system do not understand the document enough but one thing that it differentiates internet search and internet searches in internet search when you try to find the answer to a question maybe it's only have one page that has the answer unlike when you do a google search on the internet when you search for certain questions right you may have hundreds of pages so then in internet it's really important to understand every single document and that really motivated the team to build that natural language processing system and then at that point what we found is there a lot of problems in creating natural language understanding system but the most basic one is really like for example the do information structure identify important entities identify important events and so on but at that point we don't have a very scalable system to to perform the task so then we start building the system t where we kind of abstract out what are the common text analytics operations that we need to capture then we build a system at the beginning our concern is about expressivity and runtime performance to be able to express the kind of tasks that are important for supporting enterprise search then as we go we found out okay we can build a system that is really good to building extractors for supporting enterprise search but then we want to enable more people to use them right then we kind of say okay now we have problem of how do we build a better tooling into it how do we incorporate more advanced machine learning capability into it so kind of at a different period of time we have different focus but overall because we have this one bigger project we can build one thing on top of the other so i think that's really the advantage of you know working in our industry lab where our research is not bounded by a fixed period of time for grant right but it is kind of generalized the overall life cycle like we have a concrete business problem for example we started of how do we support a better enterprise search right then we turn that into i cl like how do we build a system to support a class of such problem and then we saw some of them we build some prototype we evaluate with concrete use cases and then we get a feedback when when we can solve certain kind of concrete problems we can push it out into the wild and then we get additional feedback from the world then we incorporate to say okay now we have this system should we focus on improving the system itself or should we focus on the tooling should we focus on other things i think a lot of the work from the team really kind of come with the same life cycle well let's maybe jump into uh some of the challenges that we referenced earlier you know when you are thinking about and trying to to help teams productize nlp in the enterprise what are some of the the challenges that you run into and kind of how do you think about those those challenges have you organized those in your head yeah so that's a very good question i would say we can think of a four category of major challenges one is complexity complexity in terms of the because we'll talk about natural language processing right in terms of the complexity of the document we have to deal with and the complexity of the tasks themselves we have to deal with the second category is small data so again when we talk about enterprise facing challenges right we often have two new things that works for everybody but we don't necessarily have data representing everyone's problem right so therefore we need to be able to uh address this small data problem either based on you know what we have internally or have some way to enable people to provide data for us to help themselves and then the third party is customization no matter what we build right how well we build it the out-of-box capability may not be sufficient for a customer's specific use case or works as well as it could be on their own data then how do we enable the customers to quickly customize what we build to fit into their use case their data very quickly and easily do not necessarily require someone like me right someone with a phd degree to work with them so instead maybe someone with a business knowledge to be able to do this work on their own and then the fourth part explainability i know you know everybody talk about trusted ai and so on these days right one big part of trusted ai is explainability then here we need to understand what kind of explanability makes sense to what kind of audience right and how do we provide variety of explanability to help people so i would say yeah those are the four major challenges we are focused on from enterprise natural language processing point of view when you think about those challenges do you think about them independently meaning does do you have a set of solutions or a set of research directions to try to address complexity and a separate set to try to address small data et cetera et cetera or you know is there a way that you kind of unify these in the solutions that you're trying to apply to these various problems yeah i would say we have a few common approach a combination of them to solve all the problems maybe with different combinations i think the four things we often use as a foreign number one is data augmentation right the second is to have a very powerful um declarative language that captures some of the main primitives that are important for con to conduct the task and the third one is neurosymbolic ai really leverage the best of both worlds in neural network as well as a symbolic system and the fourth one is human loop how do we involve human beyond just provide labor data so those are the four things that the business security formula that we have been following and then for some different challenges we may use a combination of two of them or three of them got it you got it i want to go through those in turn but maybe to set some context it'd be helpful to to have you talk about you know some specific challenges and where each of those uh come comes in sure i think maybe i can describe a particular system we have recently built and i think the system kind of representative for all the challenges so what we have been working on with the washington discovery team was to build something called watson discovery content and intelligence what this system does is to enable lawyers and law professionals to quickly review contract document if you think about contract right it's probably one of the most important business document in the business world and it's very expensive to review those contracts because you need to have legal professionals and the contractor often have many many pages and contain a lot of information that is challenging even for you and me to understand right so to build such a system what is required first of all your the first step is a lot of contract uh in pdf format so we need to convert the pdf into our form of the consumable by the machine right then secondly we need to build models to simulate what the north professionals would do so that we can help them right so what contrast understanding involve on however is a foreign giving every single document identifies the clause for simplicity you can think of sentences for every single sentence identify what are the categories of that particular sentence and what are the parties involved in the sentence for example if i say the buyers shall pay supplier one million dollars right so in legal terms this is an obligation and the party involved is a buyer and a supplier right and then there is some amount then we turn this into a sentence first of all identify this sentence right identify sentence from the document and then uh do a classification so that we can categorize this sentence and also do extraction to extract what are the parties involved like supplier and buyer so now then once we have this system built it will not work perfectly right so we need to service it to our new professionals to the legal professional can say okay now i want to review all the class related to obligation and then look at those sentences and to be able to say okay this is in data obligation i want to do something like market as a high risk or low risk or no problem um but think of the challenge we just mentioned right complexity first of all the the complexity involving the document itself the document is a multi-page pdf document that we need to preserve all this structure all the information and also be able to identify like assuming the document conversion is correct we also need to identify the document structure and to be able to identify individual sentences under the context right then why it's a small data problem because when it comes to contracts nobody's gonna share their contract with ibm for us to create training data right so if we want to build a model we have to rely on ibm contract data but at the same time what we build needs to work for other people right then why it's a customization problem because even though we're talking about the the legal professionals follow some general taxonomy related to the classification every single company they may have some difference for example if a sentence mentions patent right or a trademark it can be viewed as a class related to intellectual property but some companies want to separate them they want to say okay i only identify all the class related trademark as trademark but everything else related to intellectual property is intellectual property okay right and then finally explainability when we surface the results to the legal professionals we also need to especially when there is a mistake from the from the prediction we need to explain to the legal professional like why we predict this sentence as something right like this is obligation rather than something else we need to explain that and also what that will help us is when they say this is not correct we can also look into potentially how to fix them and they can explain to us okay why certain sense is not labeled correctly like what i just described earlier related to customization right when they come back to say oh this sentence should be labeled as trademark instead of intellectual property they can explain to us to say okay for this particular thing you identify this crew related to trademark but in our company this is regarded as a trademark rather than international property so they also help the model developers to make some changes um yeah so i hope this example kind of illustrates all the different threads um but i will also be happy to talk about you know how do we solve them yeah yeah well we'll definitely do that one question that does come to mind um yeah i've talked in the past with with folks that are you know working on these production nlp systems and you know over time the need to supplement the the learn models with rule-based systems and heuristics has decreased and i'm curious in the cases that you deal with you know in addition to the data augmentation declarative neurosymbolic human and loop you know do you also need to you know are you still worrying about you know rules and exceptions and all of these things as well and and do you see those decreasing over time i think depending on the use case so for example for some use case when full explainability is important we actually see requirements to combine both so let me give you an example right so if we think about model development it has a life cycle right you have data gathering we have developing the model self we have tests and validation so for some industry for example for retail banking the fact you know the training test and the validation what we learned is if you use a block box model right like a neural neural approach assuming you have sufficient data the training is very fast tests will also will not take too much time but the validation will take a lot of time so what they do in validation really ensure the model work as predicted under different variety of conditions and also be able to explain why certain predictions produce the particular results versus what is expected right so this takes a lot of time but if we use rule-based approach right it's kind of opposite so the development of these rules takes a lot of time testing also takes some time validation doesn't take too much time so overall you kind of see the time and effort distributing different way so what my team has been working on is actually kind of getting the best of both worlds so basically what we do is we take a neural network we leverage all the advantage of neural network to be able to learn from large amounts of data right but then we produce set of rules that are completely transparent and explainable so the advantage of this and then i the expert can actually come in to do further inspection and augmentation to ensure the model not just learn from the data the model also learn from the domain expert the corresponding domain knowledge right so the benefit of this approach is the training will not take as much time as the the purely rule-based system right it takes similar time as i whatever approach we are taking for black box model testing is also similar but the validation takes significantly shorter amount of time than if you take a black box model um depending on the use case right in some use cases you really don't need to have full explainability then i think we are using neural model probably is the way to go but in the use cases where explainability is important it's important to hold someone accountable when the system makes right so then this approach works really well another aspect is remember what we talked about about a small data problem right so this issue is no yeah in a lot of the area we don't have sufficient data so when you learn from that alone it will not capture the whole domain knowledge right so i think what our experience the domain experts really like the facts they can not just provide input by labeling they can also provide input by inspecting what learned from the model and make changes not just to understand how model performed but able to make changes for example uh some persons like what i just described earlier right to say okay far share pay supplier a million dollar right i potential rule that we can learn is when there is a predicate indicating like a verb indicating uh purchase indicating some business transaction and then there is a model there is a modality of necessity modify that particular verb then it's potentially an indication of of negation this is something completely transparent right we can learn such a rule but what are considered as an important business transaction verb this can be given by the domain expert not just by learning from the data right we can also that is the domain expert to inspect the needs of dictionaries we learned missile rules we learned so that we can ensure their knowledge is captured by our model and then we can combine right so this model can be then combined with a black box model so that we can really ensure you know the overall performance is as good as possible but at the same time we're able to predict or explain significant portion of the prediction and how the two models are combined together depending on use case like for example if you want to say i want to have as good performance as possible then if i can explain some of the results that's good then we can combine it in one way or you want to say explainability is most important to me right then with that i also want how good performance then you can combine these two in a different way so i think with this particular approach we really can get a best of both worlds you know be able to leverage the fact neural network is very powerful to be able to really capture a lot of nuances from the data that is very hard to express in rules right but at the same time capture as much as possible the domain knowledge that can be clearly articulated through rules another thing i want to mention is the rules are not based on syntactical pattern right so the rules are based on obstruction we obtained through natural language understanding like for example giving the sentence fair share paid supplier through natural language understanding shallowest money pricing we will know there is an action of pay and then the performer of the auction is the buyer and the target of the action is the supplier and then the manner is paid with you know a million dollar so this kind of semantics is captured through advanced natural language understanding therefore we can express rules in a very concise but a very powerful way i think that's another thing people need to think about rules are not necessarily one token for another right so traditionally when we think about rules is like okay well when you say buyer maybe follow in two tokens there is a supplier but then that cannot capture the nuance of the language right but if we build a rules on top of semantic abstraction they can be much more powerful and then behind the scenes also leverage the deep learning right so be honest in leveraging neural network model to do natural language understanding um got it got it and now just to to make sure that we're on the same page is what you described i kind of came at it from the perspective of rules but is this the neurosymbolic uh ai that that you spoke about earlier correct correct correct yeah so i think neurosurgical you can basically think of the two approaches right one approach is you embed symbolic information or expression into a neural model right that's how people can ingest some domain knowledge into neural model but then the neural model coming out still not completely transparent right the other way you can do this is you can use neural model to produce symbolic model or symbolic obstruction then we can build a symbolic model on top of it so right now um imagine we actually have both approach but what i'm describing more of a ladder where we leverage neural model to empower symbolic model because the symmetrical model has a lot of advantages that people is fully aware right explainability and the and the part where you know it's easier for people to interact with and understand um yeah so this is one particular neurosymbolic approach we have been taken and then one of the the other techniques that you mentioned was the use of declarative languages i'm imagining that you know you've used this neural model to create kind of this abstract rule let's say about a particular scenario you want to make that accessible to well the humans in the loop that we also talked about a way to do that is to kind of represent this abstraction using some kind of declarative language right exactly so i think yeah i think that's a very very good point right so you can express rules in grammar but the challenge is that grammar also detect how the rules will be executed right then that'll make the rule creation much more challenging and less expressive and also how runtime performance issue so we kind of overcome that by leveraging the declarative system we have mentioned earlier right like the system i have been working with since i joined ibm is we can separate execution from the actual semantics so you can specify the semantics using declarative language but how it will be executed is determined by the optimizer the optimizer will determine what's the most efficient way to execute so for example if my rule is to identify all this sentence contains business transaction and it's uh an included modality indicating necessity i can do it in two different way right one is i can identify all the sentences with necessity and then see whether there is a business transaction next to it or i can do the other way around which one more efficient were decided by the optimizer like when i created a rule whether manually or automatically i don't need to worry about execution i the execution will be done automatically by the declarative system to ensure we get the most optimized the plan in terms of execution we talk about execution in this context what do we mean we you typically think of uh you know neural network inference or you know even more broadly ml model inference as kind of this one you know you make a request you get a prediction back are you thinking about execution more from the perspective of like information retrieval and queries and that kind of thing um so here when we have a rule based this like based on this kind of model right so for example for simplicity let's say we do two parts one part is just do natural language understanding the second part is just enforce the predicates right so the execution composed of a few different options one option is that you perform this natural language understanding operation the neural model we have right on every single sentence then for every single sentence you also check whether it contains any business transaction mentions and also some words indicating necessity so that's one way of execution another way the execution is that i can actually do some optimization instead of look at every single sentence i can first retrieve only the sentence contains the business transaction then i perform natural language understanding then i check my predicates so it's it's really how we produce a final result so that's the execution i'm talking about got it got it so to maybe kind of recap this and and put it back in the context of this original problem where you're trying to enable legal discovery across lots of contracts um you know one one approach one might take is to you know use traditional deep neural networks and you know maybe a supervised learning kind of thing or even unsupervised and kind of clustering the the different you know clauses or sentences um but you know what you've found is that by combining by using the the neural network instead of uh to classify but rather to kind of do something akin to like entity extraction or or to or semantic parsing you can use that to generate this essentially a rule set that's kind of expressed in this declarative language and then you've got a system that executes those rules against the given contract and uses that to essentially make sense of it and identify whatever it is that you might want to identify you know at a given time correct yeah on the high level you can view it that way um but we can also do combination right so you can basically think of i can either build a black box classifier or i can build a white box or i combine the black box and white box together so that i can get the best of both worlds so i think in reality we actually do both um so i think the challenge of doing this blockbox one another another challenge is the following especially when we start with building the product we don't have even a lot of labor data and and also as i mentioned right we have this small data problem we have data from ibm but we don't have data from other companies how do i ensure the model i build works well so here the out of domain performance is very important right so what we we actually started with amount of different approach right you can think about in ibm research we also build a lot of different neural networks right that's really anything that are accessible to other people we also have it but we when we build those side by side what we found is that the intermittent performance for the deep neural network works really really well but then it drops very significantly when we do auto domain so that really passionately motivated us to build more of this transparent model so that the out of domain performance is very similar to in-domain then we can augment this particular model if we want to have a better performance to say if we have additional training data we can also kind of augment this with a black box model but at the same time we can benefit from the explanability transferability and also be able to really quickly start with a particular customer's use case without worry about training that hub because i think a lot of the time and i mean i'm deeply involved in natural language the process in research right so if you if you look at the papers people often assume we already have labels that are very nicely available and right then if you don't have label data you can just do crowdsourcing but in the case of enterprise applications like i cannot even label this data right the contextual information is really really rich so i remember at the beginning when we built this product we look at all the sentences and like true sentence looks almost identical we go to ask the lawyer to say okay why why those true sentence looks identical labeled differently the lawyer tells us oh because one is within this section with this information but the other do not right again like how do you even capture like let's say you have this information i can quickly basically call that rule to say okay take this section information into account right i can do that in a few minutes but think about how do i return my model to do that like now i have to yeah so i think in general i would say depending on the use case right there so in my opinion there is no one solution we need to really use the combination of all these technology techniques available in our toolkit and take the practical challenges and also what's the final goal in mind right like for example in our case our goal is not to say build the best possible model for ibm contract if that's my goal i don't need to have this approach right i can just do this training and then have a model that works really really well for in-domain but our goal is to build a good model for other people and enable them to customize and that really comes to the solution that i mentioned right so we want to take advantage of both and in describing that solution you mentioned a lot of different options you know the white box the the black box etc um how do you i'm wondering like where you're making those decisions is it you know it's not at the level of product because you want i'm imagining a platform that kind of does you know that can handle multiple use cases you know do you have different paths that you go down for different use cases or or am i thinking about this incorrectly and and you know it's more of you have this tool kit and you you have to apply this tool kit kind of more in a consulting or orientation when you're faced with a particular use case i think that a few different ways to approach it right one is like for example when we build the what i just described this particular offering in washington discovery content intelligence it's it's one it's one offering right and and then in that offering we take this particular approach and then we show it's really useful but assuming like the customer comes to me with something else um then one one thing we're actually doing right now is to take more of our auto ai approach where the user give us a constraint we will figure out based on the constraints they give to us what's the most uh like what's the optimal combination um so you probably have is aware right like although auto ml is kind of a very friendly topic so for example hacking face recently pushed out a new feature called the auto nlp right because even with the black box you still have a lot of challenges like which models you use how do you change the hyper parameter and so on right so here is actually think about the main parameter we introduce is we have additional type of models and we have additional ways of assembling them together and how do we do those could be automated when the users specify constraints so for example the user can say i want to have best performing model with some explainability right only the user can say i want best performing model i don't care about explainability right then with those constraints we can automatically figure that out this is something we are building in in our project called auto air for text so we're building all these operators that are supporting different kind of model and different model have different property right so then based on the user constraint we can decide which combination model to use and then which model to refine further look at it you know refine the hyperparameter tuning and refine how do we do the final ensemble got it got it got it since you mentioned the auto nlp uh and hugging face i'll quickly mention to folks that i interviewed uh one of the folks who worked on that uh abhishek thucker and we'll drop a link to that interview in the show notes but i want to get back to uh data augmentation because we haven't really talked about that so much here and i think it's you know it's you know often easier to think about how to do that in the context of computer vision you know add noise or change the orientation or flip or rotate or that kind of thing what does that mean in the context of nlp and have you included it or or incorporated it into your projects yeah sure so talking about data augmentation nlp actually there is an excellent tutorial or service paper come out very recently i summarize some of the techniques i i can i can still do a link later on but in the context of what we are working on we do data augmentation a few different ways um the first one is um so i mentioned we do natural language understanding right we do you know giving a sentence we pass into some semantic structure so we have enough data for english to do that however for other languages we don't have this nice data right we don't have labels that are to perform the same task but we want to also support other languages so one technique we have been using in terms of data augmentation is we can leverage the fact there's the by-text we have sentence in one language like english and in a sentence in another language that is translation of the english sentence we can do a sentence alignment right align each token with each other and then we can project the annotation we produce on english onto other language so now we have some very noisy label data to start with right because you can think about they're always lost in translation right so when you do this projection problem is there are going to be some mistakes so we also do additional filtering to ensure that the data we produce is as good as possible and then we can use this uh data to trim a uh semantic password as a manufacturer in the other language to further augment that kind of repair the data so maybe after projection for example in my sentence i only have some portion with annotation the other portion we do not then we can do this bootstrapping to add additional ones so this way we can automatically at a scale produce large amounts of data so this is like one way to do that augmentation another way you can also do it is so for example one of my colleagues they have been working on dependency passing for other languages right so again we may not have sufficient data for other languages so what they have done is they translate the english sentence of other language into english sentence um and then back on the force so this way you can also produce larger amount of what we call silver that it's not good that because it's just still contain good amount of noise but when you train from this data the neural model is able to distill the noise and to be able to learn a pretty reasonable model so a lot of the data augmentation in natural language kind of doing this synthetic data or annotation projection annotation transfer there are also ways to generate that like for example if you have this actually i recent paper we were publishing acl earlier this year so one of the tasks that is challenging for pastor is handling questions because most of the training data for natural language processing are statements right news reports and so on but often we have data loads that actually are questions so what we found is those the positive trend using the typical data set do not work well very well for questions so then what we did but but then label question is very expensive right producing label data for questions very expensive so what we did is there is a data center called the question bank it has some maybe a couple of thousand in the sentence and so on so we basically generalized from those questions uh from from the question bank to generalize into template so with those templates we can generate generate a lot of the label data for questions and then we can add those generated questions back into the training data so that the password can handle question very well so i think it's actually to a great extent similar to uh what people are doing in computer vision right is just how you manipulate the data is different because well you don't deal with text here instead of vision right um nice nice we haven't talked uh much about human in the loop i think a lot of folks are familiar with uh some of the common uses of human and in human in the loop uh uh you know of course exception handling uh is one that comes to mind is that the kind of thing you are referring to when you mention it that's a very good question because i can have a very strong opinion about that i think you know human loop is actually very broad and i can describe a few different aspects in my opinion human can be very important as part of the model development process right often when you think about human loop it's more about labeling right how do we involve people to provide labor that that's very important in fact we have spent a few years trying to figure out how do we enable average person to be able to provide high quality data for shallow demanding person because our initial way of doing this it shows that the human cannot perform as good as the our machine learning model then we need to come up with a more intelligent flow to enable average people to do as much as possible and then enable expert to do the rest so i think that's one big part of when we talk about human loop is how do we enable human to generate high quality data but not necessarily all be done by crowd source worker because not every task is possible but then at the same time we don't necessarily always fall back to the expert in the loop chain so we need to kind of have crowd in the loop but at the same time involve you know experts or automatic method as much as possible then one thing people don't really talk too much related to human loop is the model development the reason is very simple right most of the time when people talk about model development you are talking about a black box model right but when we talk about the white box model or grey box model we want to involve humans and loop in a few different ways right one is what i described before we use deep neural network known a transparent model now human can inspect the model augmented model modify the model that's one big part of human loop right the second part is maybe the user do not need to give us a lot of data at the beginning instead we can have the model creation and data gathering in one intelligent simplified mod mode like for example when i talk about the classification of this particular cross right so instead the user gave me you know a few thousand labels that are trained model and then we write what we can do is okay users may not even specify a task upfront they usually have to say hey this sentence interesting i will label a few interesting sentences as an example and then that is a machine figure out what are other similar sentences right but again the machine may not fully understand what i mean by similar so the machine will propose some potential interpretation right and as the user provides feedback and then as the user provides feedback the machine can learn a bit more and refine what they have what the machine has learned and in the end it will produce a model but it's a very collaborative process right and in between the user may go beyond just give me an example the user may even give me rationale to say hey why i think this sentence is interesting or why i think what you propose is wrong and then it's very interesting to incorporate this kind of what we call rich feedback into the machine learning process again this is like a model development process right um and then another approach is we so as i mentioned before right we have this declarative system decorative system is actually very good for tasks that has very obvious patterns right so rather than trying to net the black box model figured out we can we can specify that right but specify that can be expensive um so we can also have this process of user giving example i can explain to the user in my declarative language what i have learned and then the user can give me either additional feedback or modify another particular declarative statement directly so that's like one big part of human loop on model development finally about user feedback right so how do we enable the user to provide additional input to the system when the system already deployed so again that's also a important part like where think about today in a lot of systems if the user not happy you cannot really do anything right um so the system produce whatever you produce that's it but you can imagine a system where for example the contract intelligence system i mentioned before right when the user sees the outputs the user not completely happy with it the user can explain why she's not happy with the results right and then can also maybe provide some additional input to say okay here in let's say for example in my company right we don't we will differentiate trademark and ip so we can learn from those and that can come back to our development team to be able to say okay is there something we need to address in our baseline model or this something we need to address but enable the user to be able to do some customization so that allows to do much more intelligent model improvement and maintenance got it got it it sounds like the the elements of this that you feel most strongly about are uh kind of a combination of closing the loop so you you put the human in the loop but then close the loop so that it improves your model but also as an idea of like intelligent human interlude use machine intelligence to optimize the way the human intelligence is being used in the loop itself right exactly i think we need to give a human more agency right we need to empower humans rather than just you know sometimes if we if you think about it if all the user can do it give you label data and then leave everything to the machine the human is almost powerless you cannot do anything right you know the machine is not is not correct but as a domain expert you don't know how to notify the model you don't know how what to do right you're kind of waiting waiting with him hopefully someone will incorporate the the data or the knowledge properly but when we actually give human agency i think that they have they can do much more so for example um actually in recent in 2021 we can see a very interesting trend so in 2021 we see three workshops from top nlp conference that is related to human and like human and nlp so we have a workshop on interactive machine learning we have a workshop on nlp and hci and we have a workshop on data science with human loop language advancement and next year in 2022 we are also going to have a special theme in naco that is focused on hci for nrp so basically more and more like when nlp become more and more used in practice human become an important aspect right when it's just in the lab you know all you need to do is produce some numbers and then compare with benchmark data you don't need to consider a human right but now when arp actually impacting business when it's actually used by people you know how do we use people to help with the entire development life cycle including evaluation and so on become one more important and you can see the trend awesome well yunya thanks so much for taking the time to share a bit about what you're up to and walk us through these four different tools that uh you've had some success with in delivering enterprise nlp thank you so much for having me here thank you

Channel: The TWIML AI Podcast with Sam Charrington

Views: 155

Keywords: Ai, artificial intelligence, data, data science, technology, TWiML, tech, machine learning, podcast, ml, yunyao li, nlp, semantic parsing, deep neural networks, IBM, IBM Research, document discovery, supervised learning, unsupervised learning, entity extraction, enterprise nlp, enterprise, human-in-the-loop, ibm debater, productizing, Neurosymbolic ai, system T, natural language processing, natural language processing tutorial, natural language processing in artificial intelligence

Id: UhaXPEArKkc

Length: 56min 36sec (3396 seconds)

Published: Thu Nov 18 2021