Future of Industrial Automation based on Natural Language Processing.

um all right so with that i would like to get to the to the demo part of this thing so we have seen already the uh salesforce demo uh and maybe just to switch a little bit the the order uh maybe uh carl you can study the whole pistol this time sure um i have a very small can you hear me yeah okay i have a very small presentation and then a live demo so i'll share my screen sure okay you should see my screen now um so very quickly going over this presentation just getting everybody up to speed um we're trying to combine autumn optimization with a text and speech interface uh vazelan already talked about why this is important i i also think this is the direction things are going in they're emerging a lot of these textual or or these spoken interfaces to typical platforms and if you can do it good i think it's really like also with the demo that mikhail gave about the playbooks i think there's a lot of value there it's very difficult because there's a lot of ambiguities you can say the same rule in many different ways we have almost no data to do it and it needs to be quickly adaptable to new sensors so before i talk about my proposed solution i would like to again bring you up to speed with what i mean with a canonical representation here in green we have three sentences which basically say the same thing so for example at least sms veselan if the sun is shining against tell him hello all these kinds of things well we can map this to a kind of english pseudo-english sentence so english with some parentheses that really captures um the semantics of this rule and so if weathering gans equals sunny then send sms to vaseline message hello and our idea is actually to to tackle this problem instead of trying to tackle the whole language space and translate that into alias rules we'll try to translate natural language into this canonical language and then we can easily parse this and the idea is that that should be way more robust and extendable in the future so the proposed solution um i made a couple of micro services so we have a gateway database uh then a machine learning model gt3 and a parser so text or speech comes in and it gets sent to the database and what this database does is it looks through the few examples we have of rules and tries to um with an algorithm i'll not go into detail tries to find some similar examples so if you have 100 examples it'll pull out the 10 most relevant so for example if you're talking about monitoring a patient's heart rate it'll not pull out an example of turning the sprinklers on and off so that's just the first filter then did this get passed to a model and actually so what this gt3 model is to to to explain simply you know on your phone you have like this autocorrect where you can take the next word um it's basically that on steroids super big so it's just um tries to predict the next word the next word the next word and so the idea is you give it a bunch of these examples and then you give it the sentence you want to translate into something canonical and just by predicting the next word it'll get there hopefully so then we have we translated a complex sentence into an easy sentence and then we can we can send it to a parser um because it's it's very easy to parse sentences that always follow the same you know the same uh format uh carl is it fair to say that what digital team does it's to basically uh go straight to the kind of a parser because they create a canonical representation for the simplified ui yes so so in the in the as much as i understand in the digital twin they indeed they just already have a very structured format where they input it so then it's very easy to just make the rule out of it without any ai the thing with speech is people will not always adhere to a structured format or even if they talk in a structured format the speech detection might lose a few word words so you can't assume that your input is structured so that's why we need this database and this ai model to translate it and to get the structure back and so that that gets sent back um it requires user input on two things and this is something i was really scared about because we're trying to make a no code or low code interface here we don't want a lot of user input however it's um it's still impossible to just say give me the temperature to just know which temperature the user is talking about when you have 20 temperature sensors so um the user should give a few examples of his or her sensors like how do i want these sensors to behave in natural language and in canonical language and then the user or or or services or or whatever um needs to write one piece of code that helps uh create then the whaley template so i'll now talk a bit about this parser very quickly and then i'll show the demo just because it's still a bit of abstract so imagine if you have a piece of a canonical sentence so it's weather stating gans equals sunny and weather humidity and gans above 80. at one point this needs to get parsed into a valet rule now how do you do this um you use a kind of visitor pattern it's also the same which amazon alexa uses and that means at one point your algorithm will will see well i i am um i have an answer so you can see it in the get key phrases it returns an and so it will match on this and and then it will run this visit function and basically if you look for this pseudocode you can see it does something very simple it parses the the string it takes the the condition to the left and the condition to the right it processes them so it also visits them and then once it has those two things for example these are now relay sensors initiated it can say okay now create an end relation between those two things so that's the way how you can how you can translate them that canonical language in [Music] in in the whale a json the template json with minimal amounts of code so i basically it's kind of a python sdk actually for creating rules um and and that's how it works but i'll i'll show it in a bit more detail later now i will show you a demo um you should be able to see my new screen now the application i i thought about two ways of doing this demo i can either show the full potential right away or i can kind of walk you guys through it and try to make it interactive and together explore what this technology is capable for so first let's see which rules are in the database there are only four rules in the database so um the entire application now is resting on only four examples of what it should be doing and so for example um here i have the weathering again to sunny san fezel and sms with message you want to go for running here the the canonical version which we can easily parse if the weather is not snowing blah blah send something only do this when it is raining okay so let's input a uh written rule now so maybe ask forzalin what's up via sms if it if it is snowing in paris so maybe reflect on this so in the previous demo the zealand made about the parsing this would already fail because there's no key message here you're saying ask for xalan what's up so so you it's very difficult to parse a message out of that but i'm going to submit it and hopefully it'll work okay and it works so what is our our output this is really not that important for you guys that's just the actual input to the model so if you see we here have our ask fazalen question and these are the the database examples that our model model pulled up and sent with that example to our model then we have here the canonical so the if weather state in parents equals snowing then sent sms to the zealand and for example it knows to translate what's up in a question and then because this sentence is a canonical sentence our parser can turn it into a json uh yeah in the json so get weather with the open weather key the twilio sms with the what's up to the xalan and then the trigger based on the snowing state of get current weather to the sms so um now it's time to have some fun with this so i think if it is snowing in moscow tell jill a joke via sms it's running yeah so the problem now is it outputted a too long joke which my model can't handle it says something like a snowman walks into a bar and says i like beer okay it's not that funny but the thing is um there what i would like to illustrate is sometimes people just you don't have to literally input what you expect the model can make um assumptions and has a lot of prior knowledge into it so maybe it tells you a short joke via sms it's parsing it's parsing it's parsing did i process submit okay it's still outputting the same joke okay it's not very funny but maybe let's try to do some more complex things so if the weather in the uh capital of belgium is is raining um some i don't know uh sand kyle a wake up message okay so these rules do not really make sense but it's to illustrate the technology um and here you see that it even correctly translates the capital to belgium to brussels and that's really not only processing i initiated so even based on and that's really the amazing thing here for me you're only giving four data points typically in machine learning it's completely worthless don't even try if you have less than a thousand data points but just based on the on smart use of the prior knowledge of this model um with four data points we can do things like i say the capital of belgium but it knows it needs a city name so just translate that to brussels because it knows the capital of belgium is brussels for example let's say i would like to send a message via mail there are absolutely no examples of um email use in the database so let's let's let's see how it'll handle it um so i'm hearing echo grants but you meet yourself no problem so um it works but it actually does something very smart it uses the same format it says sends email to somebody with a message the problem is um there is no email sensor yet in the back end so maybe i'll now walk you through how you can add a new sensor and and get this new capability um here my screen so i have here this um i i downloaded the json for the mandrel mail sensor um and i've written some codes to support the email sensor so basically the only thing what that this code does is once you get this string say how to parse the string and how to interpret these words and then i also need to register it that's just because my code is not optimal now i'll build again and i will see if the mail sensor will work oh i need to import this also okay [Music] this should not take too long but so basically the idea is it's it's somewhat a drawback that you still have to write a bit of code but if you can really define what type of sensors need to be accessible over voice for example what kind of sensors and rules a factory worker will be using for example in the production line you can implement these in advance and then supply a bit of rules and then you'll know it'll work quite robustly and i think you'd rather have a robust working on a limited set of sensors than than than shaky working in general so where is my demo here let's refresh this let's uh hope that it now knows how to work with email yes so the sentence sent email to kyle message wake up it now processed it it used the manual mail sensor but again there's there's not a single example about using um males in the database so i i can go on and on i think there there are some really um i think i have uh for example so i'll do something like essential awake a message you have a typo but we saw that's no problem i i don't think it'll be a problem do this only on monday so monday i don't know which sensor works with monday um and i also have no examples of it but let's just see how it handles okay so it it it just dreams up if day of week equals monday then and also the typo didn't matter since digital message wake up so basically the only thing i need to do now is in my codes link the day of week through a day of week sensor and i'm good to go so let's add some noise some noise okay and this would be very hard to parse without any uh good thing smart things going on and it's here it it did something here it it it starts to fail so it says it it removes the noise but it dreams of uh it starts talking about if the weather in paris is also rain so this is the um the drawback of a live demon because it didn't happen 10 minutes ago but okay it's work in progress um maybe if i fix this typo i don't think so no okay so here it um it good it's not performing well which is you know to be expected because you would recommend your typical inputs to be a bit cleaner um i think i don't know how how much time you have left i can i can show a lot of cool capabilities that it still has with other sensors yes because again we'll record it and i mean because this is one of your last days so uh maybe some bad to just go over that just go ahead sure i'll maybe just try the speech input first so uh the speech inputs you need sometimes need to try it a couple of times there are ways how you can guide it so i think in further research you can really make the speech work better but now it's really just out of the box so [Music] if the weather in ghent is nice tell david hello via sms okay so i have a couple of alternatives but this one translated pretty good submit um if okay yeah it's it can't handle the word nice it needs to be sunny yeah well if you give actually one example of nice and sunny it starts translating nicely to sunny indeed if the weather in ghant is sunny send david a hello message via sms sunny david here okay it dropped the scent but i don't think that will be a problem ah where is it in gans this sun okay didn't translate well if the weather in gantt is sunny then send david a hello message via sms send me dancing okay submit uh i don't know why this is not parsing because this cannoli is good yeah um but okay i was making some last minute changes to the code but anyway so so the the most difficult parts went well and that is translating it into something canonical um and and and you can you can start going going crazy so you can you can add let's maybe add something more complex [Music] maybe here i have i have a rule pre-written so here i can input rules to the system so let's maybe add something within between statements so this is again how i expect the natural language to be and this is how i just wanted to represent it so i'll add this rule and maybe let's also add something about humidity and rule so again i'd like to stress that this entire technology is now leaning on five data points which is nothing so if the humidity in the capital of france i don't know if it will work with exceed i don't know let's see 80 and the weather in paris okay where is this sunny uh tell this to me let's see let's hope okay wow so it says if the weather humidity empire is above 80 and weather stayed in paris equals sunny then sent sms to me high and i said tell this to me hang it set the humidity is above 80 so so your input is really quite bad like this this is not a good input to describe exactly how the rule should work but it knows it needs a message and it takes some basic knowledge about language to to just make an informative message so i personally think that that's um quite impressive and so you can imagine that that now you input this in the way you get it back and then you can you know rewrite the message or or add something else maybe [Music] make it an or statement there's no or statements in the database let's see if it works okay basically no yeah sure if you remove paris with it no you still refer to paris good question i hope so yes well um maybe something else like um if the weather is sunny [Music] send kyle a message about unicorns the weather in guns this would be really difficult to parse with normal things but it knows okay weather is sunny yeah i'm talking about the weathering gun so let's do the weather in dorado in san francisco let's hope it now changes it to san francisco okay so you can really use free-flowing language and i think this is really the best approach for again based on almost no data five data is the same as zero data in machine learning to have something that can kind of start interpreting humans and the interesting thing is that this is really built upon a general technology which i know is getting developed further by companies like openai google and facebook so you can just i think you can just copy paste this technique to the newest model every time like we're not even hosting the model we're not even retraining the model we're just pinging an api so um i think this is the most robust thing um you can have and also it said uh unicorns are real okay uh enough playing uh if anybody has a request but else i'll i'll i'll wrap up my demo yeah thanks thanks carl i'd suggest i mean that okay of course we're going to have a follow-up meetings and you know to discriminative technology and uh maybe record a little bit more of this but i mean i have to say it's very impressive what you've done in five weeks absolutely impressive it's just i i didn't know you that it was possible to do these things together well it um it's not possible for a lot so it wasn't possible a year ago i would say this is this is just to give maybe a bit of um a bit of history typically in machine learning the first thing people did in machine learning way before i even knew about computers was engineer features make better features and then design an algorithm on these features then people were like okay let's let's let the algorithm make features we'll just gather more data then recently they said well let's just pre-train a model on a lot of data then inject a little bit of our data and let it work and now with these models you are because this model this costs 12 million to train so that's why we are not trading it that's why a company has trained it you just have a model so big generally trained that has a lot of these capabilities and you just ask it questions so this is basically just this is just text we inputs and then we ask it predict a next word predict the next word predict the next words and this coherent thing comes out which we can parse yeah amazing thanks kevin thanks very cool uh yeah cool all right so i have and yeah is there anything specific in [Music] or or where is is the language included if you would have pre-filled your your rules actually with uh dutch sentences would it have worked well um so where is it encoded uh the language the the model is just trained on english however it is um they're working on making multilingual models out of this so so trained on a lot of languages at the same time and actually these models if you they have they have actually just read so much of the internet that also some dutch and some french sentences are scrapped in so if you ask it please translate the following sentence to dutch and you give it an english sentence sometimes it just starts outputting the dutch sentence um while it was not specifically trained to do that but it mostly now works on dutch because most of the interface uh just how to close can we try the dutch settings to see what would come out of it i have no clue i don't even know how to um oh maybe just say okay dutch like okay i'm trying it now but my um your excitement or disappointments i yes yes you will see my excitement or disappointment live from pro yeah okay um i don't think this should work again it has read a lot of data so australia is via sms submit if if weather stated in paris equals sunny then sent as a mass to villain message the weather is sunny muy is um so yeah the thing is they just basically downloaded the entire internet and just train it on it so it has also a lot of dutch um this was this is also the first time i tried this but again to show how how powerful this technique is and maybe also how scary i don't know you decide we will make it scary don't worry it's scary

Channel: waylay

Views: 324

Keywords:

Id: 8pckQSoJSss

Length: 26min 20sec (1580 seconds)

Published: Mon Sep 27 2021