Document Processing with AI Builder in Power Automate

[Music] microsoft ai builder in power ultimate is so easy let me show you how to get started or actually i call the biggest ai builder expert joe fernandez hello joe and welcome to the stream maybe you could introduce yourself hi anders yeah good morning and good good evening or whatever you are in the world uh thanks so much for having me indeed my my name is istio fernandez i'm a product manager in the air builder team which is part of protomate and i'm i'm super excited to to here to share uh what uh what a builder enables you to to automate you using using ai and i just want to say as well understand i'm a huge fan of the work you do on how you empower so many people to automate uh their careers through through automation and you even have so i know some folks who have considered switching careers uh through your videos so super excited here and i hope i hope this session will be useful to anybody watching this thanks a lot joe and welcome to all you viewers out there this is a session for you joe will show you how easy ai builder is we will answer every question you may have just post them in here in the live chat and i'll give them to joe i'll just probably just sum up every five minutes or so with all the questions and andreas uh is with us from a train somewhere in sweden hello andreas nice to have you here and nice to have all you other people here joe i know you have a presentation for us and more importantly at demo you know we like damos here on the channel so i will switch to that awesome let's let's let's get started uh so indeed as as as as you're saying to i have some few slides to get started to just give a brief overview of ai builder but i need the fun part is a demo so i'll go quickly through the introduction so everybody has the same context on what is their builder and what it fits within microsoft and then the best way to see the product and what it's capable of straight demo so let's do that and as and as you were saying others will uh will take questions throughout throughout the presentation any any questions it's more than welcome okay so ai builder is part of the poor platform so for those of you who don't know the poor platform is microsoft's local no code platform which means that is a suite of applications that you can use to build digital solutions without having to have technical knowledge without having to know how to program without having to have a background in computer science or data science they're available for anybody to build these solutions and we have five big applications the first one is power bi which allows you to build the dashboards and and models over your data for apps it's all about building mobile web and desktop applications using a very simple interface uh as is as if you were putting together a powerpoint then we have which i know many of the viewers here will be familiar and know of is is our automation or local automation platform i will see throughout this demo as well how how it works then we have power retail agents which allows you to easily build chatbots especially like when you think of those chatbots we see in support scenarios you can very easily build chat boards and integrate them in a large number of different communication channels and the most recent member for platform family is called power pages and it allows you to build uh public facing as well as internal company facing websites again in a low code no code way and we have a number of share components across the platform the first one is data connectors which allows you to connect all these applications to more than 700 services so both of course microsoft but also third-party non-microsoft ones so you can connect to a very large number of data sources then we have dataverse which is our no code database database solution so if you're looking for a place to store your data and you don't want to take care of a worry about scalability security and all that we have a great solution with drivers and the topic that brings us together today is a builder which is all around easily infusing ai capabilities into your applications and automations can you can you use ai builder in all these five applications or is it only reserved to some of them that's a that's a very good question so today it's mainly uh available through powerapps and protomate porous and protomates also integrate with the different applications so it could be a solution for example of you could build a website with four pages that every time somebody uploads a document it triggers a flow with protomate and then calls a builder so there are ways to also use a build through other different applications but you will mainly find it today within proatomate to add ai into your automations and perhaps to add ai to your applications cool and we have a lot of people from around the world here we have mariska from south africa we have mckeel from united arab emirates and then we have felix from ghana trailers from netherlands s from belgium hunts peter from germany edward from south africa and we actually have a question already john yes is there a free trial of this can we get started with this quickly yes yes this is a very good question there is a free trial and i will see uh i'll also share the links uh throughout the presentation uh if you log in into port automate and pour apps so both pro automate and perhaps have free trials you can start for for free with those applications and then uh sorry have it here so everything what we see here today we can get started with that for free that's important to say right yeah yeah cool so ai builder so ai builder is a suite of ai models that you can easily use in your applications and automations and in a very uh point-and-click interface as you can see in the screen the screenshot and we'll see also in the in the demo you don't need to have a background in computer science or data science to be able to to add ai easily uh to automate a lot of processes so we have um we have five big categories of air models today in airbuilder the first one is around documents so everything around extracting data from documents and i'm going to deep depth specifically on this pillar in this in this session today then we also have a category around language so we all work with a lot of text whether those are our emails or chat messages and so on and there's a lot of knowledge in those texts so we can use ai to extract that knowledge so we can do things like the text sentiment in a customer email and if for example the sentiment is negative give it a high priority we can translate text we can for example we have a model called category classification which allows you to for example receive emails in a shared mailbox and you want to route emails to the right person you can build a category classification email to understand the content and forward that email to the right subject matter expert to take care of that of the topic the third pillar is around images which is all around being able to detect and count objects within within images and last but not least we also have a pillar around structured data which is if you have data in tables and you want to predict some future outcomes so for example you have historical data on your customer sales and you want to predict whether customer is going to turn or not you can very easily build a prediction model that's going to help you get those those insights so these are the five big pillars of of ai capabilities that we have today within builder and if i jump something funny with my presentation today uh if i zoom into the document uh aspect part we have a lot of capabilities around extracting the data from documents we can extract data from structure and unstructured documents and cover what that means very soon we have a number of what we call pre-bin models for common scenarios for example if you work with invoices we have an ai model that is able to automatically extract all the key information from invoices without having to configure anything i will see that in the demo as well you can you can build your own custom models so if you have your custom documents and you want to extract specific information for your documents we have the capability we support a lot of languages or 70 languages we support pdf images we can extract tables complex tables and and so much more and then because we are part of opportunity we also benefit from the connectors ecosystem so you can the air model gets used just so far you're able to extract the data but then you probably want to do something with the data so with protoman you can do a lot of transformation and you can push that data over to 700 connectors and through our rpa capabilities with port my desktop you can also push the data to any basically any any legacy system or desktop running application that there might not be an api connectivity but you might want to push the data and we also have what we call a document automation starter kit which is basically a one-click install solution where you can set up an end-to-end process and then easily configure it i will see that as well throughout throughout the presentation so some some very quickly some new features before jumping into the demo so we recently released support for what we call unstructured documents so up until recently ai builder was optimized for structure and semi-structured documents so these are documents that always have a same layout so for example a tax form or have for a specific layout the information is in similar places so for example things like invoices where you can always expect to find the invoice id somewhere like a table item somewhere the total amounts in a specific location what we now release is support for unstructured documents so an example of social document is a contract where every contract is different there's pages and pages of paragraph and you want to extract specific information from those paragraphs now with vi builder and using the simple interface that we'll see through with the demo you can very easily build a model to also extract data from these type of unstructured documents and last uh last update that we've released recently is now when you train a custom document pricing model we're gonna give you performance metrics so that's gonna tell you whether the model or not is ready to be used in uh in an end-to-end scenario if it's not we're going to give you recommendations on how to improve it so our goal with ai builder is to get you step by step to build the best model and to guide you on how you can uh once you have that model how you can improve it over time to to get the best results and automate as much as possible all right and i will interrupt you for one second as doubt says hello from netherland and a special greeting to joe so thank you for that levita says hello from new zealand nicholas is joining us from france scythe is from turkey and my wife is even watching steena is joining from denmark i'm not i'm not sure where she is she's not home so some probably some cafe or something tanger is from singapore i have uh questions for you but i'll i'll bring them to you in a little while in the demo so let's just continue joe perfect yeah it's awesome to see so many people from uh from around the world um i'm i'm based in france specifically specifically uh i'm gonna say hello to nicholas in france uh all right so let's jump into the demo let's see the product in action uh so this is the home page of ai builder and you can find it if you log in into proatomate.com and you log in with a work or school account you will see the air builder section in the left navigation bar you'll see here a homepage which is called explore and here you can see all the different ai capabilities i was showing in the slides before organized by different inputs so documents text structure data and images and this page also offers so much more you have tutorials so if you want to get started and you need a little bit more help we have great tutorials here i know the anders also you have some very nice videos i'll also uh reference those as well i mean that your your material is also very very valuable to get started with for tomato and any builder so that's some great material out of these six videos i would definitely pick number six i can see that one to the right and then before we get to the video so we also have case studies but what we also hear from from users that sometimes they want to get inspiration on what they can do with ai and here we've published a number of common use cases with the amount of customers so if you're looking for inspiration on what you can do you'll see some case studies here and we'll explain how to how to build those indeed we also have a great video gallery so if you prefer video i think more many of the viewers here do prefer video that was a good collection of videos here customer stories again if you're looking for inspiration on how real customers are using a builder you can go ahead and look at some of our customer use cases using a builder and then we have a what's new section if you want to catch up on what's the latest updates with the ibuilder you can also see that through um with with this page so let's uh let's start with documents so these are the different air capabilities we have around documents so we have a number of what we call pre-built models so these are models that you can use right away out of the box without having to configure anything so we have pre-built for invoice ocr so if you just want to extract all the data from the text we have text recognition receipts ids like passports and driver licenses business cards and document browsing which is the model that's going to allow you to build a custom model to extract the data from any type of document when you click to any of these styles you get this experience so you get more information on what you can do with this capability so here you get the introduction of what the invoice pressing module is able to do and you can also see some examples so you can get inspired to to understand what the ai is able to do and you can even and for free you can go ahead and try with your own invoices so uh if you don't trust uh these these samples you can just bring your own real samples and see how they work so for example here i have a fun one [Music] which is a totally handwritten invoice so one of the things that we also supported in a builder is handwritten text so if you have a mix of printed and handwritten ai builder is going to be able to recognize that and here we see an example of of a totally handwritten invoice and we can see that without having to configure anything a video is able to extract already a good amount of good amount of data and it would if i if i were going to handwrite another invoice it would work with that as well so it's not that this is a predefined one right definitely definitely it's it's it's a it's uh it's going to look for command fields within universe it can be any any english layout yeah cool so let's let's see this uh in in in action let's see how we build the automation through this so uh if you want to use invoice we are by using a flow button here so you click on it and we're going to tell you we're going to take you to a pre-built for automate flow template so this is an end-to-end automation uh already built for you that they can you can then customize uh so we give you a simple example so you can get inspire on on the possibilities but then you can customize it to whatever it is so we'll start with a very simple example where basically we're going to trigger an automation by clicking a button it's going to ask us for an invoice file and then it's going to extract all the data from the invoice or some of the data elements and it's going to send me an email with those elements so very simple automation but it's a great way to just see the whole entrance working and then get some confidence and and be able to customize it then because this is for automate you can then add any actions so these are we have all 700 connectors so you can then customize it let's say that you want to to push that into excel or put your data into a sql database or anything you can think of you can if we have a connector for you to be able to achieve this so i'm not going to modify anything in this first example i'm just going to save it and run it to see to see how it works so i just click save on top right and once the flow is safe i'm going to be able to click on the on the test menu so let's go ahead and click test and because the manual button i'm going to click manually and then it's going to confirm it's just going to confirm that all my credentials are okay and then it's asking me okay i need an invoice file so let me pick another invoice let me pick this one here and run flow and these things would happen automatically if i were going to process these things in my company right correctly so what what you're going to do so this is a very simple example where you need to manually trigger the automation but you could change the trigger to for example every time i receive an email i want this to happen every time somebody adds a new file into a folder i want this to happen uh so yes you can you can customize it so for every event every time a new invoice is received at your company trigger this this automation cool so this simple flash run so if i go to my inbox here i see that i receive an email and i can see if i open the invoice i can see that all the data has been extracted the invoice id the invoice date we also do a number of nominations so we're going to provide you for example for dates we're going to give you the date as it is written on the invoice but we're also going to give you in a standardized format so it's also easier to then work over with them similar with the with the numbers with the with the amount so we're going to give you the total as it is building in the invoice as well as a normalized value in the in the format and something we've also included in the in this template is if you want to once you have run that and if you want to learn a little bit more we also give you a documentation link so you can go ahead and learn how to take it to the next next level wow that's cool that's very very easy to process an invoice like and we could do it in two minutes definitely so talking about the next level let's do something a little bit more more complicated uh so in this case uh let's imagine that we work at a company and we receive these documents which are sales orders and every time we receive these sales orders somebody scans them and put them into a sharepoint folder here and then what i want to do is every time a new file is here i want to add the data into this excel file and then i want to also write this data into this legacy all looking website and today i do this manually and it takes me a lot of time because i receive hundreds of these sales orders per month i want to be able to automate this so i can do more more interesting uh things throughout uh throughout my day so let's see how we can use ai builder and broad to me to achieve this so we're back to ai builder and in this case um so these are sales orders so we don't have yet a pre-built model for search orders but that's no problem because we can go ahead and build our own model to extract the data from from our own documents so this is called document pressing so i'm going to click on it here and again we have the familiar experience go ahead and us so a custom model is that when when we have our own data is to repeat what you said or to understand it correctly correct this is when when you have uh when you have your own documents um and you want to build a custom model to extract the data you need from your own documents you go with the custom one if if you have one of the pre-built that we support so invoice or receive for example you can get started with this if it works then just stick stay with the pre-built it's going to be easier it's just going to as we showed them before like in basically two minutes you have it up and running if you need additional information from for example for an invoice or you have a totally different document you can go ahead and build your own uh custom document version model so for example if if the invoice was in spanish like you speak or if it was in in danish like i speak we would we will create a custom model for those ones or could they work within the predefined one so that's uh that's a that's a very good point uh thanks for raising that one um so today the pre-built invoice pricing works with english invoices from the us but the good news is that next month we're gonna massively expand to amazing basically all the main european languages so french spanish german italian portuguese dutch i think i did not forgot any other and to the viewers out there joe didn't ask me to ask this question this was a question coming from myself so that's really nice to hear too looking forward for that definitely that's that's also the the the the video of of ai builder and and build uh being a cloud service that we do update the service required so you can expect these models to just get get better over time and add support for more languages and more markets cool um so yeah let's let's see how we build our own custom model so let's go ahead and get started when i click on get started we're going to enter into what we call a user experience so it's going to guide me step by step to build by my model and the first thing that the wizard is asking me is what type of document is it a structure document or one structured document which as we were saying before our structure is is a new type of document type that we support here in this case because these they do have a sensor structure so we see that there is there is a table there are a number of elements here uh these are uh these are these are same structure document so i'm going to click semi structure let's move to the next step and then the second step is going to ask me what information i want the model to extract and i can define to extract fields to extract checkboxes and also to extract tables so let's do a number of fields let me go ahead and define a number of fields so let's do for example the sales order number i also want to extract what is the vendor of the sales order so i'm going to define this as a field i also want to extract the total amount there's also i believe in in one of the providers here there's a checkbox to specify if it's priority shipping or not so let me also define a checkbox so we can know if if the shipment was done in a priority way or not so priority we are creating shipping [Music] and then we also want to extract some of the data from the tables so i'm going to go ahead and define a table and let's give it a name so we can call it for example items and then we can define which columns i want to extract i don't want to extract all the columns i just want to subset of columns so i'm going to do for example the the description we want to extract the part number let's do also the unit price and the total price drops let me write this correctly all right uh done um any any questions at this moment than there's uh no but i want to say to all you people out there if you like these sessions with experts like joe you can give the video a thumbs up that will help the channel a lot we can get joe and this channel out to more people so yeah back to you joe definitely yeah uh okay so we've said that uh our document is of type semi structure with the final number of data i want to extract then let's move to the next step and here's going to ask us to upload sample documents so the model needs to learn from some sample documents and the way we upload documents is through collection and collections are basically a group grouping of documents that have the same layout so for this demo i have two providers i have adatum here and contoso and they have two different layouts so i'm going to go ahead and define two separate collections so i'm going to start with the first one and i can give it a name let's say i'm going to upload here sample documents from ada2 and i can upload documents either from a local device from a sharepoint site or also from a from azure blob blog storage so we'll see that's that appearing soon there we see so i can upload from a device sharepoint nashville storage so i'm going to pick my device i have here for that doom i have five sample documents and five are i just uh what you need so if you have fan several documents you can get started to build your own custom model i have a question very related to these things here the viewers they definitely want to try this out after the session can we get these invoices somehow to so we can try it definitely so let me open here if uh there's also a link on the on the on the homepage we're looking before but if you go to the air builder documentation uh in the guest store section we have a sample data section here and here you can download sample documents so here we will give you samples of both the structure semi-structure and social documents you can try all different uh document types even here and if you want to explore also the other ai capabilities so for example you want to explore what we're able to do around object detection or text we also have a lot of simple data for different model types cool thank you joe yeah good question okay so we've uploaded five samples from the force provider let's upload another five samples for the our second provider so i'm gonna repeat the same process i'm gonna this time it's contoso so let me take this five here okay and you can also give it a name so it's easy easy to reference okay so third step completed we've uploaded uh five sample documents for each of the providers we want to to automate and then what is left is i need to teach the model where to find all these all these data so the air builder is not going to show me all the simple documents that i have uploaded and basically what i need to do here is to go ahead and tell okay these are all the data i want to extract to i just need to tag what uh what where the different elements are so the tag is very simple i just have to click on each item and say okay this is the sales order number you can zoom a little bit so it's easier to see the vendor i can do this so this is item industry and we can see even if it's a part of a logo i can also also tag this so i'd like to total amount it should be around here so let me go ahead and tag the total amount and then for the for the table uh i just basically have to draw a rectangle a rectangle around the area then maybe adjust it a little bit more here specify that is the item stable and then basically i need to draw the separations between row and columns so i'm just going to draw so in this case we have two rows now we have a number of columns let me go ahead and define those here and then what you're doing here joe is showing ai builder the structure of these documents and then you do that for five documents and afterward the ai builder would be able to automate the things without us have to look or click somewhere is that right correct and it it's it's it's it this is an initial process where we teach the model okay about our documents and where to find the information once i've done that then the model will have learned and it will be due for any future documents for us automatically cool so i've tagged the columns i thank the rows and i just have to define where what each column corresponds to so here is the unit price here is the the total price and here we have an interesting case where we have we want to extract the description and also the the part number uh so i'm not able to here to tag both because they're both in the same column but uh even though airbuilder is in no code as you can see we also give you all the tools to go as as complicated as you need so we have an advanced mode where basically if i if i toggle here the advancement i can just go ahead and and really fine tune what i want to extract so here i can say okay the description it's just this part here and the part number is this part here and i'm going to do that for the second row as well so this description here and the part number and i'm done and uh if i go here and click preview table i can see that this is how the table is going to be extracted which is how i want it i want the description in one column and the part number in one in another column so even though in the document they're all in the same column as interval layout i can decide how the model runs to extract all this information cool and then the the last time is the priority shipping checkbox in this case so for this provider ada2 there's no checkbox around priority shipping but that's not a problem what i can say is hey for uh for this collection you won't be you won't find this this field so just don't look for it so i can i can define a number of fields but not all the fields need to be present across all documents if onefield is not present i can just say hey don't don't look for this field in in this document so i would do this for the rest of the documents in atom let me just do one more for the second collection so for the conductor collection here it's a different layout i just have to teach tomorrow where each of the items are so i'll see here the list so i just have to uh say that there is a sense order the vendor is here so it's total amount tables i will repeat the same process so i select area here items and again i just draw the separations between rows and columns and this table is easier since there's no nested item i don't need to switch to advanced mode i can just tag all the different uh um can you repeat what was the nested item again yes uh so in this case there's no nested items so every column is clearly separated so there's no there's no issue if we go back to the previous collection to the to the editing collection um let me again we see here that let me zoom here we see here that the part number is nested within the description so within the same column i have two things i want to extract i want to extract the description and part number but i want to extract them separately so through the advanced mode we can we can do that cool and then the document that we were doing before uh so we do have a checkbox here so checkbox is also very simple i just draw around the area it tells me that the checkbox is checked i said it is practice shipping and i'm good to go so i would i would do this process uh through the five documents and once i i've done i've tagged all documents i can go ahead and train my dock my model i i won't do that to speed up the demo to don't so you don't have to watch me tag all these of these documents i have trained uh the same model with the same documents and these were kind of here i have a question jill yeah when we see these these are scores to see how accurate our model is but what is a good score what is a bad score when should we retrain our model can you elaborate a bit on that definitely very very good questions randy so after you train a model you're going to land into this page and we're going to give you this discord uh this is the indication so excellent we are confident that you can take it to a production environment and it will do a very a very good job good it's is uh you probably want to add a few more documents but it should be good to go uh uh to also using in the end-to-end process there might be some exceptions so you might want to put some session process in place where you for example you press a document and there's a lot of confidence you might want a human to review that before pushing data into another system anything before good so anything before anything below sorry 90 we probably want you to uh add more more documents so it probably means that the model is not very confident that's going to do a good job and you will see so in this case everything has a very good score but if there was something with a bad score you will see that when i click in more details is going to give me for each field and for each table the accuracy score and if there was a local purchase score it would tell me okay if i imagine that sales order number had a low accuracy score it will i will tell you here that hey you might want to edit the model add more documents for the cells or the number and tag more and more documents with with many things in ai the more data you bring the better the area is going to get so the recommendation probably will be we're not very confident how to extract ourselves or the number so just come back uh to the model add more documents tag more samples and and and that will be it or another possible way that we also see that sometimes uh you might have tagged the sales order wrongly in one document and the way so the model is confused like hey like at some point the search order completely changed so you also might want to go go back and review that everything has been tagged correctly but here we will in this in the screen because everything is green i don't see any recommendation but if there was any low accuracy score we will then give you recommendations on how you can improve the model to get to the green step so to to recap your question if it's green it's good to go anything that is not green you might want to review and either add more documents or review that the tagging has been done correctly cool thank you and then to if to edit them all it's very simple so if you need to make changes you just click on edit the model and you go back to the training wizard that we saw that we saw before can you see our models across teams you can share so that's an excellent question as well you can share so today you can share the model with people in your company uh indeed across them within your company to to use it uh today they can only be with there is one limitation that you can there can only be one model owner and uh but this is something that we're working on to us to address in the in the coming months so you also be able to have multiple owners in for a single model but if you want to share your model with many people in the company to use it in automation process you can you can already do that through the share panel yes cool so a couple of more things you can do in this page so you can test it so once probably uh what we also recommend is aside from looking at the currency score you want to test whether the model works or not so we have this quick test feature which allows you to basically take a document that you did not use for training so it's important that you use a document that was not part of the training set and you just adore it here and we're going to be able to see what the model is able to extract and also it's going to give us the confidence score for each of the items so we see that he's able to extract credit the vendor the sales order number and we can see that the confidence document score is very high we can also verify that the table is extracted the way we want to be extracted so everything is put circuit i can then go ahead and use my model in an end-to-end alteration if if i were to get the bad results then what i probably want to do is go ahead edit the model and add more samples to improve the model over time good so uh to recap so we started this journey because we have this uh issue of that issue but we have this process where we receive these sales orders we add them into uh the into a sharepoint folder and we want that every time a new file is added to the sharepoint folder we want to extract the data uh write it down into excel and then write the data into this legacy website so we now have the ai model that does the job to extract the data but now we need to push the data so for that we're going to use the model so i'm going to click here in this model and there's a number of ways we can use the model one is through bora automate so we have a number of pre-built templates for common scenarios you can also use your model in a poor app so if you if you want like a more imagine that you want to build a tablet application where somebody can take a picture of a document and see all the data from the document you can very easily build that using uh using a horror app or we also have what we call our document automation solution the documentary starter kit which is basically an end-to-end solution uh which includes a validation station that you can put in place in you know in a few minutes for this demo i'm going to pick uh portameade i'm going to pick build an automation using protomate and here we see some common scenarios that we see among among our users so documents it's going to ask me where where are the comments coming from they can come from an outlook account a share mailbox a sharepoint library a onedrive or of course there's an other option if you have any other source of data we can also support that with protonate in this case our use case for this demo is sharepoint so full uh sales orders are added to sharepoint so i'm going to pick sharepoint document library and here it's going to take me to again a pre-built template already set up for me so here's give me a description of what the template does it's just making sure that i do have access to all the services so i have x2 to outlook and sharepoint and once that is verified i can go ahead and proceed to the next step cool and to the people out there if you don't have a sharepoint if you for example have a school account you can use run one drive here as well that will work exactly the same way correct yeah yeah um so here we have the the port automatic flow already set up for fresh um the first step is what is going to trigger the flow in this case every time a new file is designed to a folder it's asking me some from additional information so it needs to know okay what sharepoint folder is going to be so i'm going to specify we have here a sharepoint site called northwind traders and within this sharepoint site we have a folder specifically to uh store the such orders i'm gonna pick that here then we have some uh the second step is just some some guidance on how to use this template uh so i'm gonna walk you through it so don't don't need to read this today then the next step is to call the airbuilder model that we just trained and it's already set up so it's already set up to the vmware live stream and what it needs is to what type of document it is and the actual document and this is something that sharepoint is giving us back so streaming is going to return that is a pdf and the actual content of the of the form you can also do also for more advanced users uh just to show you you can also configure to only analyze specific pages so for example you have a 100 page document and the content is only in the first three pages you can just uh speak to the first three pages to be analyzed and that way the process is going to be easier and you're also going to be you're going to save money in in the process and then this template is very it's very easy easy as well it just sends me the data back into an email this is not what i want to do so i'm going to just i'm going to just going to remove these steps and we're going to add the steps that we will actually want to do so first step we want to push the data into excel uh so let me go ahead and add a new step here we see all the different connectors that we have out of the box for protomate and we can see that there is one for excel so i'm going to pick this one and what i want to do is to add a row into a table so i'm going to select the arrow into a table action and again here's going to ask me a number of questions so where is the excel file in this case it's again it's on my sharepoint folder so let me go ahead and pick the northwind traders sharepoint site uh where is the folder uh documents where is the file uh it's here so says orders excel and if i have multiple tables it's gonna ask me okay in which table should i insert the data in this case i just have one table the excel file and now it's going to show me all the columns i need to fill so when i click on each column for automate shows me all the data that is extracted by the model so we we of course give you back the the values but you can also do things like confidence score so something that you users use confidence score for is you can put conditions you say hey if the confidence score is below 75 i want to trigger a manual alert so somebody reviews the document before i push the data into excel or any any other system joe one second i think uh your connection lags a bit so not not a bit but i'll just re-re-um refresh your desktop and then i will take some questions because i can definitely hear you but um so i'll just do this yes perfect yeah yeah yeah and um what we want we take some questions and then we come back to to the demo so um perfect and and to all you people out there thanks for all the great questions we will take them as they come and the s is saying is it possible to accustom with some js css to jscss so i my javascript css yeah to custom the i'm sure you mean the ai models is can we custom them with some code i believe that what's meant otherwise corrector says okay so uh to build the model so it's it's how it's also it's using that that interface so point and click so uh no no no need to write to write code if you want to do uh something uh more complex so let's say that you once you extract the data you want to transform it in a specific way and so on you can do that through protomet so here i'm just adding like simple actions like excel and we'll also see how to push the data but if you're a professional developer you can for example call any http service so you can say i have a service or study in azure or any other public cloud or anybody will have an http connection you can basically call any any advanced input endpoint and and do any transformation with code so there is extensive if you need to uh have a code service and you want to call that code service there are some ways today to do that yes cool and to the people out there just let me know if it's my screen can you actually see joe search did it could you see joe search for that http because i can see it in the small window but not in the big one let me know in the chat please and then we'll get back to that nevita we have a lot of great questions joe so uh nivita looking for multi-page support in document processing how long will he or she look very very good question so multi-page so if the document if you have a document with fields that so we have a multi-page document that fields that is supported today so you can process uh a lot of pages in with the builder apparently the question is around multi so tables the spam calls multi pages so that's also a common ask from from our users today we do have an experimental feature i probably have if you recall when we got through the wizard and we were in the step where we defined the data there was an option to extract single page tables and multi-page tables that one is experimental which means that uh we know it works for some cases and for some others we still need to to do a little bit more work the good news is that that multi-page table is going into a stable state in basically in november so in in a few months from now so you can expect to see some some great improvements in this area in in freddie shortly it appears you are on point what uh me and the viewers we demand so another feature that will get implemented uh right around the corner joe it appears that it was only me who couldn't uh see you up at the screen so my bad to you george and to to virus but uh sally says hello from turkish felix says hopefully some new knowledge i already got a lot out of that steena says nice office um prakas says hello from india now we have another question and hello to all you viewers out there thanks a lot for joining we have a question here can we build does this pack have intelligence to identify different values from ideas where the data is not consistent can we and again can we mask the confidential info good good question so with the new support so with the unstructured capabilities that we released fairly shortly now you can build a custom model to extract data from documents which are brilliant structures so where data can be all over the place from one document to another so that so that is definitely something that that we can look into in terms of masking custom personal information we don't have a service to do that today within within a builder so what we ask you is if you don't want to train models with custom with confidential information you can just generate a copy of the pdf and mask that confession information so so we don't don't look at it another way to do it is if you want to train the model but you just don't want to extract the data here you could add a step to remove that data so you can say hey for for every time you receive an i don't know uh a customer name and you don't want to store that you can just discard it so you're so you're not you don't have to store that data anywhere um so the there are ways so today what what you bring is what we're going to extract so we don't we don't remove any data from from your documents but it's then up to you once you have the model you can then decide if you want to use the data or if you want to discard it and don't save it anywhere so that there are ways to uh to not do not uh use uh confidential information that you don't want to push anywhere we'll take one more question and then we're back to your demo though um and eric from portugal rpa consultant says hello hello eric nice to have you here and then the last question before we continue the demo that is um satca says can the exported data from the pdfs also be put into an xml xml file this way we could automate invoices in our company definitely so there are some there are some steps here i need to uh to transform the data from here into into xml so that there is a there's a possibility as well yes cool so back to the demo joe and we will take and now uh make sure that joe will answer each and every question a little bit later in the session so just keep all the great questions coming definitely uh so in this case we're not saving the data to xml we're just sending the data to excel so i i've added here the step to add the n2 into excel and when i click on each of the columns of the excel file a port domain shows me all the data extracted by by ai builder so i can just go ahead and map what has been extracted to the excel file uh so in this case uh just says order number values here i want to mount the vendor so if i search for vendor and i can so for each data we give you a lot of information we give you the actual value we also give you the confidence score if you want to build conditions around that we also even give you the the page number and the locations if you want to do more advanced things or like is it is this field on this page or not you can also you can also do that in this case i just want to show the values i'm going to map the different values to the different columns let me do that real quick determine value priority shipping this is this is the checkbox uh priority shipping value and then these are the three columns from the table so i'm going to search for them so description and what happens here if i if i click on the description value or automate knows that this is an actual table so it's going to put the data into an apply to each tool so it's going to automatically loop through different rows without me having to configure anything so that's that's uh that's very nicely done for me by power automate let's do that the same for the part number and last quantity value there we have it uh okay so right now where we are so we have an automation that every time we add a new file into a folder it's gonna pick that file it's gonna extract the data using the airbuilder custom model that we trained and then it's going to push the data into into excel so we have one last step missing which is we need to push the data into this legacy website so of course because this is on our website there's no api connectivity so there's no uh connector out of out of portame that i can use but there is no trouble we do have and i know that many of your viewers are familiar with we do have automated desktop which allows you to build rpe automation so basically replicate what anybody will do in a screen with point and click and keyboard and what i've done is i've built with proton my desktop a desktop flow to basically take all this information and push it into into into the website uh so i i won't show the process of making that desktop flow but i'm just within this flow i'm going to be able to call the desktop flow that i have already built so i'm just going to say i'm going to want i want to run a flow that i built with product on my desktop i've done a number of automation so the one we want to use for this demo is the one that enters the data into the website we have many of your viewers might be familiar with this term so attending unattended so attended is i want to belong the machine and see the process running and unattended is i want if i if i don't need to be logged into the machine in this case because the demo want to see it running so i'm going to pick the attended mode and then as it was in the case of the excel file it's going to ask me okay uh the the rpa desktop flow is going to push this data just tell us where where what this data is so again i'm going to map what is returned by the ability model into the desktop flow action so again i'm going to go ahead and map the different values so let me do that real quick and everything we see here today we can build ourselves uh after the session right yeah yeah so there's there's uh as we were saying at the beginning of the session there are free trial for everything so for port made ultimate desktop ai builder so you can replicate the same demo today for for free yeah and then we do have so in the in the every documentation we have simple data documents so if you don't have documents that you want to use the the sample data from the from those documents feel free to also download them from the from the ai builder documentation page all right so uh we've built our first version of our automation so again new files come to sharepoint we have saved the data we pushed into excel and we run the desktop flow so let's see it running i'm going to click save i will i will disable our picture so no it was just a figure that i disabled your picture and then disable the sound as well so that wasn't really clever so we'll just see it in like this now where is that that happens that that proves that this is life it's not uh no no no we are running it live okay so now what for automate ask me is okay to see the floor running i need to add a firing to sharepoint so let me switch back to sharepoint and let me push one data so let me take uh from the contoso collection let me take this one for example i'm gonna write it here okay a file has been uploaded so let me go back to the flow so now we should start seeing in a moment we should start seeing the the flow running there is it has picked up the the file point now the app the model that we train is extracted there from the sharepoint folder and once it does that it will push the data into into excel and then uh an rpa bulb would control my machine and enter the data into into our website so now it's entering the data into into excel and i'll switch back to excel shortly so we can see it we can see it okay let me switch to excel there you go there are three three rows in the in the in the in the pdf so you can inspect them and now if i switch to uh so i have here virtual machine that has a portable desktop configured so now at the moment rpa bot will take control of this machine and enter the data into into website uh there you have it so i'm not uh i'm not touching my keyboard and there is entering and centering the data into into a site nice and that's it so it's uh in this case we just did one uh one document uh so so it was very simple but you can imagine if i receive a hundred documents the time i would save doing this is quite significant so that i can then do other more more fun things that just entering data into into excel and into into a legacy into a legacy website that's so nice john so so yeah that was uh that was the demo i hope i hope this gave some ideas and some uh some inspiration on the possibilities of portomate and nia builder uh definitely we do have time we're happy to take additional questions or discuss topics at the audience i will bring all the questions so if you out there have a question for joe just bring it and put it in the chat and i will ask so the first question joe is that can we use these models by going into the ai builder over to the left and then using a flow but could we call a model directly from a flow like could i create a flow and then call the model definitely yeah you can you can you can at any time you can just start from let's just quickly start from tomorrow and then uh when you when we add a new step we have all the connectors and uh airbud is actually the the second connector here in the list so here we see all different emulators that we've extract and and we can pick any ability mode that we want from from here so yes you don't need to start from the from the page you can start right away from the from the flow editor and that is once you have built it so you build it in in the ai build and then you can always use it in your flows as much as you want cool yeah um and i'll take all the questions now we have um navita says i have customize your manual flow and instant flow that was a guide i made with ai builder it's working fine but not for multi-page but you already covered that question multi-page support will be available just around the corner yep cool and we know that novita so that will establish this dutch question mark i think i called you dutch sorry about that doubt if you weren't dutch um then we have steen that's my wife she says he asked do you have a pre-built model for kyc procedures so we have today we have what we call the identity document reader and and here is basically it's able to extract uh data from passports and our licenses so we do see uh for newer customer scenarios we have customers that that are using this capability to to extract so one one common use case is for example you have a process where you need to you need your customers to upload up an an an id to be able to verify uh a signup process you can basically use this model to extract all the information from the id and check that what has been entered corresponds to what's on the id so yes with you with the id document we can use it for know your customers scenarios thank you steven see you later for kovi and thank you joe for answering okay asks uh what languages does it support so it depends on the on the so the the the ui is translated in over 42 languages so so if right now we're singing in english but you can change the language of portomate and you will see it in your own language it depends also that the depends on the model type so some models support more languages than others uh the way to to see that is basically if you if you click on any of the models let's say that you want to learn about the customer that you're pressing there's a view documentation link that will take you to the every documentation and for each of the model types we give you uh there's a requirements and limitation section and here you can see the list of of languages so for example for for uh the custom model we support a lot of languages that you can see here if we look at another moral type so if we do for example the invoice person that we were discussing before today we uh we only support um uh english but as we were saying as well before uh in a month from now we're also supporting all the main european languages so here here in documentation you can see uh for each monotype what languages we support cool thank you joe michael says what if the po has multiple pages again there will be multiple page support in in the very very issue and this is a question actually a great question now from md asks will we get to see a similar similar ocr cv in power automate desktop so can we call this ai builder from power to made for desktop ah but will we see it in the future that is a very good question uh so today what is possible is uh from a cluster so uh there's two type of flows in protonate there's cloud flows so the one that we built for the demo and there's the stuff flows the one that you build with protoman desktop so within a cloud flow as you can call uh i think i closed the the flow but you can call a desktop flow from a cluster so that we see our customers do from a desktop flow today you cannot call an airbnb the model but there is something that is there is on the roadmap i think we recently announced that uh sharepoint actions are now available in protoman desktop so you can expect to see more cloud actions like like like builder coming to bottom with desktop as well in there in the future so yeah that is looking forward all uh us power to made for desktop developers definitely look forward for that uh joe s asks can we trigger it through an api call so airbuilder because it is deeply integrated in important it's it's optimized so when you build flows with protomates so usually uh folks are using the connectors or opportunity desktops having said that protomate gives you a lot of flexibility so one thing that we have is uh what we call the http trigger so here for example if i search for http we can see that we have this request connector and basically uh what this does is that once you build your flow you're going to be able to trigger this flow using an acdp request so you could pull these requests and they're called ai builder um [Music] and then that way you can code and enable the model using an acd to be a request so yeah there is an option using using this technique here that i'm showing there is an option to call uh ability models through an http environment so that will actually be possible from power to made for desktop we could do it in this way to call the ai builder with an api call correct correct cool um everything tic tac says can we implement this into navision so that's that that's a very good uh so with from the vision um it depends on the on the scenario and on the installation i imagine it's an on-prem uh an on-prem uh solution uh if there is api connectivity what you can do is through a cloud flow you can call um you can call an sdp endpoint and push it there after that if there's no http connectivity you can use it with protomed desktop so there are ways so there's no uh an out-of-the-box integration with navision okay builder but either through an sdp request or throughput domain desktops there are ways to push to push data to the navigation and we've seen we've seen a number of users doing that with a number of legacy uh and and on-premise uh crms um so that is that is an option very cool joe uh rush up says we're doing product productive stuff and love from india thanks a lot uh your support definitely help us getting this video to two more and again if you like what you see here if you like it joe's product ai builder and joe give this video a thumbs up that will definitely help the reach of this video so thank you um now um see what's the s asks what's the pricing to run it in the cloud maybe you don't even know joe or so it's uh uh let me give a high level um overview so to run ai builder you need air builder crates um and there's a number of ways to get the upgrades one is premium for automate and perhaps licenses some of them come with crates and if you go to the pressing page of for automate and perhaps you will see how many crates are included on each of the premium licenses and we also have what we call the ai builder addon which gives you one million air builder crates and then the way to compute so to know how many crates do you need and how many ads you need we do have a calculator so if you search for a builder calculator in your favorite search engine um you should land into into our calculator um and here you can specify like okay like how many how many pages you're going to purchase per month how many uh we're talking before about ids how many ids are you going to process per month and by filling this information we're going to give you exactly how much how much capacity you need to purchase per month and one thing i actually didn't know you i think i wrote you joe or maybe one of your colleagues was that you can always extend your trial dai build a trial and then for for private use for developing use you can just extend it and try it as much as you want correct yeah as long as if you stay for for testing purposes you can you can extend it yes yeah so and felix says great information learning so much and then as doubt says [Music] it would be great if power automate can map the different values from ai builder to excel by itself for example by using the same column names like the labels in ai builder great david rankov that is a great maybe a future feature of power to me that is a very good suggestion i think i did like our goal uh with the builder and the whole platform is to make it easier and easier uh for users to build solutions so uh indeed that that sounds like a very good idea so uh that would simplify the process so uh yeah that could be a possible improvement cool navita says can we run the flow build with power automate into a power bi dashboard there is so within power bi there is a a visualization that's called protomate you can add it and from there you can trigger a protomate flow so yes it's it it's possible cool thanks for the question rest it says great session joe so thanks a lot uh can the ocr detect pdf or merge documents ej some vendors sends one doc with several invoices in ah that's a that's a great question as well today so today you do need to separate the different vendors so if you have a pdf with multiple vendors we do ask you to to separate them uh we do have a template that basically what we recommend users to do is uh look for keywords that separates one vendor from another and using those keywords you can split the pdf so there are workarounds around them uh today it does require a little bit of extra work to separate those uh but this is something something that we we are looking to to support more easily so this will also be something in a future update we will be able to support multiple forms within the same document cool joe this has been an absolutely nice session especially to see how advanced it was but also how easy it was to use these advanced features and you were a very great presenter and answering all our questions so it was really nice to have you here and i'm sure maybe in a little in the future you can come back and show us maybe some of the new features as well definitely be super super delighted thanks alexander it's been it's been it's been a pleasure again big fan of of you and your community and yes uh i think uh you do have some great uh channels also your youtube videos your discord community so if there's any questions on the air builder and you can think i can help let me know and i can happy to jump into into a conversation thanks a lot and thanks to all you viewers out there who have been outstanding throughout the session bringing joe great question so we can interact and show some some of the things you wanted

Channel: Anders Jensen

Views: 22,091

Keywords: ai builder, ai builder power automate, microsoft, power automate, microsoft power automate, joe fernandez, anders jensen, how to use ai builder, document processing in power automate, power automate ai builder, ai builder in power automate

Id: tmxvLAT3C7s

Length: 66min 30sec (3990 seconds)

Published: Wed Aug 24 2022