Invoice Processing Solution leveraging AI Builder Forms Processing Model & Power Automate

hello everyone this is residue Ronnie Microsoft business applications MVP in this video we will do a step by step walkthrough of building an invoice processing solution leveraging AI builder capabilities now ai builder is a local artificial intelligence service that's baked into the power platform that can be leveraged both in power automate as well as power apps now approximately nine months ago basically on July 30th last year I had put a video out on AI build a form processing using Microsoft flow in which I had certain invoice documents that I had uploaded and I had created an e about the forms processing model and I received a lot of comments on this video as well and one of the key takeaways from this video was a lot of questions around when I upload my documents and I start training my documents in the form processing model there are certain elements that are not read by the form processing model because the form processing model only leverages key value pairs and there were a lot of other questions regarding parsing the data because the model would return the data with a complex JSON format that required a lot of parsing of the data and plugging the data into your data source of choice now nine months ago but the data source of my choice was SharePoint and because I want to keep this consistent with the same solution that I had built nine months ago and I want to show what's new now and what's available I'm going to leverage that same back-end data source so in my case this time my invoices look like this so I have these sample invoices and in these invoice documents what I have is the invoice number I have the date for the invoice I have the name of my company and my address the person who this invoice is assigned to and the company name of the person to whom the invoice is getting assigned to what are the services that the company has has leveraged and the breakdown of it right here as well as to total balance that is do based on this invoice and so a due date for the invoice so this is how my invoices look like and I have created six sample and voices for this demonstration purpose all right heading back to AI builder so if you go to power apps or power automate I'm heading over to AI builder and because all my invoices look alike I am going to go ahead and leverage the new the form processing model now in the form processing model I'm going to go ahead and give a name for this model so I'm going to call this my invoices model and I'm going to click on create now the moment I do this this is going to go ahead and generate the experience for me to go ahead and upload and train my model so in this case I'm going to go ahead and click on add documents and I'm going to upload documents from my local storage space now you can also upload documents directly from SharePoint if your files are in SharePoint or you can also upload your files from Azure blob storage these are multiple options are available here in this case I'm gonna pick my local storage and I'm going to go to my folder that has my invoices and for testing purpose I'm just going to upload the first five invoices here because that's the minimum number of documents it requires to train the model of course you can upload more documents and train the model better but in this case I'm just going to upload five documents and once I upload these five documents what's going to happen is that these documents are going to be now leveraged by the model and then it will start analyzing the different key value pairs that are available as part of this model now if I click on analyze the the model is going to start to analyze these documents now while the analysis is going on it will take a few moments I'm going to go back to AI builder and I'm going to go back to the form processing model and let's just look at some of the best practices regarding this model so of course the documents have to have the same layout they should be very similar in the you cannot have different elements and different locations so for example if I had multiple invoices and my invoice date is in a different location from another invoice that I have that I'm uploading and training as part of this model so let's say my invoice number sits on the top left here but a separate invoice has the invoice number sitting on the top right hand corner then in that case you will have to create multiple models because each model requires documents that have the same layout I've received a lot of questions on this that's why I'm putting this point out the next thing is obviously each document should be a separate file that you're uploading don't use files that contain multiple multiple documents together so create independent files like how I did and just upload those files to start training the model and documents with primarily text of course you don't upload documents with a lot of images or checkboxes or stuff like that so to look at the do's and don'ts and documents need to be simple you cannot have complex nested tables in my scenario the document that I showcase I'm going to show that once again if you notice I have a table right here and these are not nested tables it's just one table if I have a table within a table then the model would not be able to recognize it and also forms processing only used to work with key value pairs that means there needs to be a key like in this case bill two is the key text and the value is right here in this case the key is description the value is right here the key is quantity the value is right here the key is balanced you the value is right here key value pairs but now with the new preview functionality you can also go ahead and define your own your own key value pairs within the document and we're going to look at this right now all right so let's go back to the model and as you can see the analysis is complete and now the next step for me is to go ahead and start selecting the elements in my documents that I have uploaded my key value pairs basically now in this with this public preview of this new feature what you can do is the model will automatically detect key-value pairs and additionally if there are any fields that are left undetected you can identify them and tag them okay you can also resize them and let's look at that in action right now as I go through the process I'm going to try and zoom in as well so as you can see the typical key value pairs that it aspect of course is bill to here's the name so as a key here's the value here's the key and here's a value in this case because it's a table it has directly identified this as being a table so I can just click on this and this will give me a table that it has identified now you cannot rename a table in the form processing model as of now so you have to live with the name that it breaks in every table they have to be simple tables that you pick will have a specific ID associated with it so in my case this is table underscore 0 I'm gonna click on yes so I've picked my table now once again notice balanced view is my key and the value that it has picked is 650 so I can just click on this and it's already telling me that this is balanced view and 650 if I want I can change the name for this key value pair but I'm just gonna save this for now so right now I've registered two things my table and my balanced view and both of them are highlighted with this green border now the next thing if you notice is it has not identified this is what what happened in the form processing model it would only give you key value pairs now this is not a key value pair this is not a key value pair but with the new preview feature what happens at the back end is it actually even OCR's optical character recognition the document for you and now what you can do is you can actually go and define your own key value pair so I can say okay this is my invoice number right this is my invoice number this is my invoice date so I'm defining my own key value pairs right now this is the name of my own company now in my scenario I don't want that because that's not something that's different but if I want I can even leverage individual components here but in this case notice it's a key value right I can click on this and this is the value that it's identified but maybe that's not what I want maybe what I want is I want the I want to go and define my own box so you notice as I define my own box and you can resize this box as well right I can increase the size but what will happen is it'll only pick the ocr'd component here so I just want to pick the name so let me try and zoom in so it's a little bit more clearer I am billing to this client the clients name is David GoPhone I just want the words David on the phone because I just want the name of the customer so I'm just gonna pick this and draw a box around this and notice how it is picked both you or see our elements here and I can call this customer name okay click on yes notice the moment I do this it puts it on the right-hand side and the selected fields list now this is the company name sake and I'm going to draw a box around it and I'm going to say this is the company name so I got the company name I got the customer name alright I got the company name I got the customer name and I've got all the elements that I want now this right here is my invoice do this so I'm just going to pick my invoice due date as well so because this is being ocr'd you can go and define your own elements custom custom fields in the form processing model now once you are done with this in my case I'm done with this I need to click on confirm fields because I'm confirming that these are the fields that I want to track as part of this model now the moment I do this what's going to happen as these fields are going to be registered for me and now I need to go and train the model for all the other images that I offer all the other files that I have uploaded as part of this model now notice that as part of this model if you look now on the right hand side right so if you look at the first document everything's covered for me right I've got all the things identified the balance to invoice number everything's here now if you want to still add more fields like in my case I've notice right now it has missed the table for some reason so I'm going to go back to the field selector I'm going to click on the table and I'm going to make sure that the table is a part of my field right here so it's right here I'm gonna hit confirm fields again okay so the table has also got registered now when I move in ahead in my process of tagging my fields it is only asking me to tag the fields that I have custom-built so you notice on the right hand side that says invoice number I can either say that the field is in this document or I can say I want to draw it in those documents I'm gonna click on drawing document and I'm just gonna go and select where the invoice number is and say okay and now notice the invoice number is tagged once again I'm going to zoom in I got to do this for every document that I upload to train the model I'm gonna pick this this is what this is Mike invoice date okay once again I'm gonna draw the box what is this this is my customer name draw the box what is this my company name click what's this my invoice due date remember this is being ocr'd I got to draw the boxes and define it alright I'm done with this document next same process this just keeps repeating I just got to do this for all the five documents I'm going to quickly do this invoice number invoice date this is my customer name this is my company name this is my invoice to date and I'm done with this document moving ahead to the fourth document that I uploaded and that's why I uploaded a minimum number of documents I'm gonna pick this this is my invoice date this is my customer name this is my company name and this is the invoice to date and finally I move on to the last document I'm gonna say this is the invoice number this is the invoice date this is the once again see I haven't zoomed in too much so I need to make sure that I'm looking at the right value there you go customer name company name and in this scenario the invoice do you did all right I have tagged all the necessary controls all the fields that I selected and I'm going to click on done and once I click on done notice I have seven fields that I had highlighted I'm going to click on next okay these are all the fields the this model has identified I picked the table and there were certain things like balance du which was the key value pairs that I pick and I also built my own custom key value pairs the ones that were unidentified by the model which I identified and I trained the model finally I'm going to click on the train button and what this is going to do is this is going to start training my model okay and I can click on go to models and this will take me to the models tab and right here as you can see here is my invoices model and this model is currently under training and the status for this model is training so let's wait and I'm going to forward this video so that we straight do we move to the Train model this can take up to two to five minutes and this also depends upon how many documents you have uploaded for training all right so as you can see the model is now trained so I'm gonna go ahead and this time I will go ahead and publish this model so now this model is being published so that I can leverage it in both power apps as well as power automate and there are very nice useful links here that will come alive once this model is published that takes you to reading materials on how you can leverage these models both in power apps and power automate in this video we will walk through the process of leveraging it in power automate but very similarly you can also leverage it from power apps alright the model is now published you see the documentation links have come alive and I can start using this model now before we use this let's go to a quick test and what I'm going to do in this case is I'm just going to upload one of my invoice documents here and the moment I upload this model is going to start looking into the key value pairs that we had identified and let's see what it is able to search it's always always a recommendation to test and here you go invoice number invoice State customer name company name these were tags that I defined so is this and so is this this was the table that had identified this is the invoice to date and notice it also gives you a confidence score associated with all the key value pairs including the custom one that I've built right now it says it's 100% confident that this is the invoice due date and here is the balanced view perfect now the next step what I want to leverage is I want to go ahead and start using this model now one quick way for me to do that is just click on use model and I can start by creating an app and or by just going to flow and creating a flow and starting to leverage this model in flow right and if I do this it basically takes me to a template that I can leverage in this case it deals with common data service as a data source in my scenario my data source is a little different now I can leverage the template but I want to build it from scratch so let me show you that in this case I have a site called Finance and in this I have a document library called invoices and this is where all the invoices are going to be uploaded when I will upload the invoice I will trigger a flow and what the flow will do is it will run the invoice and these are the same invoices that I have trained the model on of course the invoices that look similar I can have thousands of other and voices that look similar I'm going to upload those invoices there and what I will be able to do with power automate is called the form processing model that I just built and leverage all the data points that I had identified as part of the model and plug all the data in here at the same time as part of my process you see I have a tabular data in here as well and what I have done as part of my SharePoint setup is I have a library and all the tabular data is going to go in a separate list that I have created called invoice items all the tabular information will go here whereas all the invoice which is my documents will go here along with all the metadata associated with it let's look at how we are going to build this from scratch so firstly if you want to leverage the models is that you've built in the eye builder you have to build a solution so I'm going to go ahead and create a new solution I'm going to call this invoices on invoice processing it's the name of my solution I'm gonna pick the CD as default publisher and I'm just going to click on create and this will create a solution for me and in the solution I am gonna go ahead and create a new flow okay so I'm for a leverage power automate because what I want to do is every time when a file comes here I want the power automate to get to be triggered right so what I need to do in this scenario is I need to go to SharePoint because my data sources SharePoint in this case you don't need a source could be anything it could be sequel it could be CD s your data source your choice alright so there is a trigger called when a file is created in a folder so I am going to pick this and in this scenario my site address is my finance site and the folder that I'm going to pick is going to be my invoices document library so that's the library as to path as the folder level now once I've picked this when this triggers it will give me very limited information of the file I want all the information for the file so there's another action called the gate file metadata using path and this will give me all the data associated with the file and basically this is not the properties associated with the file but this is more related to the SharePoint property information so I'm going to pick this action and in this case again I'm gonna pick the Finance site and the file path in this case is going to be the file path that I captured from my trigger action so I'm just going to pick this alright I've got my file I've got the metadata associated with it the next thing what I want to do is I'm going to go ahead and search for the word predict predict is the action that I need to leverage to call any of the models for AI builder now as part of this video I'm going to add this to a playlist called AI build off of our platform I plan to do videos on all of the other models as well so be on the lookout for that now in this case because I built my false processing custom model which was called invoices I can just go and pick this model and when you pick this model what it is going to do is it's going to associate this action with that model and if once it associates it it will open two more properties for you one is document type so what kind of documents are you going to provide as part of the model processing component now in my case all my documents are PDF so I'm going to pick application / PDF if they're going to be image files they have to be JPEGs pngs or tips so they it is limited to what you can upload but these are all the file types that it supports and if you want to give all the file types you have to type in those words exactly the way it's given here comma separated in my case it's only PDFs I'm going to do application / PDF that's it and finally it requires the document and the document okay the document component for this is what I am going to fetch directly from my trigger which was when a file is created in a folder there's something called as file content this has all the contents of the file that I uploaded into SharePoint that's all I got to do now in the previous video that I had built regarding this nine months ago there was a lot of parsing of JSON that had to be done but in this case we don't have to do any of that because all the parsing is free done for you now once this goes through the model what I want to do is I want to go ahead and update the properties of the file and I want to fetch the data that the model provides and update all the properties of my file so I'm going to go to update file properties action pick my site pick my library which is invoices now the ID of the file that I want to that I want to update is what I will get from the get file metadata using path and that's why I had to add that extra action just to get the sytem ID all right so I got the item ID out here now all I need to plug in is the customer name balance due date invoice and all the other information that I have already trained my model to understand right if I go back to my model if I go back to invoices these were all the fields that I have trained my model to understand now the best part is the JSON parsing has already been done for me now in my scenario when I created these columns in SharePoint I created them using a lot of the different types that were available in SharePoint for example the balance column is a currency column right due date is a date and time and this model when it leverages the data it does pick the data more or less and presents it in string fashion so the trick here that I use is I normally go ahead and add a compose action okay and what I am going to do now in this compose action is for all these special column types like for example balance right balance now balance is a currency column in my SharePoint list now if I directly click here and if I start searching for balance you will notice that I am getting the balance due value but I'm getting a lot of other properties associated with it and this is what the model provides it provides you the confidence score the box height of where it identified it in what's the height of the balance do feel that it identified in the image or the document that I provided how many pixels from the left from the top from the width which page number and the document access a lot of information here provided to me okay however there is one key component of this called the value so there should be something called balance do you value that I don't see and the reason is because this is a currency field and the value that is coming from the product model is of type text so what by adding a compose action what I can do right here is if I search for the same balance to you here you will notice that this time if I look at balance do right there's something called as balance do value okay so I get this and I can just copy this okay highlight this control C go to balance control we alright that's incorrect let's try it again so I'm gonna copy this and I'm going to try and paste this in the balance field okay now once I'm done with this I have copied my balance to you right I got this right yeah now customer I know it's going to be a string so I can straightaway search for customer all of the passing is already done for me so my field in a eye builder was called customer name so out here customer name value will give me the value due date once again you date right so there's invoice to date and this invoice due date value invoice number invoice number value so if you don't find the value component of your key value fare-bear go ahead add a compose gathered dump it in here alright so I am going to pick invoice name value now I need the invoice date so I'm gonna do invoice date and once again go to pick invoice date value and invoice items is a field that I have created in my library and all I have as part of this field is it links to the items that I'm going to add in the secondary list so we'll talk about that a little later and customer address I'm going to go ahead and search for customer who I called it customer address actually it's the customer company so I need the company name right here so what I'm going to do is I'm going to go here and search for company name value its it click on save and I should have named my flow more appropriately I'll do that after this thing saves very very important you gotta save your flow don't browse away don't navigate away let's just save it all right and I'm gonna rename this flow to my invoice processing flow I'm gonna click on save so I've just renamed my flow right now and the next step what I'm going to do is just for testing purpose for now I'm gonna go ahead and upload another file so I'm going to upload one of my invoices right here and what's going to happen is the moment I upload this document in my SharePoint library this flow will come alive right so if I click back alright so as you can see the flow is now currently running so let's go and click on this and let's see the flow in action right here so the flow got triggered when I uploaded this invoice file now it's gone ahead and fetch the metadata it sent the file content to the model and this is the output that I get so you see all the parsing is already done as part of output values that is providing to me right here and after it completes all of this I have gone ahead and I've updated the properties of the file now if I head back to my invoice library and if i refresh my library notice that all the metadata associated with this invoice has been captured the invoice number right here is four one zero four I'm gonna try and open this in a separate window so we can just look at this side-by-side right I'm gonna place this right here and let's try and just this a little bit better so I'll move this across so we can just compare alright so the customer name right here at Serena Williams what a customer to have so I cut my customer name right here the balanced view is two hundred and fifty dollars the due date for my invoice is for twenty nine twenty twenty and that's the date column in SharePoint invoice number is hash four one zero three right here invoice date is four five twenty twenty and customer address is in this scenario I train the model to pick the company not the customer address but I could have also let's say if I had the customer address in here I could have fetched it so instead of customer address are basically I'm putting the company right now so as you can see I was able to upload the document and form processing model not only fetch the key value pairs but it also went ahead and fetched the key value pairs that I identified the custom ones right alright so we were able to do this much and I'm going to go ahead and delete this because there's more to do alright so I was able to fetch this information but at the same time I also have this tabular data that I want to fetch and I want to put it in another list that I'm creating where I'm going to hold all that tabular information and I'm going to link it to the invoice that was created this is a lookup column in SharePoint and this looks up to the invoice ID which is the hidden column that gets created in SharePoint so every list or library in SharePoint has a column so let me head back and let me add that column right here show an add columns ID and this is the unique ID column that gets created it's the primary key for my document library of invoices so now I'm gonna head back to my power automate I'm going to click on edit now how do I get the tabular value I get the tabular information from again there's not there is no need for doing any parsing because the tabular information is going to be multiple rows in that table I'm going to run an apply to each loop and I can do this for every table that I have identified as long as there are simple tables okay and what I'm going to do here is there's something called as table 0 entries right so look for look for the entry ski so if you've got three tables table 0 1 2 look for the entries equivalent so I have picked the entries and now what I want to do is I want to go ahead and create an item in that list in my items let's so I'm going to click on create item once again I'm going to go ahead and connect this to my site I'm gonna pick my list which is invoices and this time I need to enter all this information right now if you look at this I have description quantity unit price and total this only works for key value pairs because they were key value pairs and because this is all contained in a table all of this parsing is already done for me I'm gonna head back here so let's say the title I want to plug in the so I'm gonna click on this and I'm going to search for description once again same concept table 0 because that's the table description because that's the column and value has the value ok same steps quantity what's quantity Q T why because that's how I've named it in my table so I have to search for Q dy and right here I have the quantity now quantity in my case in the backend I've defined it as a numeric field notice I don't get the value so what's our trick if you don't find something go to compose especially if the data types don't match click here search for quantity and this time I should get the quantity value I'm going to go ahead and copy this and I'm going to paste this right here okay now notice this is still showing me the old copy paste be sure be careful and make sure that you are copy pasting the right control there okay the next thing is price so I'm going to search for price and it's giving me table zero unit price value and why is it unit price because that's the name of my column there I'm gonna pick unit price value once again make sure you copy this and paste it this is still showing quantity copy paste come on alright unit price now I need total and total right here is called total in my form so I'm gonna pick total and once again total price is right here I'm gonna put the total value right here again still showing me the old value looks like I gotta do it twice might be a bug that's business right here okay its total and finally what's the invoice ID that I want to relate this to and my invoice ID is going to be the ID of the file that I just updated right I just updated the main file I need it's ID so I can relate it to the main invoice so I'm just gonna pick the ID and this is a lookup column in SharePoint I'm gonna delete this because I no longer need this alright that's my flow and I do not even need this compose I just added that to fetch those values going to go ahead and see if my flow now if the demo gods are with me and if everything goes right let's go this time and let's just upload all the invoice files I'm just dragging and dropping all my files in my SharePoint repository in one go now the moment I upload these files notice SharePoint has this internal ID column so it generates a unique ID for every file that gets uploaded at the same time I also have this column called invoice items and I have added a little bit of SharePoint column formatting to this so I can create a link in order for me to click on this and it will directly take me to invoice items and once those items are in here I will able to access them alright now that I've done all of this I've uploaded the files I've done all of this I'm hoping I had saved the flow you need to ensure that you've saved the flow click on save once again and let's just wait and see if any of these files have run so far doesn't look like it can take up to two minutes for it to trigger all right you see one two three four five six all my six flows are running these are all the six files that I uploaded these files are being processed right now and if everything goes through smoothly notice how it's gone ahead got the customer the balance and all the information and if I go to the invoices list notice it started getting all the invoices as well and that's the tabular data okay now let's go back here and let's pick let's pick a raw file in a doll write four six seven two so I'm going to open this one let's observe this this value right here okay so of course rafael nadal abd tech right all that information is right here Rafael Nadal EBD tech all that information is passed but let's look at the tabular data for this more click on items this is going to take me to items and voices and let's look at the items and voices for Rafael Nadal he took power apps training two of them fifty dollars each and power automated training one for fifty dollars check this out power apps training to quantity fifty dollars that made a total of 100 thought I made 150 450 value and total is 15 you see how it created the and the invoice number associated with this of course this is the lookup column that I created so if I click on this it will directly take me to this parent which is the invoice which is rafael nadal very very simple so things have changed a lot right in terms of form processing and this is available in public preview and so go ahead and try this out and try various scenarios the key to all of this was when I built the model I could define my own key value pairs right secondly when I came to power automate and I leveraged this predict function the best part is I don't need to do any sort of custom JSON parsing everything's already pre done for me and all I had to do was set up my back-end and leverage power automate to go ahead and call my AI builder model which was in this scenario my form processing model for invoices and it went ahead leveraged the model got the values and plugged it into my data source thank you so much for watching

Channel: Reza Dorrani

Views: 15,133

Rating: 4.9285712 out of 5

Keywords: powerapps invoice management, ai builder, ai builder powerapps, ai builder form processing, ai builder microsoft flow, cognitive services form recognizer, cognitive services microsoft, microsoft flow sharepoint, invoice processing automation, Invoice Automation, Forms Processing, Forms Processing Model, powerapps ai builder, powerapps form processor, powerapps ai builder form processor, microsoft flow form processor, ai builder form processor, power automate form processor

Id: 3ggCWEymNXI

Length: 35min 30sec (2130 seconds)

Published: Thu Apr 30 2020