3 Ways to Extract Data from PDFs with Microsoft - ChatGPT, AI Builder, Syntex

hey everyone my name is Andrew hes I'm a Power Platform developer I also have been getting into AI I'm loving AI today we're going to go over three different tools to do OCR which is optical character recognition we're going to take PDFs and then we're going to read the PDFs so we're going to do chat GPT we're going to do Microsoft AI Builder and then we're going to do Microsoft syntax these three are going to be really awesome stay tuned we're going to go through each one and if you want to skip around you don't want to see chat gbt that's why we put chapters in the video I'm going to get opinionated I do think I put it in order from least to greatest so I think least chat gbt then you have ai builder then you have Microsoft syntax so go ahead skip around to the end if you want to watch the end but stay tuned thank you all for watching let's get into [Music] it so what I'm going to do is when a file is created in my one drive I I'm going to convert the file so I'm going to convert the file this is a new preview action here actually it's asking for The Unique identifier of the file the ID right so you can go in there and select if you want but if we're going to do this dynamically I'm going to give it the file identifier all right so we're going to give it the file identifier now after we convert it to an image the ne next step we want to do is actually convert convert it to a base 64 format so I'm actually going to do a composed statement and this composed statement is going to convert it to base 64 so I'm going to do FX here base 64 then I'm going to put my file content my Dynamic content here the file content from my converter file in the base 64 so we convert it to a base 64 format next what we're going to do is an HTTP connect so we're going to do an HTTP connection right here and so this is a premium uh connection and we're going to do the the URL we're going to go to chat GPT to do this now you could do grock there's many different options it's just kind of it's more important to learn how to do this than it is to learn the exact way right so I'm teaching you how to fish I'm not giving you the fish all right so the first thing we're going to do is we're going to connect to open AI the URL so there's our UR L for chat completions the next thing we're going to do is the content type of application Json and we want to make sure that we're doing a post method so we're going to post there and then we're going to do application SL Json and the authorization is also part of the header and then we're going to put in our key and I generated this key today I'm going to delete it right after this video so you can see my key um it's an example you need to generate your key off chat GPT API now if you don't know how to do that I have other videos one of my first videos shows you how to generate your key and all the steps to get through there next we're going to do the body so the body here I'm just going to paste it right out and we're going to use from chat bt40 you are an OCR tool but what we want to get and we're going to look at the files that I have so let's look at one of those files for Clark Ken it has name job date and skills you arear an OCR Tool uh retrieve name date job and skills but we're going to do that for right now we're just going to say you are an O OCR tool retrieve name date job and skills and that matches my document name job date and skills and notice the order is not the same next what we need to do is parse the Json so we're going to parse the Json now there's many different ways you can get this answer um I guess today I'll show you how I would get the answer we'll just run it one time we'll just run it one time so this is my folder I'm going to drag and drop a new file in there so we're going to test it out to see if it runs and so now it's going to run the power automate flow perfect time for coffee it may take a minute or two all right it looks like it successfully ran this is where we're going to get the the parse Json part so if we look down in here we can see in the body if we do show raw outputs we get to see the exact body and you can see right here name Kiana Reeves date actor skills Etc all that's in there oh influencing people interesting so we did get a little bit of a um oh no that that is correct so acting influencing people so let's take this output so we're going to take the entire output right here I'm going to take this and I'm going to copy it I'm go back to my power automate and I'm going to edit and this is where I'm going to parse Json so I'm going to parse Json and generate sample payload and this is where I'm going to paste everything that I just copied I'm going to click done and for the schema it's going to be the body okay so we have the body and we parse to the Json now we're going to do a little bit of a compose statement we're going to compose because I want just the message I don't want all the information I want just the message and to do that I'm going to use a very simple uh compose and it's we're going to do FX paste it in and it goes to the body of the HTTP the choices and takes the first message content Okay so we've taken the first message content I'm going to go to SharePoint and I'm going to add a new file let's see if I can do this add attachment no check out file create file I'm going to create file it doesn't look like I can add actual text to it but we'll get to that all right so we're going to go to OCR example the folder path is going to be let me make sure I'm in the right spot the document Library so this is shared documents the file name is going to be the same file name up here the file content is going to be our file content from the very beginning and now I'm going to update file properties now I'm going to go to SharePoint and I'm going to update file properties and we're going to go to the same SharePoint site and the same library and the ID is going to be the one that we when we created the file so it's going to be this body ID right here and then in the data here I'm going to actually paste my compos statement now that we added our data to our data column we can send it to SharePoint now the bad news is is it's not going to this is not perfect I do believe using Microsoft syntax is probably better or using the OC R tool um the AI extract data tool is actually better too and if we refresh we can see that now we have our answer so we have our document all right so you can see how I pulled in the name the date the job and the skills now I really think this way would be really great if you were looking at like license plates on an image the main reason I would use chat GPT is it's very simple and it's for extracting one single entity of text and then putting that into a data source of some sort but now let's check out Microsoft's way of using the AI builder in the low code no code Power Platform all right so next up is going to be the AI Hub in power apps so if you go to power apps in the top left you'll see under more there is an aiub and I'm going to pin it to my left side so I'm going to pin it to my left side here and then I'm going to go to aiub so let's go to AI models right here and then we are going to look at the most popular so what we did in chat GPT was mostly extract all text in photos and PDF documents right so we kind of extracted things from images so we extracted words from images that that's kind of what we did in chat GPT but what I believe what we want to do is document processing extract custom information from documents so you label out the fields and you extract the information from a custom document so I'm just going to go ahead and create the model say Okay so we're going to say fixed template documents uh we could do general or invoices I'm going to do fixed template document here's some examples and so this is where you have exact templates of when things come in now you could do General documents these are unstructured right the option is ideal for any kind of document maybe we'll do General document just to kind of test it out because we had some things in different orders maybe we want to extract from résumés and not everyone has the same resum so we'll do a general document no said structure when the form is complex you can teach this model to extract data from structured or unstructured documents that have different layouts this model is powerful but had has a long time of training let's look at this one this model is for quick training time okay let's do this one for the demo if you want a Long training time uh not for this demo so we'll do next extract information information that we want so there were four fields let's look at our documents again so in the document we have name date skills and job all right so text field name date year month day is fine skills and there's no like multi-line text so we'll just say text so job all right so those are our four um pieces of information that we're going to extract we'll click next create a collection for for each similar document add sample documents for your model to study I believe that you should have at least five so right here on the right side it says improve performance of your model you must add at least five examples of documents for each collection adding up to 20 examples could yield better results so let me stop I'm going to go create a bunch of examples all right so I created about 14 PDFs now now when you do machine learning which this is actually a type of machine learning you never want to use the same data that you train on to test with right so you should have a set of data that you train on and then when you're going to do your testing you should have a separate set of data so I have 14 PDFs I'm going to train on 10 of them and then four of them we will use for testing and it's a different set of PDFs I'm going to upload 10 of them upload 10 documents now we'll wait for them to upload nice not taking too long we're done now I'm going to click next okay so now view your documents inside the collection use the arrows Etc so at this point in time I believe we are tagging and I guess we have to do this on every single one of our documents so we'll start on one name job date and skills so on the second document name job date and skills name job so I'm going to go through all of these I'll probably uh skip through so you don't have to watch me go through all of these but I'm literally just going to go through every single document and label name job date and skills so I'll fast forward and we'll start recording again once I'm done done okay so we have that information in there we've tagged our documents now we'll click next um 10 examples document is my device fix templates and we will train so now we're going to train your model is training um I'm going to hold on here as long as I can there's no way it's already done this may take a few minutes depending on the size of your training data so go to the model okay so we have our model here it is training so it may take a few minutes let's see how long it will take to train 10 documents and only four only four fields that we are training on okay so looks like uh last train three minutes ago how long did that take 1050 oh that's not that's just when I started it didn't take that long I would say it didn't take more than 10 minutes so I walked away came back and it was done four minutes ago so maybe it took about 5 minutes to create all right so now that we have it let's see if we can actually use it so we can publish an an excellent accuracy score that's great how to use your model there's two options here power automate or power apps most likely for this video we'll go to Power automate I'll go ahead and publish and then we'll do the quick test button I love how it gives me all this information and how accurate it is right so you have an accuracy uh score here before training it separates your data into separate training and testing data sets nice and after training AI applies your AI model to the best testing data set to calculate the accuracy score so it's really doing a lot of the work for you if you were doing this in Azure you would have a lot more work to do a lot more customization to do so let's do a quick test so we're going to upload one of the documents that I did not use for the testing or for the training so I'm going to upload this one Wonder Woman superhero a date and her skills analyzing oh and it just picked them up and has a confidence score okay so that's the Quick Test I did find them so now let's go over to power automate this is where it gets fun we have these columns here this was from our chat 2bt um option let's add the columns now for name uh username maybe uh um resume name right because you can't use name name is actually an outof the-box field it's used for the document so we'll do resume name and then we'll do date I believe we can do date date is not a reserved column then we have job and skills all right so now let's create a power automate based on this list here and I'm just going to do let's see automate no integrate power automate create a flow so I want to click see my flows I there's many ways to get to this area but that's just one way we can get to power automate I'm going to do a new flow and if we do automated I believe I still have set up the this one here I'm going to disable this flow so this is the one that reaches out to Chad GPT I'm going to disable it we're going to do a new flow so let's just do automated and we're going to say when a new item is created and let's just actually let's just skip here for now and let's go straight into Power automate so I'm going to go to SharePoint that's my trigger when a five is cre but that's deprecated I don't I don't want to use that when an item or a file is modified I don't like the word modified in there when a file has created properties only so we're going to do that one and it's going to be our OCR example the library name is documents now what I want to do is get the actual content of the file now so if you notice this is when of files created properties only so now I want to get the file content so C more when a file is created using a path and it's going to be the OCR example we're going to select one of the files in the folder so it's in my shared documents we'll select one of the files but then we're going to remove the file here insert Dynamic content and we will grab the not the folder path full path no we just want file name with extension and don't forget to add your slash in here so don't forget to add your slash all right so now I'm going to try out the AI Builder um action I've actually never done this so this is kind of fun I like this is pretty fun so AI Builder which one did we do extract information from documents select the model that's our one that's the model we just created all right so we're going to do of type PDF because that's how our documents are right now select form so the form that we're going to process I believe we're going to use the file content right we're going to use the file content and then we're going to write that to our document Library properties all right so we're going to go to SharePoint and we are going to update file properties and that's our OCR example site our library name documents the unique identifier is right here when the file was created show Advanced properties so resume name this is going to be the username this is where we're doing a little bit of learning maybe we'll try name let's type in the word name first so we have a few options name value text collection name name data type confidence score which is really neat I like that we can just pull out the confidence score I'm going to say name text date so we'll search again date value I believe probably for this one we want date value but I'm not sure let's try date text let's let's just try everything text right now job job text skills skills text all right and then let's save so we have our flow it is set up it's when a file is created in that Library it then grabs the file content type it then extracts it then update file properties so we're going to test with a manual test and then I'm just going to upload one of the same documents in here so now I'm going to do mile Morales I'm going to upload it it is already here um let's do keep both just let's make a copy of it and let's see if the flow runs right here on my Morales 1 PDF all right so it is running it got file content using path let's see it looks like everything was correct let's let's go to SharePoint boom look at that we've then extracted all the data now that is way too easy way way too easy right imagine how many ways you could do that let's just go ahead and upload a couple more so I'm just going to do the same ones and upload them in there it is way way way too easy we're going to keep all we're going to duplicate it to extract what want so if we notice in chat GPT this was different how we did it right like we took everything in bulk we didn't split it up by Jon and maybe this may be possible with GPT 5.0 and maybe we could do prompt engineering to make it perfect to give us at Json every single time that was perfect but for right now it was not good enough maybe if we tried grock which if you kind of pay attention in social media grock is supposedly better right now now um and we could probably extract that data Maybe using grock Vision which I'm not sure if it exists or not but for right now today is July 2024 this is what chat GPT was able to do us do for us consistently and so you could easily you know pull out license plates very easily with Chad gbt that would be very great but for document processing I do think the AI Builder right now if we hit refresh is amazing like that's just too good that's just too easy right now it does come with a cost you'll have to research those costs as you want I am just showing you how to do it all right so thank you let's keep on [Music] going all right so now I'm going to use Microsoft syntax to autofill columns so right here autofill columns let user add prompts to to extract or generate information from file contents so we're going to use Microsoft syntax to do this so the status is on sites where autofill can use so it looks like you can edit which libraries can do this so just for security reasons what I'm going to do is I'm going to say for selected sites I'm only going to do it on my OCR site I don't want it to accidentally show up anywhere else and then I'm going to save so this is where models can recreated is just on that one SharePoint site that we've been doing and I go all the way over maybe I hide these um fields for right now show hide so I'm going to hide my data date skills and job and resume name I'm going to hide all of those add a column a single line text look right here autofill so this is what we added to it before all right so now let's say and we're not going to call it resume name again let's call it um person name we want to give everything a new name so we get this Clarity setup autofill test file is here prompt enter question or instruction here or select an example to start from what is the name of the what is the name how about what is the name test so this is really neat this is AI incredible save all right and then so let's do another one so I'm going to hit save all right processing status all right so now let's do another one text or let's do the date it is an autofill column what is the date test pulls in the date save uh person date I'm gonna give them all new uh column names all right so we have person name Person date another text uh job oh we want to do autofill what is the job test actor save and we're going to call this one person job all right and then one more text person skills setup what are the skills test acting influencing people and leading I love how they put the columns in there too save all right so now we have autofill up and running I'm going to refresh I'm going to click on K Reeves and I'm going to click autofill so if you look now it's now populated person name Person date did fail uh maybe it's still going it says it finished but person date did fail person job came through actor and person skills came through so that's an example of autofill and you can initiate autofill just by clicking on the three dots and clicking autofill look I just refreshed this time it brought in the date correctly it has the person's name uh date job and person personal skills it's really neat stuff so all you do is just come in here and click uh autofill and then it starts processing your files so you can see now it's in progress and it's literally going to Microsoft syntax and pulling out all of the information so I'll autofill uh say J and now Wonder Woman when I press refresh you can see that they are in ress and maybe it takes 2 to 3 minutes so it's very similar to power automate except for now we're using autofill so you can see that processing finished it's now walked through processing status it's filling in the autofill of the columns beautiful simple and the other thing is right now Microsoft states that you can do this for free um if we look at the documentation about that you can do this for free until July 202 let's look at the documentation so right now you can try out and this is this is not me saying it this is just the documentation I'm reading you can see it was uh updated pretty recently try out Microsoft syntax and explored Services you can see monthly included capacity at no charge 100 Pages 50 documents so that that's our autofill uh columns there and you have all these other things optical character recognition image tab tagging all these other things that you can try out for free through the Microsoft Services now uh we have given you multiple different ways to do OCR to read PDFs this is one of my longer videos but I love doing it thank you for the question I'll see you next time thank you

Channel: Andrew Hess - MySPQuestions

Views: 527

Keywords: microsoft ai, ai builder, microsoft syntex, microsoft azure, power platform, power automate, chat gpt, ocr, optical character recognition, read pdfs, how to automatically read pdfs, extract data from documents, pull data out of documents, microsoft autofill, microsoft azure ai

Id: XhjdoZd_2Is

Length: 28min 47sec (1727 seconds)

Published: Mon Jul 08 2024