Extract Receipt Data: Laravel, Livewire and OpenAI GPT-4-Vision

hello guys today I want to show you something cool with live wire and AI I recently saw this tweet by this guy Jerome paulos who created a pretty cool project of scanning receipts from shops for example and getting the total amount of how much was paid using open AI gp4 Vision API and this is not free by the way but I will get to how much it cost later in the video so I want to show you how it works I've tested it myself with my own receipt and I will show you is it accurate again how much it cost and also we will see the code with Livewire and calling open AI API first let's take a look at the actual video 9 seconds on Twitter how it worked for Jerome so he uploaded his receipt and received the total amount and category automatically scan and this is the result cool isn't it and the repositor is free public on GitHub this is laral Project that's why I'm shooting this video I've tested it out myself and this is how it looks on my local server and this is the larl code it uses Livewire I've tweaked it a bit so it would work easier for me but basically route is Livewire component new receipt with validation for file upload with form object from Live Wire 3 and this is the main method autofill and this is a bit of a fake demo because the author just left the URL for his exact image and I've tested it also with URLs from online because this is how you query the open AI API with URL of the image in reality it should be probably uploaded with Livewire to the server then you get the URL and then paste it here so first I've tested it with random URL receipt from Tesco UK shop and the receipt looks like this just Googled a random receipt and let's see how it works so I upload this image from Tesco then scan appears here and if I click that there's CSS animation here and I will show you that in a minute and in 5 seconds or so we will see the result of scanning it actually took a bit longer than the original video but it got the result where the receipt is from Tesco correct the total amount and the category which is guessed from the items now I'm not sure why but the amount is not correct it should have been 5530 but for some reason it's 5470 I assume that the pound sign was red as five let's see the actual prompt now one of the most important things actually when working with AI with open AI or other models is the correct prompt formulating the question in the right way so we load open AI as a client we have open AI package installed and then we start a new chat and this is the model Vision preview which scans the images so the prompt is not only text there is a text so system return only Json without any markdown formatting or additional text and then user content is extract data from this receipt let's actually close the sidebar so you would see everything return only Json return the data in this format if you are unsure set this to empty string do not attempt to calculate it so this is how detailed should be the instruction to the large language model and then you pass the URL as image URL and then as a result you get the Json and then you process that with HTML decode Json decode and the result should be if it's accurate here data store data amount data description and data category which is looked up in the database and then also we calculate how much it costs with response usage it also gets sent from the API response and we can see that visually here Down Below in small letters we paid 1.1 cents for this one request which took 10.8 seconds now I've also tried with my own receipt from this morning when I went to a local coffee shop in Lithuania this is my receipt here you can see it's in Lithuanian language at least partially but let's see if it scans correctly the total amount of €650 in this case and also as you can see the image quality is not that good it was just a quick shot from my phone and I deliberately didn't try to make the quality better just to test the model whether it was accurate so we upload this image and we scan and let's see the result and here's the result as you can see 650 correct the receipt is from reton convenience Lithuania which is on top which is correct is just the official name of that coffee shop and the category is meals and also it even populated the description the text description beverage and food item which is correct I ordered coffee and a cookie for my daughter so so yeah overall pretty impressive how you can scan the receipts with larl powering open AI Vision model via API with laravel package but again it's not free it costs roughly 1 cent I guess per receipt which may sound like a lot if you are doing that massively for like 100 receipts it would be already a dollar but compare that if you would pay a person to enter the same information manually and I guess you would pay that person more than $1 per 100 receipts that said of course you would need to hire the person to double check the accuracy this is in general my experience with AI and OC and chat GPT you should not expect 100% accuracy and it should be double checked Anyway by a human but of course you can increase the accuracy with again correct prompt more quality image and maybe more custom code before the scanning now a few more things to add as I mentioned there's CSS animation and this was the most popular question in the replies to the original tweet like this one and this is the code animate scan and it's also a part of the repository and I will link that in the description below and another thing one of the replies was from Helga who forked it and also created per item so item price and quantity extraction with his own package so this is the new code for new Reed scanning for items and the main difference is this one so instead of putting open AI request just like that there's extractor package with fields that you want to extract with model and then result assigned here so extractor is a package created by Helga larl package for data extraction by AI like open AI so if you are into extraction of data from open AI you may check out the package extractor and I will link that as well in the description below and if you are into Ai and want to learn more I recently started for those of you who haven't seen python MLD daily.com as a website and also a YouTube channel python ml daily and quite recently I published another example of open AI which is Chad bot with GPT kind of like Chad GPT bot with python so you can take a look at that and I will link that in the description below that's it for this time and see you guys in other videos

Channel: Laravel Daily

Views: 8,820

Keywords:

Id: ehbHx8XkNpk

Length: 7min 21sec (441 seconds)

Published: Wed Jan 17 2024