GPT-4 Vision API + Puppeteer = Easy Web Scraping

hi there in today's video I'm going to do some experimentation with the new chat gbt Vision API so previously I have created this Puppeteer GPT project which uses jat GPT to control the Chrome browser using Puppeteer now one of the biggest problems I had developing this project was actually scraping the information from the websites because if you just take the HTML from the website and send that to chat gbt then there a lot of extra stuff there and then you waste a lot of tokens and also chat GPT can't really understand the layout of the website and some of the things that are in the HTML might be hidden elements that you can't actually click so when I was developing this I was really trying to think of a way to give chat gbt the information about what is actually visible on the page and now they have released chat GPT Vision in the API so let's see if we can do something interesting with Puppeteer and the chat gbt Vision API so let's start by just writing some boilerplate code to use the chat gbt Vision API so let's call this vision dopy and here we have to of course import open Ai and actually how you're supposed to do it now is you have to say from open AI import open Ai and then we can do model equals open AI and then we can make a chat completion so we can say response equals model. chat. completions do create and we are setting the model to GPT 4 Vision preview and then we set the messages that we send to this model and we are going to have a role which is user and a Content which is going to be a list and this is another new form format that we have to send so we have to send a type which is going to be image URL and the image URL of this element is going to be some base encoded data so I have to do something like um data colon base 64 comma image/jpeg and a colon and then here we have to pass in the base 64 encoded image it's something like this I never really remember this data URL format and we have to create a function to create the B 64 image so let's create image b64 and it will take an image and we are going to open the image in the RB mode so read binary as F and then we have to import Bas 64 so import base 64 and then we have to say Return base 64. base b64 decode sorry n code and we can do f. read and we have to decode this so basically this should take an image file and convert it into base 64 format and then we can say that our b64 image is going to be image b64 and let's call this image. JPEG and let's put some image in there and then we also have to add some text to this content so we have to send a type of text and the text is going to have I believe there was like a value even in this so do I have to say value no maybe just text I don't quite remember maybe the text is just like this so I can say what is in this image and I have to verify this I don't remember if this is the right format but basically then we can get the response and in the response we should have the message which is going to be response. choices Z do message and then in the message we should have the content so the message text is going to be message. content. text. value or is it even content uh zero text. value something like this I have to look this up so let's take a look at the reference so chat create chat completion and actually the chat completion object will look like this so actually it is just this way I'm now mixing up the assistant API so this is the way for getting it from the assistant API but actually the content is just the content content right so we have choices zero message content okay that should be it so then we can just print the message text so now if we actually put some image into image. jpeg then chat gbt should tell us what is in the image if we did everything correctly now I in fact have an example of this so I might as well take a look at it so I had in gbt November 2023 and this and test sorry it's not test what is it called it is what's this.p so here we say type text and text is that and so just zero message content exactly and this was almost correct so it's the other way so this has to be that way so let's see if this works let's take an image let's go to unsplash.com and let's download some image from here here let's take this one and let's download it with a small image and let's copy this image here so let's do CP downloads and it was something Lucas and let's copy it into image. JPEG and let's run Python 3 vision. py and here we have the answer so it is an image that shows a person riding a motorcycle on a dirt road the person is now this is weird this open before too so it actually stops short for some reason I'm not sure what is going on here do I have to set like some Maximum tokens Max tokens it's 124 maybe it is like set to something else by default with the vision preview let's try it again okay now we got a longer message back the image shows a person riding a motorcycle on a dirt path through a forested area the rider is wearing a helmet for safety and the motorcycle's headlight is on indicating that visibility might be an issue perhaps due to fog or the time of day the trees on either side of the path create a green canopy and the overall atmosphere is one of Adventure and The Great Outdoors now that's a very good description of this photo but what would happen if we sent a screenshot of a website so let's try and do that so let's create a screenshot do JS let's use Puppeteer for this so let's in fact do npm init and let's do npm install Puppeteer now I guess I should be using the version of Puppeteer that bypasses all those robot restrictions I forgot what it was Puppeteer bot restriction Puppeteer St stealth this is it Puppeteer stealth so how do I install this npm install Puppeteer extra plug-in stealth let's try to use this one hopefully it is not malware and then here we have to import Puppeteer now I don't remember how JavaScript Works do I say import Puppeteer or just Puppeteer from Puppeteer maybe that is the way and then we have to like create a function which is an arrow function and it is an async arrow function and we are going to call this function and inside this function we are going to say that const browser equals Puppeteer do browser Dot and I forgot how to do this so we have to create a browser and then we have to create a page so we have to say browser dot no that doesn't make any sense let's actually Google this Puppeteer boilerplate okay we have to launch the browser so let's do it launch and we have to say headless false because we want to actually see what is going on at least for debugging and then we have to say const page equals browser do new page maybe something like this let's check out how we are supposed to do this we actually don't create the page in this tutorial so why is this so difficult I'm going to check from my Puppeteer gbt project in the index how did I do this previously okay so we have to say browser new page exactly and we have to await these things so we await pueter launch and then we await browser new page and then we can do page do go to and we can go to for example https n. Wikipedia.org SL Sam Altman and then in the end we can say browser. close is that correct and I guess we can set the viewport so let's do that and in fact before we go anywhere and of course this has to be a wait and probably this one too and let's see if it actually close it is so let's see if this will work and we can actually pass into this go to at wait until and we can say Network idle zero I think it's this way let's see am I using it here wait until yes exactly like this although this is for w for navigation but I think it's the same thing for goto so let's see if we can run this and if we actually open the Sam alment page on Wikipedia so let's run node screenshot doj and we cannot use import statement outside a module so how did we actually do it do we have to say use strict and let's do it again no cannot use import statement outside a module so do I have to say require but why does this work I have no idea maybe it is because I have package.json here and I have to say type module so let's try it again and now we're actually opening Sam Altman but that was not the correct URL so what is actually Sam Altman we have to say SL Wiki slam ultman okay so SL Wiki SL samam ultman so that should open the right page and then we can actually do await browser do screenshot I don't remember how this works as well so let's do Puppeteer screenshot and we have to say page do screenshot and we can set the path okay so actually page do screenshot and let's set a path which is going to be screenshot do JPEG and let's see if that will work so let's run it again and we open the browser and with took a screenshot and that's it so now if we do xtg open screenshot do jpeg then here we have a screenshot of the Wikipedia page so now let's send this screenshot to chat GPT so let's just go to our vision and let's change the image to screenshot.jpg and what if we change the text to how old is Sam Altman then do we actually get the response let's run Python 3 vision. piy and Sam Alman was born on April 22 1985 assuming the current year is 2023 Sam Alman would be either 37 or 38 years old okay so it didn't actually read it from the Wikipedia article which is kind of disappointing but should we actually say according to the image I sent how old is Sam mton so maybe I have to tell it explicitly to look for the answer in the image because of course it has this information of when he was born already in the model Sam Alman was born on April 22 1985 and the image does not provide the current date to calculate his precise age at the moment of the screenshot however as of the last knowledge update in April 2023 and given his birth date okay so in fact in the screenshot there is the age very small here so maybe chat gbt just can't read this small of a text so let's actually ask what can you read from this screenshot and let's send the request again okay and we got a lot of stuff so actually did it write exactly what is in there wait a minute it can read the age why does it not answer with the age that is interesting so perhaps we need to do some Chain of Thought here maybe we have to ask what can you see in this image and then ask the question but that's really cool so it summarized everything that was on the page very interesting now I wonder if there's like a different mode it has to set itself to like to read like is there like an OCR version and it detects that okay they're asking for what you can read from it so let's just pass it through an OCR not quite sure so what if we say something like this extract the following information from this screenshot the layout of the page navigation menus Etc and the content of the page or on the page and then I guess what we could do is we could send another message after this so after we get the message text from here and let's actually make this a list so let's say this is messages and let's set the messages or actually we don't want to do this so I was going to add this as messages and then I would append to this but then we will send the image again the next request so we don't want to do that so we literally should send send again a request but then we should say just this content is a string how old is Sam Alman and we should add here for example the assistant message or we might add just the information in the user message whatever let's say role is assistant and content is going to be the message text and then we get the message text from this one so that should be the answer let's see if we can now get the age so run it again and let's see how long it takes to get this answer hopefully not too long but it is taking quite a while now because we are sending two request now and actually I'm now sending the second message as well to the vision preview and it said again the screenshot does not include Sam alman's birth date or age what is this because are you use the vision preview what if I just say gbt 3.5 turbo for the second one and let's actually print the message text here again print message text and let's print some new lines and let's try again okay and now we actually got the answer and it says that he is 38 years old and we have the information over here somewhere yes age 38 so let's try one more time just to ask the question directly from the vision API so how old is Sam Alman and let's just exit here maybe sometimes it doesn't extract the information correctly so let's run this again as of the information on this Wikipedia page snapshot Sam Altman was born on April 22 1985 if today's date is on or before April 22 20 23 he's 37 but he's 38 that's very interesting so we can extract the text from the page with the vision but we can't directly ask questions about what text is on the page very interesting now one thing I would be interested to do is to get the weather information so what if I do like um weather in Florida so basically if we just go to Google and we ask for weather in Florida it will show the information here and now this turned out to be very difficult to extract from the Google source code so what if we just pass this URL to the screenshot so if we now run node screenshot do JS then we should open the Google search results for weather in Florida and we should take a screenshot of this and now we should have that so what if we then go to our vision. piy and again we we pass in the screenshot and we ask what's the weather like in Florida and let's run this Python 3 vision. pi and we got an answer the weather in Florida as shown in the image is clear with a temperature of 76 fahit or 24 Cel the humidity is at 72% and there is a 2% chance of precipitation the wind is blowing at and so on so it works so how could we make this so that we can actually just tell chat gbt to fetch some information and then it would use the chat Vision to do it and the Puppeteer so let's create a new script let's create at Vision craw dopy and I guess I should name it Vision uncore craw dop and let's just copy this code in here at first and what are we going to do first of all we should ask the user for the prompt so let's say that the prompt is going to be input prompt so then we can type in whatever we want and then we should send that to chat GPT so we are in fact going to do model chat completions create and let's use GPT 3.5 turbo and we are just going to say the prompt and we're going to add a system message so roll is system and content is going to be the following you are a web crawler your job is to give the user a URL to go to in order to find the answer to the question and let's say respond in the following Json format and let's use the new and let's use the new Json mode feature so I think this is going to be a bit easier than function calling so we can just say that the Jason format needs to be URL and we have to have the URL here so put URL here and then we are going to set response format to type of Json object and now I am going to extract this into a messages list and I'll pass it in here messages and then we should get from this the actual URL so we can do this again and we can say that the Json or let's call it message Json is json. load and we load the content of the message and of course we have to import Json and then we can say that the URL is message Json URL and then let's make this screenshot doj take the URL from the command line so let's say this is process. arv one and let's actually extract this into URL and let's say cons URL is that do I need to import process let's check my index.js no I can just use process. arv directly so now we can pass in a command line argument which is a URL that we're going to take a screenshot of so then we can import sub process and we can just run here subprocess do run and we can just say screenshot do JS actually we have to say node screenshot.jpg and URL and is this going to wait for it I think it will wait and we don't really care about the output at the moment because we will just have a screenshot jpeg after this hopefully it is not asynchronous by default so then we are going to convert that screenshot into Bay 64 and then we are going to send the message which was called prompt to chat div Vision with the image URL so if we say here what is the weather like in Florida then it will say okay you should go to this URL whatever it will be and then we will take a screenshot of that page and then we will send again what is the weather like in Florida with the image and then we get the response goes back and I guess we can remove all this stuff and let's print here gbt says this and let's call this up here let's say U and I guess we could print here print crawling URL so we know where we are trying to go at least and let's see if this works so let's run Python 3 Vision craw and it will ask me for prompt so I'm going to say what is the weather like in Alaska let's change it just so we know that it's not going to use this old screenshot I might even delete it and let's do it and we got an error and I have to use the new model so I can't use GPD 3.5 turbo I have to say 1106 and then I can do it what is the weather like in Alaska and we are crawling weather.gov okay and we got an error process. rv1 is not a function what why did we get that did I need a semicolon here process rv1 it's not a function wait a minute process. RV and we're not importing any process let's see again let's just do this node screenshot.jpg google.com and we actually get an error process Arc V1 is not a function why is it not a function wait a minute what is going on here oh wait a minute I scrolled too much up so actually now we got protocol error cannot navigate to invalid URL um is this because I don't have quotes around this thing no we still got a an error let's do console log URL and let's run it again so what is it wait a minute oh of course because we are running node not screenshot doj so this actually arv2 all right so let's try this again okay now it's working so of course arv1 is usually the first argument but we are running no which is a program name and then screenshot doj is the first argument and then this is the second argument all right so let's try Vision crawl again Python 3 Vision crawl what is the weather like in Alaska and we are crawling something and this is in Anchorage Alaska so it should work let's see now we might want to only wait until Dom content is loaded because now it's waiting for the whole page to load because we said it to network idle so it's going to take some time because there's a lot of ads and stuff but now we did it and we got an error so what was the error did we actually get a timeout we did get a timeout so I am going to do the following I will say Dom content loaded and let's try this again what is the weather like in Alaska and do we have a screenshot here we have some screenshot what is this screenshot okay it's just a Google screenshot all right so let's run this again what is the weather like in Alaska and now we are going to crawl hopefully the same place but actually the API got stuck once again so we have to start over let's try again again what is the weather like in Alaska if you know a good solution for the problem where chat gbd API gets stuck very often then let me know in the comments now you can do retries but it's kind of a pain in the ass every time but we got an answer which is I'm sorry but I am unable to provide realtime weather updates or current conditions from images so let's actually open this screenshot and it wasn't the same website again and as you can see there's nothing really here there's no information so this was the wrong website so let's try one more time what is the weather like in Alaska and now we are using acuweather from Alaska so hopefully this works but now I think we do the screenshot too early okay so we still have to do Network idols zero or should we do still D content loaded and then just wait for a while so if we do await new promise and resolve and reject and we are going to do set time out and we are going to resolve in let's say 5 seconds is that too long 4 seconds so now we wait 4 seconds before we take the screenshot this is very difficult often with Puppeteer like how long do you wait but let's try to now get a screenshot from Google does it still work so we open Google and then we wait for 4 seconds and then we are going to close okay let's see now if we run Vision craw again what is the weather like in Alaska and we're just going to go to weather.gov which is not really a good place to go so we should have some sort of thing that if it doesn't find the information then it will try another website and one thing we might do to make this a little bit more predictable is we might set the seed here let's just set some random seed here and let's see if that will work better so if you don't know already this seed is something you can set now in the completion so that when you send the same seed and the same message then you will get the same response at least almost same so if you find the correct seed that gives you a good answer then you can use that seed and then get reproducible answers so let's see what kind of answers this seed gives us what is the weather like in Alaska and again we go to weather.gov so this is not a good seed so I'll just stop this and let's use something completely different what is the weather like in Alaska and now we still go to weather.gov so I'm in fact going to change first of all this seed and I'm going to set here in the system message pick a URL that probably has the information directly on it and let's try if that works better what is the weather like in Alaska and we are stuck in the API once again so let's try again and hopefully this time it will work okay now we went directly to Alaska so now we actually we can't find the page you're looking for so damn it let's change the seed let's use 69420 and hopefully that works what is the weather like in Alaska and again we are slow so we have to try again what is the weather like in Alaska and now we go to weather.gov again damn it why did it work the first time all right after a bit of tweaking around I got it to work so I changed my system message a little bit so I change this to go to a direct URL that will likely have the answer to the user's question and then I set the seed to 2232 and then it went to this weather.com we/ today and it actually got the answer so let's see if this is now reproducible so if I run this again and I say what is the weather like in Alaska then hopefully it will go to the same URL again and it seems like we are not going anywhere what is wrong with the API to today I am going to try this one more time what is the weather like in Alaska and now we're going to go to weather.com again and let's see we go to the right place and we wait 4 seconds and we took the screenshot and we're going to ask chat gbt Vision what is the weather like and here is the weather so it works so what if we ask something like what are the latest news about open AI it is going to go to blog. open.com and error name not resolved okay so we really have to add some error handling so we should probably get from this subprocess run the actual result code so how can we do that we can say result how do we get it subprocess dot result no get output there one get status output what does this do okay it Returns the exit code and the output as a tuple so we can do that and we can say that the exit code and the output is going to be that and let's say if exit code is not zero then we get an error so what are we going to do there then we should add to the messages messages. append and let's add a dictionary with the rooll user and the content I was unable to craw that site please pick a different one and then we should go back so how should we do this let's say while true we are going to send a message so we'll send the messages and we expect Json output and we get the URL and we crawl it and if we get an error then we add to the messages I was unable to crawl that site please pick a different one and then we are going to go back up here and we will send that message and we should get another URL back and else if the exit code is zero then we are going to break and then we are going to get the screenshot and we are going to send it with the prompt and then we're going to get the answer and can I set here a system message roll system content your job is to answer the users's question based on the given image or let's say given screenshot of a website if you you can't answer the question simply respond with answer not found and then what we can do is when we get the message text then if answer not found is in the message text we should do messages. append and we are going to append a message with the role user and the content I was unable to find the answer on that website please pick another one and we should then again go back to this but we are not in the loop anymore so can we move this whole thing in a loop so if we indent this and here we say while true and then if the answer is not found then we append this and we go back to the beginning and we have a new message and else we are going to print the response and I guess we can actually say that there will be a new message which will be the role user and the content is going to be input and it will say you again so then we can ask another question and then we will go back here and we will create a new URL based on that question so let's see if this will work and I'll add an extra new line in the beginning so let's see what will happen what are the latest news from open Ai and we go to blog. open.com and we should then craw to another page because we can't find this one but actually what are we doing right now because we are not opening the Puppeteer script so let's stop this and did something go wrong here let's actually print the exit code and let's exit and see what is the exit code and we might as well print the output as well and let's try again what are the latest news from open Ai and the API is stuck once again so I'll start over what are the latest news from open AI crawling blog. openair.com and we're not printing anything which is interesting because actually this should now be a string so let's ask chat gbd how we can actually do this properly how do I subprocess do run in Python and get the result code and we are using now the code interpreter this is kind of not a good update when they change it so that you can just ask anything and then it will pick what tool it's going to use because now I have to wait for a long time when it does something I'm not sure what it did maybe it checked it let we check the code okay so we can say result equals that and then we can say return code okay so capture output and text okay so let's say result is that and we are going to capture the output and text is true and then we can say that the exit code is going to be result do return code and output will be STD out so output is going to be result do STD out and let's see if that will work run it again what are the latest news about open Ai and sorry I have to change this to run what are the latest news about open Ai and and we are crawling and now we actually opened it and we get an error okay so now if we remove this print from here it should actually do it properly and I will add a print here error trying different URL and let's add also here error answer not found and let's do it again what are the latest news about open a I and we craw this and we're going to get an error trying different URL and now we go to open.com newws and do we load the page on time we do not so I am going to add some more wait time and I guess I might as well just say Network idle Z and I guess we can add a timeout of like um 10 seconds so then we will crash and then we will try another URL if we get a timeout so let's do it what are the latest news about open Ai and we go to the wrong URL and we're going to crash and we're going to try news and we're going to load the whole thing including ads and everything although there's probably no ads here but wait a minute we didn't load it damn it why are websites so slow today so should I both wait 4 seconds and wait for um Network idle zero there's another one like Network idle 500 I guess it means like when the network is idle for 500 milliseconds or something like this let's just do Network idle zero and then wait 4 seconds that should be enough and run onun it what are the latest news about open Ai and if it's still too slow then we can try another website okay and this one will not work and then we try the news website hopefully unless the chat with API is again stuck how can this API be so bad I'm going to do this I did this once and it seemed to work so if I say model do timeout is 10 then I'm not quite sure if it will like automatically retry after 10 seconds because I saw something in the open AI Library code that there is something about that retry let's see if this will work what are the latest news about open Ai and we go to blog which doesn't work and we go to news which should work but it's very slow so let's see if we now can actually load this page and we are actually we get a 404 error so we should now say we can't find the information so let's continue with a different URL and we did say answer not found but what are we doing now we should be getting the next URL so now we should be here and we went to news again so why did that happen because we should have the message here that we can't open this URL then why did it show the same URL again is this now because of the seed well now we went to blog let's see what we can find in the blog so we took a screenshot of the blog let's see if there's anything there the screenshot you provided shows the open a Blog with two posts visible the titles of the latest updates are opena announces leadership transition opena data partnership unfortunately I cannot provide further content from the blog posts as it is not included in the image okay well at least it kind of works so now of course we would have to tell it somehow what links are on the page and then it could like click on some links but this video is getting too long so I am afraid that I have to stop but let me know in the comments if you would like me to continue working on this project and maybe make it like click on different links on the website but let's try something else now what if we just say summarize the latest news from BBC then can it go to bbc.com and then take a screenshot of it and then tell what is happening in the world and this is always a problem once again so we get this welcome to to the BBC website maybe some cookie policy or something so we can't actually see what is going on so that is another issue how do you solve that now you could solve it in such a way that chat gbt would tell you that hey there's something you have to click out of the way and then it could actually do that but we actually got seven things and actually these were on the page apparently and here we have also Sam Altman is reporting Ed to return as open a boss according to the new in principal agreement but hey that works actually pretty well let's try another weather again what is the weather like in California and we are going to go again to weather.com and hopefully this is the right URL and I believe it is right but we lost it it's down here hopefully the ad is not too big that we lose the actual screenshot now I think that with Puppeteer you can actually take a screenshot that is the height of the whole page but now again this page is waiting too long and then it will crash okay so can we take a screenshot before we crash so what if we wait wait a minute how can we do this because we have to call go to with the wait until but I want to take a screenshot before this fails I could do like here set timeout and then I could say page. take screenshot so then we would presumably take a screenshot after 4 seconds but we should call this with a wait but can we actually can't we do just async and we can do a wait so is this going to work so we go to the page we wait for 4 seconds we take a screenshot let's make it five and then we could do try this and we could catch a timeout exception what is it called do I still have the timeout here I don't have it so there is actually I'm muting it here so there's a timeout error so I can import timeout error from Puppeteer and then I can catch a timeout error error sorry how does this work we just catch the error wait I forgot how to use JavaScript where am I using timeout error okay I just catch the error and if it is an instance of timeout error then we do something so if error is instead of timeout error then we can do process. exit to can we give it like an exit code so then if we get a timeout we exit with two and then we should still have a screenshot does that make sense so then we can check in our python code if the result code in fact let's just do if it is one because I'm guessing it will be one if we get a regular error but then here we can change it to two so then we will actually try to read the screenshot still so we should in fact delete the old screenshot first so before we run this we are going to say os. remove screenshot do JPEG and let's import OS so let's see if that will work I mean I guess we might not have the screenshot now if we can't take the screenshot either so I guess we can do something like if exit code equals 1 or Os path exists in fact not OS paath exists screenshot do jpeg now actually we could just check this so we don't have to care about the result code if we have a screenshot then everything's fine if we don't have then we couldn't CW it so then we don't actually have to do any of this try catch assuming that this works so let's see if it works let's run this again and let's say what is the weather like in Cali fornia and we go to weather.com and let's see what kind of result we get so we are now loading and let's actually see do we have a screenshot already we actually have a screenshot so we took the screenshot is it actually what we wanted so if I do xdg open screenshot jpeg then we actually have it here so now we could take the screenshot even though we are not loading the page and we got the response back so now it works even though we have a slow page we were able to take a screenshot now we might even want to take like a screenshot every second or something while it is loading so then we get the latest one or even we could send all the screenshots to chat gbt so that if there's like a popup that comes that we actually get the screenshot before the popup arrives but it works now I've almost been filming for 2 hours but I want to change this I don't want it to refer to a screenshot because I want to just answer the question the user doesn't need to know how the information was fetched so let's add here answer the user as an assistant but don't tell that the information is from a screenshot or an image pretend if it is information that you know okay let's see how that changes the response what is the weather like in San Francisco if I can write San Francisco properly and we're going to go to some long URL and we had trouble finding your page so let's now see are we actually going to check another page okay and this is again the frustrating thing about chat GPT it says that the weather is not provided here but it should answer with answer not found probably because we added this extra stuff here what if I just shorten this your job is to answer the user's question based on the given screenshot of a website what if I just say answer the question based on the given information as a helpful assistant and let's try this again what is the weather like in San Francisco that's definitely not how you write San Francisco okay and we got the error page and then we should get an error and we should try another URL damn it then try another website okay then we try another website and now we actually went to the right place but why can it not answer with the correct thing maybe I should update it to use the function calling now is function calling available with the vision preview I guess it is and of course we might use tv4 for this okay and here is the information great now the answer is very long I might not want a long answer like this but let's actually check if I go back to this version that said don't talk about the screenshot and actually it's gp4 already I was going to change it to gbd4 but of course this is gbd4 already and actually if you can't answer the question this should be a comma and let's say answer with the code answer not found and nothing else let's try one final time what is the weather like in sand Francisco and we are going to go to the correct URL actually now now I guess it depends on the system message because that changes what are the previous tokens so then the probability of another token will change so depending what the system is is you're going to have a different response but now we should get the answer and here it is the weather in San Francisco as of the latest information is clear with a temperature of 50° F and all this other information so let's try one more thing because I have one minute left before I have been filming for 2 hours so I'll say what is the current stock price of Tesla because I was trying to do this with my Puppeteer GPT but often it didn't work because the page that it loads it did have the information like in big letters but from the HTML it was very difficult to get it so hopefully the vision will get it if this didn't Now get stuck again in the API I really have to implement a retry and we're actually no stuck why are we stuck let's try again what is the current stock price of Tesla and the API really doesn't want to work today so let's try one more time okay now we go to yahoo.com quotesla and do we have the information here if we scroll down a bit here it is so you should see it from this let's see if GB division can read it from here so it's 24120 and the answer is 24324 where did it get this one okay let's actually see xcg open screenshot do jpeg ah wait a minute at close pre-market I don't know about stocks so I have no idea which one I should use I'm just assuming that the bigger one is the correct one but this is the one it quoted so I guess it's good enough 24324 okay I am quite happy with that so this is the end of today's video I hope you enjoyed it and if you want to try this out I will leave a link in the description to my GitHub page and leave your comments down below how should I improve this or what should I do next with the chat gbd API and make sure to subscribe if you want to see more videos like this in the future thanks for watching and I will see you in the next one

Channel: Unconventional Coding

Views: 119,066

Keywords:

Id: VeQR17k7fiU

Length: 56min 25sec (3385 seconds)

Published: Fri Nov 24 2023