Access GPT-4o Voice & Vision EARLY Through Microsoft CoPilot AI!

so Microsoft had their very own AI event today it wasn't live streamed to the public for some reason but the folks that were actually there the press that was there made quick work of just getting all the information out so we know everything already now the reason that I am even somewhat excited for these announcements is because we do know that open aai and Microsoft work very very closely together and it's happened before where Microsoft not only themselves get access to exclusive open AI technology but Microsoft actually end up sharing that technology with us and it looks like that is what's happening today if you want an example of this happening in the past well Dolly 3 was actually released in the Microsoft Bing image Creator web app before it was even in chat GPT itself and it was also way less censored in that Microsoft Bing app so this time what are we getting an exclusive taste an exclusive Peak at well it actually appears to be some of the multimodal GPT 40 capabilities if you haven't already I suggest watching my breakdown on GPT 40 because it is an incredible model it's very impressive and it's way more than meets the eye so I'll link that down below but GPT 40 with voice and vision capabilities are going to soon be available as an integrated part of Microsoft's co-pilot Ai and this is like an AI assistant that lives on your Windows computer they had some live demos that we will take a peek at but they're also teasing GPC 40 integration in Xbox for live in-game advice which is interesting anyways without further Ado let's actually dive into some of this stuff so first up this comes right from a Microsoft employee this is a nice little video they put together basically rounding up all of these features for Microsoft co-pilot AI with PCS so I'll show that to you guys now [Music] a [Music] n [Music] so right off the bat lots of really promising features here you know they have this npu most powerful AI processor thing A lot of people are suspecting this is more of a marketing gimmick than anything else but it can do local Ai and we'll get into some of that this is probably one of the craziest features and the one I am probably the most skeptical about this is recall I think this is indeed powered by GPT 4 essentially you have this ability to live recall anything that you did on your computer at any time so your computer is essentially tracking you constantly and remembering everything you do in every application kind of like history on your web browser but for your entire computer and it's powered by AI so feasibly you could do some very natural searches even if you were using an app and you don't remember what it was called you could search your history and find it you know it would not only be able to find the photos of your road trip but it could also find the little videos you might have edited or something or maybe some documents you created before you went on the road trip everything related to that specific topic it could contextualize it all for you and that is both creepy but also really cool if it does actually work you know the ability to identify everything separately in the image and of course I mean GPD 40 is what makes all of this possible they also have co-creator and this is the AI that sketches with you so you can draw inside of paint and then have it you know enhanced by Ai and this is something that we've definitely seen before I'm going to go into this a little bit more but it does indeed run locally on that um npu processor so that's actually really really quite impressive but more of a gimmick than anything else I mean you're not really going to be using it to actually make projects it's not at that quality level just yet this is amazing so this is live captions that are I I assume also powered by AI powered by GPT 40 because it's the only realtime multimodal AI that I can think of but you'll be able to live chat with people apparently in do translation live so no longer will you have to do some annoying text translation you'll actually be able to video call people and just talk to them and I think these are just text captions unless you know it's going to to be using the open AI voice generation as well which would be quite interesting again they didn't go too deep into this they're just kind of showing you what's possible and I mean in this example we do see it is a a text generation but you know it also works for media as well so if you're watching something or trying to learn from a tutorial that's not in your language it should be able to translate it live with co-pilot AI built right into Windows which is pretty cool and so they also go into co-pilot which again this is just a bunch of chat GPT esque features so brainstorm you know generate an image and again creating an image if you guys saw my GPT 40 video is that using the do model or is it using the new GPT 40 image generation capabilities will we get access to those that's what I'm really interested about with this one because if you remember those samples I mean those generated images were incredible the capabilities at least in my opinion were beyond belief also data analysis again this also to me implies the new gp40 data analysis features and summarization with long context so just all built directly inside of Windows instead of having to navigate to a separate website which is a little bit nicer so overall though big strides from Microsoft their partnership with open AI is absolutely paying off big time now that we have like a basic overview and understanding of everything new that's coming to Windows in relation to AI let's get a little bit deeper cuz I do have some clips from the presentation right here folks is a demo of the recall feature to show you recall I'd like to welcome our product leader on Windows who built this incredible new capability karolina Hernandez welcome [Applause] kolina thanks Carolia give us a overview of recall please yeah absolutely I would love to uh my team developed recall to help you find anything you have ever seen or done on your PC we Lage the power of the mpu to ensure content stays on your device which makes the experience fast and maintains your privacy okay let me show you a few ways that recall helps me like all of you I am constantly jumping between different tasks throughout my day managing my time between a career I am passionate about staying connected with my family and friends around the world and the hobbies that I love Reco helps me get through all this tasks more quickly my family and I have been searching for the perfect dress for my grandmother who lives in Columbia to wear at my cousin's wedding I've been browsing Pinterest in shopping websites for some options that I can share with her after looking for options for a few weeks I wanted to go back and see some of the favorite blue dresses I have found but I didn't keep all my browser tops open first thing I'd like to note here it does seem like recall will actually be a preview app beta access feature just like co-pilot was so again this is promising that we're going to get access to this thing sooner rather than later and I do actually have a date for you guys but also so it's going to be an app it's not going to be something that is built within another Windows app it's its own separate thing so I actually like that I think it'll make it a lot easier for us to use and navigate when we first test it out search for blue dress and it pulls together all the Joes I have seen now Grandma decided that she wanted to get the awesome panso that we shared in our Discord chat about 3 weeks ago now scrolling through multiple weeks of chat messages to find that specific Dr would have taken an eternity instead with recall it was easy I refined my search using the scrip of language blue pan suit with a sequin lace from Abuelita and as you can see Reco quickly found the Discord chat my family and I were using to discuss the options and here it is the dress that aita liked recall analy wow okay that's actually pretty dang cool man it was able to just pull up the specific Discord chat although I will note here that this is a Discord message inside of the Bing browser so again they do say it it it it watches everything you do on your computer so it should be more than just your browser but I'd love to see it working with individual apps and does it work better in some apps than other apps there's a lot of questions with a such a broad and crazy feature like this I is the snapshot and it gives me options for what I can do next in this case I want to click the link which takes me back to the right page where I can make the purchase and now Grandma has a gorgeous outfit for the wedding this is a great example Carolina that I think is going to show how recall is so helpful for people yeah that was a pretty dang good example I must say I mean that's incredible it can give you this view this sort of macro view of everything you've been working on if you're working on different projects it will show you all the different proces that it took to get to this one accumulation of whatever you were producing I definitely could see this being useful in video creation to kind of map out my progress exploring a topic or exploring different AI apps man that's pretty wild peacock but it wasn't anywhere in what you were searching for how did recall recognize that and what's going on behind the covers yeah absolutely so recall can understand that peacock is in the family of the blue hues so it creates that semantic Association that is used to help me find a dress recall is also essential for me at work like many of you my team and I spend a ton of time collaborating on presentations and documents for example over the past few weeks we have been collaborating on a presentation for a product pitch the deck was created by my partners in the marketing team and when I finally got around to add my content I could not figure out where to look for the deck I tried to remember was a deck shared in the call maybe I had seen it in an email or maybe I could go and find it in my recent files in PowerPoint I don't really have time to spend clicking around but with recall I no longer have to go on a searching Expedition I remembered in the outline we had added a purple chart with writing a purple WR um a chart with purple writing uh and to find the PowerPoint all I had to do was simply use my voice to search for that clue that I remember yeah so it's that easy if you are forgetful about something you needed and that you worked on quite some time ago you can use very natural language and very fuzzy non descript memories to go back and pull that information up which is yeah again very useful and very impressive but does it actually work is the main question that I have and then the second question is obviously about privacy and we'll talk a little bit more about privacy later in the video in in a few seconds Reco found the deck for me and notice that it took me to the exact slide that I remember saving me even more time this is actually a really good uh point so fiend here points out new hard drive partitions 100 GB for Windows and one tbte for recall everything is on local for privacy concerns now in my opinion this is actually a cooler demo and basically what we're about to see is the windows version of that chat GPT GPT 4-o powered Mac app so this right here is using the voice and the vision capabilities and again we do have a release date for all of this so we very likely are getting access to this stuff before open AI would give it to us publicly all right so we go and we start to share our screen the co-pilot app with uh you know chat G Going hey I'm doing great and it looks like you're about to dive into some Minecraft ready to build explore and maybe dodge a few moms yeah I'm trying to figure out how to make a sword my son plays this game all the time and I have no idea what I'm doing no worries I'm here to help you impress your son I see you've got a crafting table in front of you which is a great start to craft a sword you'll need some materials can you open your inventory just press e on your keyboard yeah do I have what I need let's see you've got some sticks perfect for the sword's handle but it looks like you're missing the material for the blade you can use wood Stone iron gold or Diamond let's start by gathering some wood or stone shall we head out and find some resources yeah that sounds great sounds very naturally oh no it's a zombie run you need to get away from it fast either Sprint away or quickly build a tower of blocks straight up to get out of its reach if you can find a shelter or dig into the side of a hill do it now just make sure it can't reach you all right I think I might be safe now cool that was a close one great job finding shelter okay that's that's really interesting man it doesn't have that same flirty tone that the other voices do but it still sounds very natural and the way that it reacts to him and the way that it talks to him just gives you this feeling of like I'm actually hanging out with someone and they're watching me play this game and are actually interested and can help me so I kind of now I'm starting to warm up a little bit more to those Xbox Live features where it's like yeah you could have this AI companion that just chills with you and plays the game with you it adds another level of like a social experience even for a single player game very very strange but intriguing nonetheless that's something I'd have to think more about before I really gather my thoughts on it's something I haven't considered before but I'm definitely excited to try it out that's for sure I mean how do you guys feel about that would you want like an AI companion that just hangs out with you and watches you do your work can help you by making suggestions being like hey no maybe you shouldn't format this that way or maybe you should try this instead or hey I'm having this issue right now with my software what are some things I can do about it just having it open on the side to help you right then and there man it's got me thinking oh and not only that not only would you want it to watch your screen and talk to you but would you also want this thing to utilize your screen how far are we away from just having it play Minecraft for you so to speak or do your work for you do it alongside you even I mean this must be coming in like what a year no doubt in my mind that open AI is hard at work trying to make that happen so yeah again just diving a little bit deeper they do have that feature directly inside of paint where you can draw in real time and then it uses AI to enhance your image or bring it to life and this all runs locally on that AI processor they also have the ability to restyle or reimagine photos with AI directly on the computer so it's cool that it runs locally but that's about it for that this is nothing we haven't seen before on websites and not only that it's been available on websites for free off and on for the better part of I don't know 5 months something like that real-time AI drawing AI photo restyling so cool cool Microsoft but that's not really that big of a deal and so here on the blog post they also mentioned that they have some integration with a bunch of apps to use this new AI npu processor again it does kind of feel like a marketing gimmick because it only is supported by these few apps so obviously you know the ones that are built into Microsoft the restyling the painting Adobe Photoshop that's actually a pretty legit one I must say cuz I do use Photoshop every day and I know plenty of people use Photoshop all the time so essentially you get like a a little bit of a performance Advantage what does that really mean though because I can still get the AI features from photoshop on my non npu Windows computer so I don't know what the benefits will actually be maybe it's background removal something like that npu accelerated magic mask and da Vinci resolve that's another pretty useful one but again it's just one feature inside of Da Vinci cap cut also has again just background removal nothing too crazy this is extremely Niche I've never even heard of this but I guess you can play games with like your webcam and it can watch your face or something liquid Tex just makes annotations using AI features this is actually the only thing that I can find that's genuinely exclusive but apparently you can break down or remix any music track with their highquality version of neural mix that exclusively runs on this npu and that's inside of DJ Pro but again that's extremely Niche but it is actually an exclusive feature so that's kind of cool so yeah guys it really seems like there's some pretty incredible GPT 40 powered features coming to Windows along with some small AI features powered by their new AI npu processor but the major concerns I've seen from people on Twitter are all about privacy issues they don't want Microsoft watching them every step of the way and Microsoft says that everything is going to be stored locally they say that nothing is going to be sent to the cloud the real question is do you believe them because even though Microsoft addressed all of this people are not convinced people still don't trust them and you know I can't really blame them and also what if someone just hacks your computer what if you download a virus and then oh you get they have access to everything even the stuff you did in the past so there's more security issues even if everything does run locally anyway so let's get into availability because that's the big deal right there's a lot of insane AI capabilities this natural voice this natural language co-pilot that can chat with you while you do your work chat with you while you play your video games pull stuff up on your computer from some time ago well they say that availability is starting on June 18th 2024 and that's actually not too far away I mean that's like 3 or 4 weeks away I can guarantee that we won't get those incredible gp4 features The Voice generation capabilities the image generation capabilities before then so we'll have to see exactly what we get access to on June 18th I hope it's the whole slew of things but I have a feeling that some of the features especially recall are going to be a slower roll out but I'm really hoping that the natural language Vision co-pilot assistant is going to be available at least on June 18th because that will be that will be my first access to True multimodal GPT 40 oh and if you're wondering how I know that open AI isn't going to release them well because on the website open AI has this listed as being months away so it's going to be quite some time before on your phone you get those demos like we saw now to wrap this up I want to know what you guys think for sure is this actually useful is it more like marketing I'm glad to see that they are thinking about Ai and they are thinking about ways to implement AI into Windows because that's going to be important this technology is going to change everything in my opinion and you got to stick with the times you can't fall behind and Microsoft's again their their partnership with open AI seems to be keeping them ahead in the game because Apple doesn't have anything like this at least not yet but then again apple is in talks with open AI about some sort of a partnership I don't know folks everything sounds really awesome but it remains to scen right whenever there's an announcement we say that looks really cool but does it actually work so subscribe to the channel I guess because of course I'm going to be testing all these features out and doing not only a first impression but probably a deeper dive as well thank you so much for watching everybody please check out my Twitter for live updates and live reposts for stuff like this and also check out the Discord because the community on there is fantastic and they jump on everything thanks for watching I'll see you guys in the next one goodbye

Channel: MattVidPro AI

Views: 80,271

Keywords: mattvidpro, mattvidpro ai, gpt4-0, gpt4-o, microsoft co pilot ai, ai news, best ai tools, gpt 5, llms

Id: jcvatirXHXU

Length: 20min 56sec (1256 seconds)

Published: Tue May 21 2024