Google Announces STUNNING AI Agents | Google Cloud Keynote AI Agents

it looks like Google is going all in on AI agents the latest Google Cloud next keynote presentation which covers some of the new product developments across Google had one major theme and that is AI agents AI agents for customer service AI agents for helping your employees get more stuff done for marketing for driving for analyzing vast quantities of data for creating content and podcasts and videos AI agents for cyber security for coding for deploying new e-commerce Pages AI agents for everything in this video I've cut the most important announcements and AI agent demos that they've shown so you can understand better what Google is about to roll out our biggest announcements today are focused on generative AI you know customers have quickly gone from experimenting with generative AI helping it to answer questions to make AI predictions and I'm now building generative AI agents agents process multimodal information simultaneously conversing reasoning learning and making decisions agents can connect with other agents and with humans and they will transform how each of you interact with Computing devices and the web itself you know our customers are building early versions of AI agents our foundation model Gemini which has advanced reasoning skills with multimodal information we're the only Cloud to offer widely used first party thirdparty and open source models vertex AI to access tune augment and deploy custom models and to build agents shortly before this presentation Google rolled out its latest model Gemini 1.5 Pro with a 1 million token context window and here's the important thing to understand Google has seemingly been struggling a lot in the AI race but just in the last few months I think they made a pretty substantial breakthrough with the 1.5 pro model they've adapted the mixture of experts architecture so the model is not similar to GPT 4 which uses the same architecture which by the way Google Google pioneered along with Transformers they've pioneered they've created a lot of this technology open AI was able to use that put it together better innovate iterate and get it out faster but now Google is reclaiming that technology and advancing their models and here's the thing Gemini 1.5 Pro with the 1 million token context window is good if you've played around with some of their other stuff before and found it lackluster you might be dubious about some of the claims that they're making in the keynote but again as I've been using that model as many other people online have been using that model and Reporting back the results it's becoming very clear that it is very very good now I'm sure there's some areas where GPT 4 is better some areas where GPT 4 is worse and then there are some areas where the Gemini model is clearly better like for example the context window so let's take a look at all the big announcements from Google and keep in mind that this will be rolling out very rapidly to all the Google customers all their Enterprise Partners all the big businesses that will want to give this technology to all of their employees and as those employees and customers as they get their hands on and start using it Google will get more and more data into how people are using it how to improve the products more and more kind of starting that flywheel of improvement so whether or not you're cheering for Google to win this AI race it's important to understand that they're gunning for the Enterprise space they're really looking to put their AI products into these massive global companies and getting everyone to use their products you know the opportunity for customer agents is tremendous to help each of you build customer agents faster we're introducing vertex AI agent Builder you can now create customer agents that are amazingly powerful in just three key steps first you can use Gemini Pro to create free flowing humanlike conversations with text voice images and video as inputs and personalize them with custom voice models second you can use natural language instructions to control the conversation flow and guide it on specific topics you don't want it to discuss such as current events in the same way that you train your human agents you can also control when it hands over to a human agent with transcription and summarization of its conversation history to make these transitions extremely smooth third you can improve response quality with vector-based and keyword-based search to connect your internal information and the entire web you can also use extensions to complete tasks for customers like updating contact information booking a flight ordering food and many more let's take a look at an example of a customer agent in action let's head over to my favorite store simple fashion they just launched a customer agent and it leverages Gemini and Vector search to deliver a seamless shopping experience so last night I was watching a video of this band and I love the keyboard player shirt so I was thinking I'd really like to be wearing that shirt tomorrow night but can I find it in my size and in time to be rocking it at the con concert here in Vegas find me a checkered shirt like the keyboard player is wearing I'd like to see prices where to buy it and how soon can I be wearing it going to include the video now the customer agent is using Gemini's multi reasoning to analyze the text and video to identify exactly what I'm looking for then Gemini turns it into a searchable format how cool is this it found the checkered shirt I'm looking for right and some other great options in no time of course it never fails they only have three left in my size I don't want to miss out on wearing this shirt so I'm going to go ahead give the store a call and ask them to set it aside for me I'm so close to having this shirt for the concert let's give the store a call hi there this is the symbol fashion customer agent at South Las Vegas Boulevard am I speaking with Amanda yes this is Amanda great thanks for reaching out Amanda I see you had a session on another device I've sent you an SMS message with a link to our live chat companion if you would like to switch to chat please click the link how can I help you today I'd like to purchase the shirt in my cart with the cart I have on file absolutely I see you're also a symol fashion Rewards member looks like you have a 20% off voucher available to use would you like to apply it to this purchase yes please that would be great the shirt you're purchasing goes well with these items also available for pickup in your preferred size would any of these be interesting to you absolutely please add the white shirt and the boots to my cart great your total is $23.76 okay to proceed with the card on file yes your purchase is confirmed do you need anything else today no I'm all set thank you watching the last video about customer service agents as an e-commerce owner myself I am very excited about what I can do for my business as a normal human being that gets very frustrated with how companies go about marketing especially online using various popups and injecting pages into your history so that when you click the back button it actually takes you to a different page various autoplay ads forcing you to watch ads before you can interact with certain functionality in apps or in websites the massive massive amounts of emails that they send like I feel like that Trend will continue but now with the AI assistance so as you're talking to them over the phone there's really nothing that prevents them from giving you the hard pitch to buy something so we'll see how this gets implemented it could be a nightmare but hopefully there will be some counterbalance that will prevent these companies from really abusing it we we'll see but in this next part let's take a look at employee agents so these are agents that will help employees be more productive but also will help you as an individual be productive and some of this stuff is extremely exciting if you've ever had to comb through massive legal files like if you're trying to lease a car or an apartment if you've ever been confused about certain medical bills here in the US the medical billing industry is insane it's very complicated it's very annoying I don't know how half the stuff they do can even be illegal but having an intelligent AI assistant seems like it could be a big big help take a look there are lots of ways Gemini for workspace is helping our customers save time let me show you the magic of one example in action I've been asked to evaluate two proposals for a new payroll system I open up my drive and I see that there are two statements of work here or SS that outline the vendor proposals Google Drive is AI ready without any additional pre-work and everything here is protected with our industry-leading security all I need to do to get started is drag and drop the two files over into the Gemini side panel which is currently in preview in just a few seconds Gemini for workspace will give me an outline of these proposals so that I can have a good idea of what they're about and I could go even further and ask compare the uh uh the price let's see price of the two offers each pages each of these documents is over 70 Pages it would have taken me hours to review these docs but instead Gemini is going to help me find a clean answer to save me a ton of time and as you can see I have the answer right here in the Gemini side panel let's go ahead and dive into one of these proposal so I'm going to pick this one one of the things that you'll notice about Gemini in docs is that if will proactively show me a summary of this file so that I can get a sense of what's going on without needing to take the time to read the whole thing but before I proceed with this vendor I need to ensure that no compliance issues exist and I'm going to be honest I have not memorized every rule in our compliance rule book because it is over a 100 pages I would have to need to scour the 80 pages of this proposal and compare it manually with 100 pages of the rulebook so instead in the side panel I ask does this offer comply whoops with the following and I'm going to at mention our compliance rulebook hit enter and see what Gemini has to say okay so interesting Gemini has found an issue because the supplier does not list their security certifications because Gemini is grounded in my company's data with Source citations to specific files I can trust this response and start to troubleshoot before selecting a vendor Gemini for workspace was terrific in this example saving me a lot of time without compromising on accuracy and this technology can be applied in so many other use cases a sales team analyzing rfps a recruiter developing interview questions and so much more many of our customers are building employee agents to automate workflows that are tedious and repetitive one of the nation's leading Healthcare Providers HCA Healthcare is using our generative AI technology to help caregivers spend more time with patients and less on paperwork they're testing Katy so Katy is a nurse handoff digital assistant helping to ensure continuity of care when one caregiver shift ends and another begins it delivers summaries in everyday language to ensure that caregivers can share information easily and provides them up to the minute patient data to expedite the discharge process hi folks so I know you all want to hear about awesome AI stuff that's coming but I need to talk to you for a minute about my annual benefits enrollment see I forgot I have to finish signing up by today and as you can see I might be a little bit busy so if you don't mind let's go ahead and look at this open enrollment email together okay yep I've got a deadline I knew that thank you we got FSA stuff I've got an online portal for my company HR Ben okay there's a lot here H they included video Let's see if this makes my life easier and the video is ah okay so it's almost an hour long yeah I'm not going to have time to review all of this stuff let's see how this employee agent that we've developed using Google workspace Gemini models and vertex AI might be able to help me as you can see it's integrated directly into my Google Chat so I don't have to contact switch while I'm figuring all this stuff out first things first let's have it summarize the email and the video that it sent me SU Miz the body and attached video from my recent email with subject open enrollment closing so behind the scenes the agent is referencing that email body and its attachments as context in the prompt using retrieval augmented generation that way its response is limited to the content that matters to me the Gemini model's multimodal capabilities allows the agent to understand and reason across text audio and video from a single prompt I mean this is a way quicker read okay good and I can immediately see that the medical plans have been completely revamped this year let's go ahead and jump into the benefits portal to see more now I've already done my dental and my vision but I procrastinate I mean saved the most important plan for last my medical plan let's see how this option Stacks against my existing coverage compare these coverage options to the p PDF doc I have on the Platinum plan the Gemini model's long context window paired with vertex extensions enables the agent to cross reference large amounts of data from a variety of sources including unstructured data like PDFs leveraging Gemini's Advanced reasoning capabilities the agent is able to understand the complex details of my current plan and compare it with the new options for 2025 and since the Enterprise grounding features links me to the exact data that Gemini used to draws conclusions which you can see linked here I can confidently trust its recommendation that the gold plan is best for me and done so one last thing now that I've officially completed enrollment my daughter's going to need braces this year let's use the agent to help me find the right orthodontist my daughter will need braces this year help me find the best in network orthodontist near my home with vertex extensions the Gemini model can integrate with any external or internal API this allows the agent to join my dental coverage details directly with real-time Google Maps and places data to help deter determine the best ranked in network providers near me okay so it looks like symbol Orthodontics is the only one that offers Saturday appointments which works best for my family schedule but what do I really know about them how do I know that the agent actually picked the best orthodontists on a quick scan of the summary shows me that they're great with kids which is great I particularly love this customer review I'll go with them let's schedule a call I don't know how it's going to find a schedule though I mean I'm here the agent knows that I'm at Google Cloud next because it's integrated with Google Calendar and it can find a time to schedule the call yes please awesome I've got a call do I need anything else nope I'm good no thanks perfect nice and easy just imagine what else an employee agent can help you with so overall I'm pretty excited about that it allows everyone Everyday People to spend less time doing boring manual tasks tediously going through various documents and just kind of empowering us to do more with less time offloading some of the cognitive effort that we have to spend doing tasks that are important but kind of boring next up they're talking about a brand new thing that they're rolling out just like they have Google Sheets and Google Docs now they have Google vids which is going to be pretty interesting for making video presentations so think PowerPoint but with video but also completely automated with AI unfortunately this will make creating various marketing campaigns online a lot a lot faster and easier take a look how many times have you heard should we start with a dock or a deck well we can do a lot better I'm absolutely thrilled to announce our newest workspace app Google Vid sitting alongside Google Docs sheets slides Google vids is an AI powered video creation app for work with Gemini in bids you have a video writing production and editing assistant allinone let me show you how simple it is to get started with bids now after a full week with all of you here at next I'm going to want to share a recap video to share all the excitement with my organization when I open up bids Gemini helps me get started started I simply type in a prompt using an existing document for context now based on that prompt Gemini suggests a narrative outline for the story that I could easily customize and edit I choose an expressive style and vids Works its magic let's see what I get so wow just like that I get the first draft with beautifully designed fully animated scenes complete with relevant Stu media and music and even a generated script I'll be able to add fun videos and photos that I captured this week from my media directly by accessing Google Drive and Google photos and even without any video expertise I can build out scenes like a pro just as easily as I make slides but it doesn't have to stop there with vids you can personalize your story your way by recording yourself or by using one of our preset voices by Gemini Beyond bids many customers are building agents for their marketing teams audio and video production teams and all of the creative people out there who could use a hand we provide a tremendously powerful platform and stack to build creative agents first it takes the best image generation model imagine 2.0 our most advanced text to image technology helps businesses create images that match their specific brand requirements this is now generally available in Vortex AI image imagine strong language comprehension and photo realism capabilities contributed to Google securing the leadership position in the recent Foresta computer vision wave second as a part of Imagine we're now introducing text to live image in preview today marketing and creative teams can generate animated IM Imes from a text prompt including product images ads gips and storyboards here's an example of a day lily opening in the Morning Dew and a pot of something delicious simmering on the stove or an aerial view of a mountain range third we are proud to announce the general availability of digital watermarking for AI generated images produced by imagine it's powered by Google deep Minds synth ID finally we're announcing new editing modes for Imagine 2.0 which will make it really easy to remove unwanted elements in an image add new elements and expand the borders of the image to create a wider view for all of you let's see how the creative agent we developed with Gemini models imagine 2.0's API and Google workspace can help first things first let's get a look into sense for the look and feel of our brand what are the key Styles and themes of our brand the creative agent can analyze previous campaigns to understand our unique brand style and apply it to new ideas in this case the creative agent has analyzed over 3,000 brand images descriptions videos and documents of other products that we have in our catalog contained within Google drive to create this summary and as you can see it summarize our brand identity as bold adventurous and aspirational this is exactly the look and feel that we're going for next I'm going to cre prompt our creative agent using imagine to dynamically generate amazing new images of our tents in iconic and real outdoor settings check this out Machu picu not bad B half doome these are real landmarks accurately presented now let's work on a narrative to really help us land the message the creative agent leverag Gemini Pros multimodal capabilities to brainstorm creative captions that match each photo these look great too generate a storyboard share it with an email group The Creative agent's going to move this into slides and share it out with that group looks great but there are three requests first something for Less experienced campers second content for the symbol Outfitters podcast and third something more visually Dynamic for social media let's start with that new storyboard and you'll see in this case we're asking for a New Image new captions and a new storyboard with that new audience that relaxed camping audience take a look that looks pretty relaxing to me looks pretty good and there's the storyboard great just like that the creative agent modified our ideas for a new audience it understood our context and made the changes that we asked for now let's generate that podcast the Gemini model's complex reasoning allows me to generate a script and an audio clip in a single prompt this much longer prompt highlights our ability to do ordered operations we're asking the agent to create a podcast script in ssml or speech synthesis markup language and then passing the output to our text to speech API the result will be an episode with two people discussing tense so we'll see the script here which we could expand if we wanted but let's just listen to a little bit of it welcome to the show thanks frya it's great to be here I'm always happy to geek out about gear that sounds really good and I want you to notice the regionalisms in the speaker voices finally we were asked for some engaging content for social media let's see what the creative agent can do using our new live image generation capability we've run this prompt before and so you'll be seeing cash to speed things up but these are simply beautiful creating content like this could take days of weeks days or weeks of scouting and shooting but our creative agent used imagins new text to live image capability to make them in minutes the creative Agent shows how Google's generative AI capability can make producing custom content easy seamless and a lot of fun next up we have coding now there's a piece earlier in this keynote that I've cut out where the C of Goldman Sachs talks about the various ways in which they're applying this generative Ai and one thing that jumped out of me was saying how various coding assistant can easily add a 40% productivity boost to these software developers and I've been seeing similar numbers across the board people are saying how much easier it is to do software development to do coding when you have some sort of an AI assistant with Google 1.5 Pro the Gemini model that has a 1 million token context window and an ability to intelligently code and test code and offer Improvement suggestions This truly does seem like a game Cher this is where I do expect some pretty drastic changes some pretty drastic disruptions to occur in the software development field our Enterprise Focus AI code assistance is now called Gemini Cod assist first we're integrating the power of Gemini 1.5 Pro into to Gemini Cod assist this upgrade brings a massive 1 million token context window to coding which is the largest in the industry this revolutionizes coding allowing you to perform large scale changes across your entire code base I'm a new developer with symbol Outfitters and today we show recommended products to customers only after they've made an initial selection these suggestions are powered by our custombuilt recommendation service based on previous purchases but now the marketing department has asked me to move this feature to our homepage so that customers can see products that they might be interested in as soon as they get to our site our design department has created a mockup of what they would want this experience to look like in figma and for the developers out there you know that this means we're going to need to add padding in the homepage modify some views make sure that the configs are changed for our microservices and and typically it would take me a week or two to even just get familiarized with our company's code base which has over a 100,000 lines of code across 11 services but now with Gemini code assist as a new engineer on the team I can be more productive than ever and can accomplish all of this work in just a matter of minutes this is because Gemini's code Transformations with full codebase awareness allows us to easily reason through our entire code base and in comparison other models out there can't handle anything beyond 12 to 15,000 lines of code and even then they struggle to get it right Gemini inside of cod assist is so intelligent that we can just give it our business requirements including the visual design so let's ask here I am prompting Gemini to add a foru recommendation section on the homepage along with an image of the future state to show the improved design almost immediately Gemini code assist starts by reasoning about the code changes that it needs to make and has insights and experience teammate would have for example because we asked Gemini Cod assist to change the recommendation service it was able to find the recommendation function and extract out the exact details needed to make the call to the recommendation service it highlights the files needing to be changed and reveals the reasoning behind its recommendations using our own codebase for context Gemini code assist doesn't just suggest code edits it provides clear recommendations and makes sure that all of these recommendations are aligned with symbol Outfitter security and compliance requirements in code assist we've also added an option to apply the edit which keeps me as the developer in the driver's seat so let's take a look at source code changes that Gemini Cod assist has made in our code base it looks like we have multiple edits across two files handlers. go and also home.html Gemini codacy even applied these changes to the full repository and to put this in context no pun intended it would have taken me over 70 hours non-stop to even just read through all of these files and just like I would with any code change my next step is to check the workout by testing out the modified app locally so let's try it and there we go the for you recommendation section is exactly what our marketing team was asking for and as a new developer I was able to make a meaningful contribution on my very first day so let's recap behind the scenes Gemini has analyzed my entire codebase in gitlab it's implemented a new feature and has ensured that all of the code generated is compatible with my company's standards and requirements we are so excited for you to try out Gemini code assist a while back openi released code interpreter it's now called Advanced data analytics I believe and when I started messing around with it I was pretty blown away I did not think it was going to be as good as it was I could upload a massive Excel spreadsheet with thousands of customer orders and then just ask questions about it I would talking with Chad GPT just chatting with it asking questions like what is the lifetime value of all my customers what is the lifetime value if we separate them into different segments and very very quickly code interpreter would answer back with those details if I could formulate the question in English or whatever language in natural language it would create the code it would run the code it would parse the Excel spreadsheet and it would figure out the answer understanding data all sudden became much much simpler much easier if you knew what questions to ask the AI would do the hard tedious boring work of combing through that data and figuring out the answer for you A year later less than a year later here are Google's data agents we're thrilled to be helping Walmart use data agents to serve their customers we're helping many customers build data agents for example bear is harnessing big query in vertex AI to develop digital Medical Solutions and drugs more efficiently as well as streamline the creation of regulatory documentation News Corp is using vertex AI to search data across 30,000 sources and up to over across 2.5 billion news articles updated daily all right let's explore with our data agent what you're seeing here are realtime kpis the metrics that matter the most being continuously monitored from my Enterprise data in big query to surface anomalies and Trends let's dig in let's explore Cloud Sho sales over customer segments and territories in a heat map our data agent uses the looker semantic layer to effectively ground it AI in specific business data and data definitions that give us trusted answers and in this case it's right down to the territory and even customer segment how does demand correlate with other Core Business metrics okay our data agent again uses the looker semantic layer to find other business metrics that are similar and looks for anomalies in fact I've got three helpful insights I can actually dig into Supply app and demand levels over the next 3 months by clicking this first tile our data agent is now using big query machine learning to build a futur looking forecast and it looks like it's found a gap in supply versus demand and it's only growing I might need a substitute product to help me meet demand and we're going to upload an image of our Cloud shoe and with that our agent has all of the information it needs to do that search and it's found three Top matches and combine these images with structured data about pricing and delivery schedules well given I need a similar looking shoe and delivery times are important I think option one looks great one big concern that a lot of people have is the growing risk of cyber security attacks in a previous video we covered the xcia director I believe giving a talk in front of Congress on C-Span talking about how our cyber security has to get a whole lot better very very fast because of this Avalanche of potential cyber security attacks that are coming not only will AI empower the ability of hackers everywhere to make these attacks but also a lot of the kind of random harmless data that we generate online that we share online in the past it wasn't a big deal because it was just little pieces uh here and there it couldn't really harm us but with ai's ability to take in large quantities of data and make meaningful insights there's almost this new kind of attack Vector that we have where somebody that's just Gathering up like hoovering up all the data that's online could potentially find damaging ways of using it by you know using AI by using neural Nets to process that data in this next segment Google talks about security agents actual AI agents that work on your behalf 24/7 trying to spot cyber security attacks and also helping you defend against them the demand for agents like this or cyber security systems like this will likely rapidly spike in the upcoming decade now let's talk about security agents with security and generative AI we focused in three important areas you know security agents will help each of you in every stage of the security life cycle from prevention detection and response let's start with threat intelligence we're able to take the experience in intelligence We Gather from protecting Google's own Services combine it with mandan's leading Frontline Insight from their work in incident response to show you emerging threats the severity and risk factors committing to train over 200,000 professionals on our gen products we're at a pivotal moment in our industry together we're creating a new era of generative AI agents built on a new truly open platform form for AI and we're Reinventing infrastructure to support it m West Roth and thank you for watching

Channel: Wes Roth

Views: 118,747

Keywords:

Id: -fW0v2aHoeQ

Length: 36min 7sec (2167 seconds)

Published: Wed Apr 10 2024