Google WebRTC project update & Stadia review

all right yeah my name is how clio - I'm a product manager here at Google and I'm Justin Uberti and I'm a distinguished engineer at Google so yeah we wanted to go over a few updates and what happened from Google Sites in the vapor HC project so in 2019 we we finally got to to wrap-up web RTC 1.0 but we also moved the platform forward took some steps towards a and V and are looking towards new use cases so putting the last bits of vapor tc1 o0 in place took a little bit more time than we expected but we got from this but we got from this like this napkin design like almost 10 years ago - to actually something that's robust and a fantastic ecosystem and we there were a few bits that we had to get in place like perhaps most notably getting switching towards a standardised SDP format for all the browsers but Safari has done that and chrome switched early this year there are now 97% of all the connections that make use of these formats if you still opt out and we if you're still on the plan B in the plan B section you really have to switch it's gonna be deprecated but it's working pretty well similarly we moved away from the old statistics API which included a lot of books that something this or that and this has been cleaned up standardized or deprecated and now we only have one API to to work with to get your quality metrics of a party C chrome implements more than half of the of the stats that are specked out and lastly mdns there's been a session about it from excellent session from Philippine gift before very important to make sure that we we don't offer the ability for fingerprinting through web RTC I mean is launching Chrome we seen I've seen that it has been very effective in eliminating this problem and we're working on with Enterprise admins to resolve like complexities where it doesn't work so yeah of course a big thanks to everyone worked with this like a lot of you in this room in this audience today contributed documentation testing experimentation like committing to the platform before we were there so like a big thanks of course but is more of course before we were at 1.0 we already moved forward together that's how it works on to the new things and I just want to know I mention a few few few things that we did from Google sizes are quite interesting so most of every applications they either use h.264 or vp8 and of course if you one is on the horizon promising a lot of benefits and goodies but not there yet and a great intermediate step is vp9 vp9 is available in Chrome and we configure that in a configuration that we call k SVC where which is a combination of simulcast and SVC or the keyframes they reference each other and using this configuration you save almost 40% in aggregate bandwidth also vp9 volve vp9 is more CPU costly for encoding it's actually cheaper to decode a vp8 so if you have like in a typical typical conference scenario if one stream that you encode and decode many like many terminals around then overall the CPU increases like almost nothing another related improvement for conferencing is that we made tap capture a lot better and easier so there's an improvement in the UX that makes it easier to share content from taps from web browsers and provide a good experience the benefit of sharing taps instead of sharing your whole desktop and conferencing is that it gives a better privacy for the end user we don't see notifications popping up and it's more efficient because the browser can shortcut some parts from painting to capturing content a lost highlight on mention is that we added multi-channel opus support so multi-channel was already existed but it wasn't wired up entirely in the party see now you can use it both for encoding and decoding and for playback in surround in in chrome so you can really provide this cinematic experience just from straights and live a browser we also made some baby steps towards envy we hurt represent a lot today how we could possibly use this to shape our custom components do things that are not possible in in the web part C stack today like special audio gaze correction bringing develop your own codecs it made some progress there so pep assembly has gotten more capabilities same D extensions have been specs and will be implemented the the chrome team started with implementing web transport and we drafted a proposal for web codecs how you can interface both hardware codecs and develop your own custom codecs and this all is gonna be tied together believe it streams we proposed a way to interact with streams in a peer connection today so we had some good progress on what the platform is going to look at in the future and while we extend web RTC with new capabilities we also pushed things in terms of performance and reliability as Estonia from from our team mentioned earlier today the chrome process has chrome audio process alone as launched in Windows and Mac and basically resolves any stability problems in with audio instead of like taking a crash in your whole browser if something in related to audio crashes it is the audio process seamlessly restarts and it removes a lot of the junk in audio itself and news from Mozilla he mentioned before that I made audio multiple audio streams three times more efficient well we've worked on latency in Chrome so the part from like receiving a video frame which is network socket to displays displaying it on your monitor we made a three times as fast and a lot more stable than it used to be it's vp9 as you can see in this before and after growth so with vapor to see like adding these new capabilities like pushing the limits on performance and latency it opens a lot of for new use cases that we didn't envision when the party she started from like streaming video vr with spatial audio doing for 4k and better thing devices remote operation of of cars and building low latency interactive services these are new use cases far beyond what we ever imagined in the beginning and Justin is gonna tell you something super exciting that Google has built with it yes so I think one of the great things we look at a platform to understand you know how successful is it is first of all was the able to realize the goals that it initially has set out was able to basically address the the problems that were known at the time good platform should do that but great platforms were able to then address problems that people didn't even know they needed to solve at the time and as hype just talked about there's a number of new things that are coming down in WebRTC you know we're gonna hear about phantom auto and kind of operating vehicles and stuff that like when we are thinking back in 2009 2010 and we want to just get a video call working in the browser and the fact you can now drive a vehicle like it's incredible and so some of the other cases they're coming out in terms of low latency interactive services is Google's new cloud gaming service stadia and so these services all have the same sort of needs high performance ultra-low latency in the build on top of all the work that's kind of gone into the WebRTC platform over the past few years and so we've been working really closely with the CD team on this and I'll get into the details on the some things that we're doing so cloud gaming have some really unique and stringent demands first we need to have ultra-low latency second we need to provide a really smooth consistent player experience and lastly we need to give the developer the tools they need to realize really realize their artistic vision when it's in this cloud gaming environment let's let's talk about in latency first so why why does lazy matter well pretty much this like when you take an action in the game you expect that to happen immediately so we can't have anything like this happening in stadia now how are we actually gonna do this well here's the kernel like state of the universe today we look at video streaming you know they're usually you've got at least seconds if not tens of seconds of delay you know for existing sort of streaming services and now some smart people I've started using WebRTC for video streaming and they're able to get into that sub-second delay and doing this for live streaming they're able to do this for auctions you know for things like trivia apps even and those are great but for cloud gaming we really really need the drive latency down even further and push the boundaries of WebRTC so you know let's talk about low where are the lanes he come from in typical streaming services well what do you got essentially is like you've got the video streamers who are pushing out 5 to 10 second chunks of encoded video to see the ends through a file copy operation who are then allowing those chunks to be downloaded by browsers over HTTP TCP and then those things are then buffered in a pretty large client buffer and while this guarantee is a easy to deploy and scale service in ensure it's fairly smooth playback it comes at a cost and the cost is that you've got pretty significant several seconds multiple seconds of latency that's not going to work for cloud gaming so why well first of all service interactive not passive can't you know push a button it can't take effect in five seconds so we need to basically eliminate buffers everywhere in the server in the network and on the client and we also need to make sure we avoid packet losses or things that require retransmission because those will cause the same sort of visual stutters they were trying to avoid in the first place now here's a diagram of how the actual gaming service looks and we're at a webpart to see conference so i'm guessing to many of you this all looks pretty familiar you basically what we've got is a WebRTC call and for this call the difference is that one side of it rather thing a person is actually a video game running on a server now since we're using WebRTC this removes a lot of the typical Milian's that you would expect because weber sees already tuned for low latency and then the latency that you know that because they ruined the buffers you know we don't have that same sort of latencies you would have from like the typical HTTP streaming service now there's some people out there gonna say but Justin you know you've got the speed of light to contend with how you ever going to deliver this console type game experience when you're having an extreme this from a data center well it turns out even with the console there is latency there's latency in the television or monitor there's latency in the USB sort of polling system there's millions to even in how like data flows through HDMI so your brain even hasn't latency in it you know the human central nervous system typically as a higher latency is inherent then your round-trip time so this notion of like zero latency is a fiction and really what we're trying to aim for is how do we eliminate any perceptible latency well what does that actually mean so stevie team studied this and it turns out that it really really differs based on the type of game that you're trying to go after for RTS games like Starcraft you can have a fair amount of latency and the game is totally playable for third person avatar games sports games like fifa you know other sort of like fantasy games like Diablo you can have hundreds of milliseconds of latency games still very very enjoyable but the Stadio team wants to make sure they can satisfy you know deliver all types of games including the most latency sensitive games which is basically driving and first-person shooters in order to get to like making sure those games work magically in the service we do need to do a little more work so here's one of things that we came up with new congestion control algorithm you know for the stadia streamers and those of you who are familiar with WebRTC congestion control you know will recognize a bunch of these things already at the concepts that are here that typical Network protocols or network congestion control protocols basically work by filling the network until the network starts trapping packets because the buffers have filled up well folks that that happens it means we've already stored enough data in the buffer so we've added a whole bunch of delay so what we want to do with bbr or it was already being done in WebRTC is basically use the transit time as a signal that buffers along the path or filling up even the buffers start filling up we can say hey we're sending too fast and we need to back off so bbr is using this at scale already in quick and it turns out that bbr is actually a really good congestion control protocol that can also be used for stadia with you know some specialty when the other things that we're doing here of course is also providing a really really rich data center footprint we've got over a hundred network interconnects and nearly ten thousand ten thousand edge nodes and so this basically makes sure that we can deliver like the necessary you know counting for speed of light low round-trip time for a very very large number of users and then lastly we're building on a pretty large foundation of investments across Google we've got smart routing within our data centers we got machine learning for load balancing load prediction and then load balancing when a lot of players come in line all at the same time turns out the gaming happens between a very very narrow window rather being spread out you know diversity throughout the day and then there's the other stuff that we've been working on that's hype mentioned in the WebRTC stack to minimize the time for actual play out and even sort of capture that occurs like within within the browser so all these things working together allow us to hit what we're shooting for which is to basically deliver a cloud gaming experience that's as good or better than a local gaming experience to deliver 4k 60fps HD are at sub 150 milliseconds and n ladies so latency is not the only thing that we're trying to optimize for if that's all we cared about we just said like a super global resolution stream and just called it a we actually have this balance between low latency and also delivering there's really originally a rich experience that sort of match is what people have come to expect from modern gaming platforms so there's a bunch of things we can do to try to and like achieve that first of all it's like well let's figure out how we can actually use the codec that gives us the best sort of bandwidth efficiency and control over the system then encoder strategy how do we make sure that we're using those bits effectively to avoid any sort of visual encoding artifacts and then lastly make sure we deal with the fact that for the various client devices that are out there they each have their own decoders and implementation and possible idiosyncrasies so we want to make sure we give a lot of control aid to the developer and be within the service to deal with all of that now in order to figure out how we actually address this problem we first need to figure out what a success mean and so to figure that out we need a success metric and so the Stata team came up with this Maas metric which is pretty common across you know video calling and communications and through you know a bunch of subjective training people playing the service actually you know scoring it able to sort of understand how to train an objective model that basically says how are we doing how was the level of visual fidelity that we're actually able to deliver for a gaming session now surprisingly or unsurprisingly we found out that the actual satisfaction with the gaming playability really really corresponds directly to the actual video quality as described in the video mass model and so a lot of the work that went into is like how do we make sure we we make sure that video mas you know it is high at all times and so one thing that comes into is okay we see that higher resolution turns into higher perceived video quality turns into higher perceived playability so what codec can we get that gives us the highest resolution across different bandwidth and so we looked at a bunch of things h.264 has a bunch of hardware support but vp9 has even much much better over 40% better as I've said you know bandwidth efficiency it turns out that allows us to get the 1080p and a much much larger range of bit rates than h.264 and so vp9 is what that is the the codec that CD has chosen to use for its initial rollout we also looked at okay the objective stats don't tell us the whole story there's also this sort of question of well where do we want actually you know display you know high resolution high frame rate and where we're going to do something slightly different and this graph shows that players who had you know when bandwidth was plentiful preferred 60fps but when bandwidth was restricted were happier to have a consistent 30 FPS than intermittent 60fps and so basically want to takeaways from this was that optimizing for smoothness of experience above almost everything else turned into a higher playability score and a higher satisfaction from the user so it turns out that there are a lot of different factors that all sort of work together here in terms of what actually turns into playability we went through these same sort of tests to understand what was the sort of right set of parameters for all these different variables to figure out how do we deliver the best overall streaming play thought experience um one of the things that also came out of this was the notion that the actual game developer should have input into how the game is actually streamed and encoded and one of the things that you know is interesting is that there are some perceptual enhancements in terms of the encoding that don't turn into differences on the PS and our scores but as we turn into a subjectively better for the MA scores improvement and allowing the game developer to indicate which are maybe complex areas of the game or special scene transitions or other things that are hints essentially to the encoder allow us to have a very very large increase in the perceived playability score and then finally these are the hints to the encoder also help ensure a smooth player experience and also make sure that the actual encoder is not stressed or overloaded or having to quantize heavily and then maintaining this consistent bitrate and consistent video quality turns to a better player experience and also turns into and a war even number of bytes on the wire which turns into less packet loss um better quality so all these little hands to the encoder work together to provide a much better streaming experience so lastly we want to make sure that developers have the tools to do this type of fine control and some of these things we might find are actually interesting to not just stadia but other web RTC applications so here's one thing we've got basically our dev portal you can go in there and run any game any build of it and control for both ingress and egress packet loss delay delay variation even like you can go and record video compare what you think is being sent versus what was being received see what the differences are see I where you might need to tune the game differently to make sure it works well on different types of networks or in different regions we've also got this video diff tool where you can go in and see might be specific areas within the game that have a lot of complexity where you want to have maybe some region of interest or others or perceptual improvements and basically where does the game interact poorly with the encoder it could benefit from additional hints or custom settings finally here's an interesting one what we call the smoothness view and so this is an assessment if the overall sort of smoothness of the stream over the past 10 seconds and when you're playing the game and you kind of realize that something that just felt kind of off you can then pop this up and see like was that a problem with the game was a problem with the stream or is that just a problem - Breen like you know these are all things that you know you can then use to say like if you're trying to have this really perfectly smooth experience why would did that actually happen and I think this is something they probably could be applicable and many other sort of fields of web RTC today so these tools are all available you go to stay da dev sign up to be a developer you get a Chrome extension you plug it into the browser you can check all this stuff out but one thing I just want to just take a brief minute to talk about is um you know it's really amazing as I said in the beginning to see this all come to life and really benefit from all the work that you know we and and really everyone here in this has put into the WebRTC ecosystem you know studio depends on WebRTC and it gets a lot of value because it can deliver this super high experience quality experience at any WebRTC browser and that also means it has access to literally billions of deployed stadia ready endpoints and as I've talked about the WebRTC platform continues to improve do things like webassembly web transport and this means that you know Studios gonna have an even more rich set of tools in the future to really control and deliver an amazing experience so we're all super excited about where this is going you all be able to get your hands on it next Tuesday starting at 9:00 a.m. and it sort of gets you all sort of appropriately pumped and ready for it [Music] [Music] [Music] [Music] [Music] that's it folks cloud gave me from Google built on web RTC

Channel: Kranky Geek

Views: 9,762

Keywords: webrtc, krankygeek, stadia

Id: avtlQeaxd_I

Length: 24min 12sec (1452 seconds)

Published: Fri Nov 15 2019