Build Fast and Smooth Web Apps from Feature Phone to Desktop (Google I/O ’19)

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
MARIKO KOSAKA: All right, good morning. Thank you for showing up at 8:30. Hi, I'm Mariko. I'm from web developer ecosystem team within Chrome. Let's talk about building fast and smooth web apps. So it's safe to assume, I think, that all of us use the internet using computers, touch device, or your phones every day. But what if I tell you that millions more people are accessing the internet or coming to access the internet using device like this-- a feature phone? So for those of you who remember the days of feature phone, you may remember typing the text on those keys. I actually have it here. I grew up in Japan where the first popular mobile web network was built. And using feature phones to access the websites and doing all of the emails and texts was like, my part of high school and college years. So it feels really nostalgic to me. But also, I went into the industry after that. So I remember the days building a website specifically for feature phones. In Japanese we called it garakei saito. It's really hard. You needed to use a subset of HTML or sometimes, completely different markup language. Yeah, so bad memories a little bit. But don't worry, these phones are new phones that are coming out to the market. It's developed right now, sold right now all over the world. And it's not just me telling you that these phones are popular. According to Counterpoint Research, 370 million smart feature phones are expected to be sold between this year and 2021. So what is that smart feature phone? This is a new feature phone. So smart feature phone means it has a new OS, sometimes such as KaiOS. There is a few other OS, too. KaiOS is awesome because it's all web based so you can build an app using web technologies. It has a modernish web browser. It's a little bit of versions behind. But it supports HTML, CS, and JavaScript. For those who developed the website for feature phones in Japan, it's great news. And then it also comes with app ecosystem, which means you can download Google Maps, YouTube, games on your phone, just like you would do with your smartphone. So software-wise, it feels like a smartphone, but it comes in the size of a hardware. So the screens are usually small. I think most of the KaiOS devices are QVGA screen size. Navigation wise, it does have mouse shows up on the screen, but you have to navigate it with your D-pad. And if you want to input anything, and text on the field, you have to use the number key and the T9 key input to do that. So that part is a feature phone. So what does it look like to browse the web today using these new feature phones? Here's a website you may have visited once or twice before, Google.com. Loads great, it looks great. But we all agree that Google.com, the top page, is just a bunch of links and one input field. So that's not impressive. Let's see how it's like to load web apps, right? So here's Squoosh It's a web apps using WebAssembly, you can do image compilations. It's a PWA. We build it last year for CDS. How does it perform? Not great. The CSS layout is completely off. So the thing is when we built Squoosh, our team built it, we wanted Squoosh to be the best in class web app. So the first load is only 16 kilobytes. It loads really fast, even on this feature phone. But we didn't need it quite tested on the screen size this small. We tested it on the desktop, tablet, and phones, but we never thought of having somebody accessing the site using these phones. One of the depressing layout of the website is that I/O website. Not great, not great. It's so frustrating to navigate I/O website using it. It's almost unusable. But not everything on the web is bad. My favorite is Twitter. So Twitter and mobile Twitter.com-- when you access that on the feature phone, you get the same exact same experience. You can tweet, you read tweets, you can load a video, you can search for GIF, you can attach an image. Everything that's provided in the Twitter website-- available on the smart feature phone. That brings me to the project that I am going to discuss today. So as I briefly mentioned I am part of web developer ecosystem team in Chrome. And within, that I'm in the small subset of the team that tried to build the old world of web applications that we can share, so that we can share our learnings by building something real right. So our first project was Squoosh. It's an image compression application that is completely built within a browser. We used [INAUDIBLE] to provide a newer file format, like WebP, for browsers that don't support WebP. And you can select all of the knobs and a bunch of settings to see how much better the compression can be. And you can download it and upload it to your blog. So that's the idea. And then after we built this last November, we got to discussing, what should be the next project? And we decided, let's just build a game. We wanted to build a game because whenever we asked web developers, what is the web really good at, everybody says it's a document. And we wanted to say completely opposite of document to see if web can handle that. So games seemed like a good topic. And developing a game came with a bunch of problems that we face as a web developer every day, such as like how do we handle a lot of inputs coming from all of these UI? Or can we really provide a graphics heavy application on the web? And on top of that, because we were hearing that the feature phones are getting popular, we decided that our app this year is going to support everything from feature phone to desktop and everything in between. So I would like to explain what we built. First and then we're going to get into how we build it. Introducing Proxx. Proxx is the gave a proximity inspired by the legendary game of Minesweeper. Game is situated in space, and your job is to find black hole. You can play Proxx on any kind of devices from desktop, to tablet, to D-Pad, even with screen readers. SPEAKER 1: 1, dimmed, button, column 9 of 16. Hidden, button, column 10 of 16. MARIKO KOSAKA: It is a PWA, a so you can download it and save it on your desktop or on your phone, and play the game wherever you want, even when you're flying. So that's the game we built. And you can access the app at proxx.app, that's the URL. So let's discuss how we build Proxx. So even before we go into the project, we decided on the baseline-- like, three of us got together-- Jake, Surma, and me-- and talked about, what is this app going to be. Three points. We decided that every device will get a same core experience, meaning we are not going to build three different apps for desktop, tablet, and feature phone. We decided that it has to be accessible both on the input device-- so like all of the mouse, keyboard, touch, and D-Pad to support it. And we said, why not make sure that the screen readers are accessible, too. Performance-wise, our team really likes to build a Performant web app, so we said it has to be really, really good performance. So we said our performance budget to be initial payload of less than 25 kilobytes. Time to interactive less than five seconds on slow 3G network. And animations should run 60 frames per second whenever you can. So with that understanding, three of us got to talking. So let me explain how this app is laid out. So the game started with a game logic file that Jake wrote on a long haul flight, because he wanted to play Minesweeper, but there was no internet on the flight and he wrote it. He's that kind of engineer. Game logic just contains the logic. There is no UI element to it. It's just like how big of the field is, where the mine is, and when certain points are clicked, how should it be revealed. That kind of thing. On top of that, we built a UI and state services. So UI, we use Preact And state service is a [INAUDIBLE] on top of game logic to communicate game logic and the UI service. We also wrote our own rendering pipeline, which we will get into it later for why. And we also have a little bit of utilities to glue it altogether. So simple-ish, so you can understand. All of this could learn in main thread, everything in one file. Well, it could be separated, but it could learn in main thread. However, from the get go, we knew we wanted to do graphics heavy design, we wanted to animate it. And we were like, not sure about this. So we decided to move a game logic and state service into a web worker. So web worker, those of you if you are not familiar, is a way to align your JavaScript of the main thread and separate thread. To communicate between worker and main thread, you use API code post message. And it's like not an enjoyable experience, keeping track of the messages, passing. It's a lot of work. Luckily, team member Surma, who is sitting there, wrote a library he called a Comlink. It's a [INAUDIBLE] or abstraction on top of post message to make using worker a lot more enjoyable. In fact, we improved Comlink by doing this project, Comlink v4. So we were working on this project using comlink and testing it on the feature phone. And realized that it's not only working great. And realized that the comlink has processing intensive work happening. So we fixed that, and Comlink v4 was released. So if you're interested in offloading all of the tasks to worker, you should definitely check out Comlink. UI-wise, we use Preact. We chose Preact because three of us use Preact on the previous project, and we liked it. And then also, it's still the smallest UI library out there that fits in our performance budget. App wise, it is a standard single page application, all of the custom elements rendered on one div, and then appended onto the body. But we knew we have an aggressive goal of initial payload less than 25 kilobytes. So we decided to do a kind of a strategy where we will build time, pre-render the first load, the first thing traction using a little hack using Puppeteer. So basically, we have app. And whenever we build our app, we learn Puppeteer. And Puppeteer will download-- oh by the way, Puppeteer is a way to control a headless Chrome from your script. So a Puppeteer will open up the headless-- Chrome, do the thing, download the Preact. Preact will just build a HTML, put it into the [INAUDIBLE].. And then we just grab whatever output that was in there and put it into index.html. And that's what gets uploaded to our static site hosting notify, and then that gets served to the user as a first payload. This is just showing you how easy it is to start with Puppeteer. You just start an instance, browser puppeteer.launch. And then create a page. Go to that link. Evaluate that link. And just put it into the HTML. And that's all there is to it. So basic architecture was two key points. We use walker to free up the main thread as much as possible, because we knew going in, we wanted to do graphic heavy thing. The Graphic stuff, main thread can only do. Like, worker cannot do the graphic style. So free that up for graphics. And then we also pre-render the build time for speedy loading of the initial bundle. So that leads me to talk about graphics. Perhaps the biggest performance choice we made was to have our own graphic pipeline. So our initial plan was to completely use the DOM. We were thinking, oh yeah, we can just have a table and then put a bunch of buttons in it. And we can use a CSS animation, like transform, opacity, that thing that lands on the GPU to do the animations. And that would be great, right? Well, turns out we think that we might hit a Chrome browser bug, when all of this was the one layer, and want to update just a single button, Chrome was painting an entire table, which is not a great for performance. One way to solve these problem if you see it is to put the buttons or some elements that you want to update in the separate layers using things like [INAUDIBLE] transform. But we have a lot of buttons on the game, because each of the game cell, it's going to be a layer. And that might solve a painting problem, but then it creates excessive amount of layers. And that hogs up the memory, so we are creating another problem. So this route is not good. We should go for another route. So we decided to do all of our graphics in Canvas. In fact, we have two Canvas on our screen. One is for background animation, and one is for great animation that is doing the game itself. So these are generated and rendered every frame, 60 frames per second, using the requestAnimationFrame. So if you're not familiar with requestAnimationFrame, browser-- requestAnimationFrame is a way to scale your script. At every tick, the browser refreshes the graphic. So you put some kind of JavaScript, in our case, the drawing call for the Canvas, and you put that in callback of the request animation frame. And next tick, that gets one at the next tick. And if you're doing animation stuff, you probably want to recursively call that, so that you put that task into every frame. And then this is how we update our animation. If you are curious about all the stuff that I was talking about-- painting, layers, composers, CSS, requestAnimationFrame, I wrote a four part blog series about inside look at a modern web browser that explains what happens when your code gets the browser, and how it's executed. So you should check that out. And so we have two Canvas now. And there's a few other things that we did for the graphics for performance. For example, the background animation, which we call it nebula animation, in fact, it's only a one fifth of screen sizes. So whatever device you have, we only create a Canvas size one fifth of your device and just stretch it out to full screen. We were lucky, because the design that came up was already a blurry image. So creating a small one and stretching it didn't make much, and saved us a lot of memory to do that. So for the grid animation, we basically do a sprite animation. And we generate these sprites on the client side. So we do not send any image data down the wire to users. We send JavaScript to generate the sprites, and then JavaScript creates the sprites. And once that's done, it's saved in indexed DB. This way, we can create the most optimal sprites for each of devices. So different devices have different device pixel ratio. Some of them have one, some of them have two, some of them three. So this way, any devices accessing our site, we don't need to create an image. They can create the image or the sprites on the client side. So that's graphics. Let's talk about accessibility, which is exciting. I worked on it, so I'm really excited. So as I explained, the game is now into Canvas and we can totally build the game just with Canvas. A lot of games do. Whenever you use a mouse or click on it, you just get the coordinate of the mouse, and then you write your own JavaScript to say, hey, did it hit the square underneath. And then you just redraw the animation. But we decided we are going to keep a DOM that's tables and buttons. Remember, that was doing a painting [INAUDIBLE],, so we just fixed that by putting it opacity zero. There's a reason why we kept the DOM version on top of Canvas. Because if we have an element, we can focus on it or we can attach event listeners. So when you are playing Proxx, what you are seeing on screen is a Canvas. But what your JavaScript is interacting with is invisible buttons and tables. And this way, we can tap into browsers' Native Accessibility features. So here is a screenshot of me playing a game with voiceover. It is written down what the voiceover said. It says, hidden, button, column 15 of 16. Hidden is a state of the button. And that's the only part that we manipulated from our JavaScript by adding a [INAUDIBLE] label. Everything from button to column 15 of 16 kind of like suggesting users where the locations are-- that just came out of the box by using the table. So when we start the new game, we generated a table. Cleared a table. And in theory, you should just add a row or grid. And the screen reader should be all taken care. But somehow, maybe because we are displaying the table opacity 0, the browser wasn't quite listing that as a grid. So when we cleared all of the rows and columns, we also needed to specify that this tr is a row and this td element is gridcell. And that solved this problem. But at the beginning, we were like, why is it not working? The documentation said that if we put the grid, it should work. So that was a fun challenge to do code for. And inside of each cell, we generate a button that a user clicks on it. Speaking of buttons, we use a accessibility technique called row tabindex when we create a button for each cell. First, top left corner of the cell gets tabindex of 0, which means it's tab accessible. When keyboard navigation user hits the tab, the focus is on that button. But then everything else, we set tabindex of minus 1, which means it cannot focus by tapping it. This way, the keyboard user doesn't have to tap 100 times to get to the end of the game so that they can get to the other menu button down at the bottom of the screen. And when the screen keyboard user accesses the game, they focus on one cell. And they switch to using a [INAUDIBLE] key. And [INAUDIBLE] key will emit which direction, how many times that the focus should move. So the focus method basically takes like, OK, call in focuses here. Just going to make it a non-tappable or non-focusable. And then the new button that will be focused is going to be focusable by putting the 0. And then just set the focus. Let's set the focus. And this is how we implemented [INAUDIBLE] tabindex. I did not know any of this until I did this project, embarrassingly. We got a lot of help from Rob Dodson, who is on our team, who does a lot of accessibility work. And he has a guide on web.dev for accessibility to all, which explains all of the techniques that I explained right now. So you should definitely check that out. And thank you very much, Rob. So this is the half way of the project, and we were feeling great. Our animations were running. Feature phones, game was working fine. But then we noticed-- I explained that the feature phone to desktop, everything in between. So we were testing iPad, Android Go phone, which is a low end side of the Android phone. And we were noticing that the game is crashing on Android Go phones quite a lot. We were like, but how? Like feature phones are the weakest powered phone. Android Go should be an upgrade. Why is it crashing? Well, turns out how many pixels are on the screen matters a lot. So on the feature phones' case, they may have a weaker chip, but they only need to care about a certain number of pixels to drive. On Android Go phone, which is a smartphone, with touch screen, with much bigger screen, they might have a slightly powerful hardware, but they have four times or more pixels to drive. And that makes them sad and just shuts down sometimes. So we decided that OK, at this point, we should just check if the hardware can support the animation. And if not, just render a static version. So basically, when the game loads, we do a little bit of a check. Say, can this hardware support animation. And if they can, we load WebGL animations. And if not, we support a Canvas series static graphics. Static graphic is something we needed to make a new way for accessibility, for those who prefer reduced motion option. So we just exposed that to a little lower grade hardware phones, too. Our approach to checking this might be a little naive, but this is just one check. Shader box, this class is just a class abstract on top of WebGL. And we check a high percentage vertexShader. And if that's possible, then we just learned animation. And if not, we go with a 2D animations-- sorry, 2D static rendering. We know it is a little naive, but we found that the devices that supports [INAUDIBLE] usually can handle animations. Of course, if users have a preference set for reduced motion, we check for the media query. And then that becomes a default, too. Let's talk about supporting different input devices, because I've been saying keyboard, touch screen, D-Pad. So game has two main functions-- click to open the cell. And click to flag the cell. When you are playing on the mouse, it is just regular click and right click. If you are doing that on the keyboard, then we assign a toggle button to the F key for the flag. So you click on it, and then you can go back and forth with the mode. And then navigation wise, you use the arrow key and enter. For the phones and tap, we went with just tap and toggle method. So you can see at the bottom, we have a toggle button. And every time a user wants to switch to the mode, you just click on the toggle. And you might be feeling, looking at this plain video of a phone, saying, I want to pinch zoom in to see if how many cells are left, right? Or want to zoom in, zoom out to see how our game is doing. And this is something we discussed while we were developing this, and we actually needed to discuss whether to do it or not. Because we had this goal of the app has to be performant, running smoothly, we had this debate about like, yeah, but if we support pinch zoom, then we lose native scroll. And that means scroll gets slow. And is that OK? So eventually, we just decided that for this app, we are going to go with native scroll over pinch zoom. So we're not supporting pinch zoom yet. Maybe hopefully. The thing about pinch zoom is that once you have the pinch zoom action, then you lose the native scroll. And you need to implement your own scroll physics, right? And that is not going to be comparable fast to the native scroll. And on a web platform, we don't have the way to tap into browsers' scroll physics. We would love to have them, and we would love to explore the possibility, but for now, native scroll only. I also wanted this interaction was a double tap to flag. I was like, why do I have to toggle the mode. Can I just double tap and flag it? And it was immediately shut down because of performance. If you implement a double tap, that means we all have to wait for a single tap to check, is it double tap, is it double tap? Which means we create a few milliseconds delay for the user interaction. So based on the baseline that we agreed that it has to be smooth, we said no double tap. Possibility, though, in the future-- hasn't implemented it-- long top, holding on the thing. We can do that performantly. This is just a question of a UX and design. We don't know how to notify the user that you hold it enough to flag it, now you can take it off. And you know, the black hole is not going to get revealed. So once we figure out the design, we might implement it. Which brings me to this phone that does not have any of it. I mean, it has a click. You can click the button, but like, there is right click, there's not touch screen. A toggle key for F. F doesn't exist, it's only number key. So what do we do? Yeah, we added a custom key navigation to the number key. So when you click on 5, the cell's focus goes up. When you hit on 0, goes down. And then when you hit on 8, it is a click action. And then when you hit a hash sign, the sharp sign, then that's the more the toggle. So another thing we found was that we need to show users where the focus are. It's very hard on these small screens to see which button are they about to click. The also have a mouse indicator, too. But sometimes, you can't really see like, is this mouse pointing to the one button or the other button. So we made sure that we highlight the focus and tell the user, this highlighted element will be open once you click on it. Another thing we added, with is my favorite, is a key shortcut guide. So if you access our game on a feature phone, then you will see these tiny icon indicating, you can click on the hash button. To start the game or you can click on the asterisk button to open the information. This is piece of UX that I took from 2000 mobile development in Japan. So Japan had a feature phone web network. And when you go to a long document site or something, they usually have a table of contents on top. And then the in-page link inside. And those are usually mapped to number keys. So you would have a number emoji right next to the table of contents, indicating oh, if you want to go to chapter 3 just, click on 3, and then you can just move down. So I took that UX and put it into this game. Another thing that's very important if you designing a website or web game for feature phone is to have a way for user to get away from that view. So I have a close button there. Whenever a user opens the settings model, which is quite long, because it also contains how to play the game, and scrolls down-- whenever they think, oh, OK, I got it, they can just press on the asterisk key, and then just close that model. If we didn't do this UX, and have the standard design that we have for the smartphone and desktop at all, which is a floating x button, this happens. In the middle of the page, users have to scroll up, up, up, up, up, hit the top. And finally, the mouse can move to that close button to close it. And this is really frustrating. So the element itself, the floating x button and the Close button at the bottom, is the same element. But depending on the device, if it's feature phone or not, I just change the CSS to put the location differently. So that's the feature phone designs. Let's talk about offline strategy. As I talked about, this game is a PWA. It uses service worker to cache all of the resources. So even if you're offline, you can play the game. And whenever you do offline, there's always the question of, how do we update the game if there's a new update is there. We might have seen these more saying that hey, update is available. Reload this up or dismiss this. And then use the older version. This is a directly took from the previous project that we did. But in this case, we did not want to block the users of wanting to play the game or playing the game right now, so we hid this logic inside of this page. This button to be exact. So whenever a user comes to the app and hits Start, and whenever there is a network, they make a call to hey, is there an update. And if there is an update, then it starts fitting down the new version. And then once that's done, it loads a new version of app. Skip the opening screen, because we already know the game settings for it. And launch it into game directly. So when a user sees this page, the game is already updated to new versions. And this is how we do the offline versioning. Lastly, I want to talk about resource loading. So after all of this WebGL, and feature phones, and all of that, our total packet became 100 kilobytes Zzipped. We feel quite good about this size. And out of 100 kilobytes Zzipped packet, 20 kilobytes is a first payload. So we hit the goal of under 25 kilobytes Zzipped first payload. Basically, this is just an index.html that gets sent when the first request goes in. Which means this index.html contains this page. I like actually, this page. So all of the animations and the opening title role that we handcrafted with CSS animations are lazy loaded. So this is the minimal set of features and buttons that users need to start interacting the first action. The first action could be starting the game. First action could be opening that information icon, clicking the information icon to open the settings, or clicking the full screen button to go into the full screen mode. So we even subset the font. So we looked at all of the glyphs that's used on this page, subset the font, and inline it into this index.html. So really, our index.html is the 20 kilobytes of data that kind of looks like this. Yeah, a lot of inlining. But once that gets to the users and users start interacting, then little by little, chunks are downloaded, lazy loaded, and then game fully interactive. For doing this, we used rollup. We really enjoyed using rollup. We even wrote our own plugins for things that rollup didn't really provide out of the box. But we even felt comfortable doing that and kind of mixing and matching it. Which was not the case on the previous project-- we used a different building process. And rollup worked really great for our set up using worker. So as I mentioned, our codes are separated into the worker and main thread. And comlink is a shared dependency to communicate each other. If you do this in webpack, then webpack creates two different chunks as a dependency and separated it. And that's just duplicating it. But rollup out of the box, just keep it as a one chunk. And then share that as a dependency for worker and the main thread. So this was out of the box, great fitting for our project. For module loading, because JavaScript modules are not supported in Web Workers, we use AMD. And Surma wrote a tiny plug-in called rollup-plubin-loadz0r, which is an AMD-like loader, but is really tiny. Specifically made for rollup output. So you might want to take that out, but that's part of our build process. And even doing that, tools cannot really help the fine tuning of shaving down the data. We needed to go in and look at our index HTML, and what gets loaded. And then see, why is our index.html getting bigger and bigger and bigger. So if you want to check what kind of refactoring we did, there is an epic PR up the Github called I made stuff smaller by Jake. And thing that he did or things he discovered was things like this. So our game screen has a element called the topbar that has the number of cells that open, the timer counter, all of that. But those are only for game time, right? Whenever the game is not running, like opening screen or win screen, it's only just a title banner. But when we are loading the index for HTML, which only need title banner, they also load the logic of timers and logic of the open count and everything. So we separate that element into topbar, the full on version, and topbar simple, and just loaded two differently. And [INAUDIBLE] some data. So this is great if somebody has time to go in and dig through, and every now and then check if we are doing great. But we try to keep reminding ourself that every pull request that we make be conscious about the size. So every pull request we make on our repo, we run a little script called Travis size report on Travis CI to just check what changed. It's just this file name changed or this file size changed. And you know, this screenshot isn't particularly interesting, nothing really changed. But sometimes, you find unexpected change, like, why did this false name change, or why is file was suddenly this big. So this was a good reminder. So that's the process we took. I would like to end with a three learnings that we had. I definitely think that having a set baseline for the project was great. We started the project with a set understanding of what's important to us for this project and how we make decisions. And that got us showing up to the stand up saying, hey, I want to implement double tap. And immediately, Jake says, no, you can't do it, it's not performant. And I'm like, I wouldn't be offended or not feel like defending, because I'll just go, oh, that's right, performance was the important thing for this project. Another thing. We think the worker is crucial for learning a smooth application. We need to learn JavaScript off of the main thread as much as possible. I don't think we could make this game possible on the feature phone if we didn't do the worker. Lastly, if you are feeling like, I'm not a game developer, I'm not going to do the game on the web, one thing you can take away from this talk is just study what's on the first interaction for your website or your web applications, and just remove everything you don't need. That makes your first load data small. And then users can get to your service quicker. So if you want to check out the app, here's the link for the game. All of our source code are sourced on the GitHub, so you should check that out. Future requests or bug fixes are very much welcome. And if have any questions or want to play the game on the big touch screen, all of us will be at the Sandbox tent A after this. Thank you very much.
Info
Channel: Google Chrome Developers
Views: 22,475
Rating: 4.901515 out of 5
Keywords: type: Conference Talk (Full production);, pr_pr: Google I/O, purpose: Educate
Id: w8P5HLxcIO4
Channel Id: undefined
Length: 39min 5sec (2345 seconds)
Published: Thu May 09 2019
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.