[MUSIC PLAYING] ERIC BIDELMAN: My
name is Eric Bidelman. I'm a web developer,
but basically, I'm an engineer that works
in the Chrome team, but I also work in
developer relations, which means I help you
guys, web developers, build kind of the latest,
greatest web experiences, adopt new APIs, and
lately I've been focused on testing automation,
headless Chrome, and Puppeteer. I think it's a really
exciting space, the fact that we have headlesss
Chrome now, Puppeteer. So feel free to hit
me up with questions. It's @ebidel on
Twitter, if you want to talk to me after
the presentation. So it's really important for
me to get this out of the way. This talk is not about testing. I think you should
all test your apps. Don't get me wrong,
it's very important. You can certainly
use headless Chrome to do end-to-end testing, smoke
tests, UI tests, whatever. But I want to stick to
the other side of things, kind of the automation
side of things. So this is something that I
realized a couple weeks ago-- headless Chrome can really be
a front end for your web app front end. And this was kind of
an a-ha moment for me, kind of a double rainbow moment. Once I started working
with headless Chrome, once I started to bake
it into my, kind of, workflow as a developer, it
actually makes my life easier. I can automate things, I can
put headless Chrome on a server and do really interesting
things with that. We'll talk a little
bit how to do that. Some really cool and powerful
things you can do with headless Chrome. So the agenda for
today is we're going to talk about what
headless Chrome is, get that nomenclature
out of the way. We'll introduce Puppeteer, which
is the node library that we built to work with headless
Chrome, and along the way, we'll just see 10 interesting
use case-driven demos that I've built that I kind of
want to share with you guys. And we'll talk about Puppeteer's
API to do some of those things. So that's today's agenda. This is something that I'm going
to refer to a couple of times throughout today's presentation. This is the pyramid
of Puppeteer. It's basically the architecture
of where all of these things kind of fit together. So at the very bottom
is headless Chrome. This is just the browser. And so normally, when
you click on Chrome, there's this window
that launches. The user can input a
URL, there's a UI menu. The page is interactive, so
you can type in the page, you can click around. You can even open the
DevTools and tweak styles, and kind of modify
the page in real time. And the DevTools, of course,
has many, many more features. But with headless Chrome,
there's none of that. So something is
happening, right? Chrome is running, you can
see it in the taskbar there, but there's literally no UI. Headless Chrome is Chrome
without Chrome, so to speak. It's Chrome without UI. So if there's nothing
to interact with, how is this thing useful to us? We'll talk about that. If you want to launch
Chrome in Headless mode, it's just a one-line
command line flag. So --headless launches
Chrome without a UI. Simple. But by itself, this
is not too useful. We need to combine this with
something else, which is the remote debugging port flag. And you can pass any port
number you want here. But once you combine
these two flags, this is going to open Chrome
kind of in a special mode. It's going to open this
remote debugging port, and then we can tap in to
the DevTools programmatically using this remote
debugging port. And so that's where
things get really awesome. So what does headless Chrome
actually unlock for us? One of the most exciting
things, I think, is the ability to kind of test
these latest and greatest web platform features. Things like ES6 modules and
service worker and streams-- all this goodness that's
coming to the web, we can finally write
apps and test those apps because we have this
up-to-date rendering engine that's kind of following
us as the web evolves. The other thing
that it unlocks is all of this really
awesome functionality that you guys are used to
using in the DevTools, things like network throttling and
device emulation and code coverage-- all these really,
really powerful features. We can now tap into that
stuff programmatically and write automation scripts and
test these things and leverage some of that work that's
been done for us in the past. So headless Chrome has a
lot of interesting things that you can do. And I do encourage you to
check out this article. This is about a year
old at this point, but it's a really good article
that I wrote a little bit ago, and it still is relevant. So you can do really interesting
things with headless Chrome without ever having to write
any code, which is kind of cool. So you can launch it
from the command line, you can take screenshots
from the command line, you can print to a PDF, just
create a PDF of the page, and do some other
interesting things so. Do check that out if you want
to know just more about headless Chrome. So maybe I've sold you. Headless Chrome, it's a thing,
headless browser, a thing. So what can you actually
do with this stuff? Well, let's go back to
the pyramid of Puppeteer. So we've got the browser
at the lowest level, right? All the web platform features,
all the E6 stuff, all that is at the bottom level. On top of that is the
Chrome DevTools protocol. So there's a huge,
kind of, layer here that the Chrome
DevTools itself uses to communicate with your
page and change the page. It's a whole API surface
that we can tap into. These are kind of like the
yin and yang for each other, so I actually rank these
as among the greatest duos of all time. Headless Chrome
and DevTools-- you can really take an awesome
adventure with these. And, of course, you got Han
and Chewie, you got PB and J, you got Sonic and Tails. But headless Chrome
and DevTools-- awesome, awesome duo. So the DevTools itself,
it's pretty straightforward. It is complex. There's a lot you
can do with it. But it's basically just a
JSON-based WebSocket API. So if you notice,
I open a WebSocket to localhost 9222, which is that
remote debugging port that you saw in the previous
couple slides, and then you can just
basically do message passing. In this example
here, I'm basically getting the page's title. I'm just evaluating this
document.title expression inside of the page using the
runtime evaluate DevTools flag or method. And you can actually
see this traffic. So DevTools itself,
again, uses this protocol. So if you open the protocol
monitor panel in the DevTools, you can see these requests and
these responses kind of fly by. So any time you tweak a style
or do something in the DevTools, you actually see
the traffic for it. So you can kind of learn the API
as you see this stuff happen. So back to the
period of Puppeteer, we've got the browser, we've
got DevTools protocol-- all this cool stuff
we're going to tap into. And on top of that is
where Puppeteer comes in. So Puppeteer is
a library that we launched last year
right around the time headless Chrome came
out in Chrome 59. You can get it off of NPM. And the reason we
created it was there wasn't a lot of good
options for working with headless Chrome
at that point in time, and we wanted to
sort of highlight the DevTools protocol. Make sure people know
how to use the protocol, kind of make it high-level
API for some of the really powerful things you can do. So we actually use a
lot of modern features. You're going to see a
lot of async and await and promises in my
code samples today, and that's because
of this async nature of everything happening
with WebSockets and Node talking to Chrome. And all that stuff
is asynchronous, so promises lend themselves
very nicely to that. But you can use Node
6, so if you're not in a later version of Node,
you can totally use Puppeteer. Don't have to transpile
or anything like that. We wanted to create a
zero-configuration kind of set up for you. So when you pull down
Puppeteer from NPM, we actually download
Chromium with Puppeteer. That's because it's kind of
hard to launch Chrome and find it, install it, on,
like, a CI system. There's just a lot
of issues sometimes. So we wanted to make it easy. Just bundle a version
of Chrome that's guaranteed to work with
the version of Puppeteer that you guys install. High-level APIs-- we'll see
a bunch of examples of that and create a canonical reference
for the DevTools protocol. And so that's why we
created Puppeteer. So let's look at a
little bit of code, a little bit of Puppeteer code. One of the most
common things people do is just take a
screenshot of a web page. So in order to do that,
we'll call Puppeteer launch. And this is a promise. It's going to return a browser
instance that we can then interact with headless Chrome. So we've got headless
Chrome launched, got a browser instance, and then
we'll just create a new page. And so this is going to open
just a new tab in Chrome. You're not going to see it
because it's headless Chrome, but it's opening about blank. And once that promise resolves,
we can navigate to the URL that we want to take
a screenshot of. Just call page.goto. That's going to actually wait
for the page's load event to fire before it resolves. And then we can just
use Puppeteer's API to take a screenshot. And this has a bunch of options. You can take a
full-page screenshot or actually screenshot a
portion of the page, or even a DOM element. But you can see it's pretty
easy, it's very high-level. You don't need to deal with
the buffers or responses or anything like that, you
just pass the file you want to create and you
get a .png file. And last but not least, you
just close the browser out when your script is
done, clean up Chrome. That'll shut down Chrome. So all in all, it's, like,
four or five lines of code to do all this stuff-- launch headless Chrome, find
it on various platforms, open a new page, navigate to a
page, wait for its load event, take a screenshot,
close the browser. So this is what I mean
by the high-level APIs-- very task-driven things,
but it's very easy to accomplish in
Puppeteer's APIs. So pro tip-- there's
headless Chrome, and this is what
we use by default when you call Puppeteer launch. You're not going to see
an actual browser window, but there's actually
headful Chrome. So headless Chrome,
headful Chrome. And this is just normal Chrome. If you include this flag,
the headless false flag, this is going to
actually launch Chrome. You're going to see it,
and this is really handy if you're debugging
scripts and you have no idea what's going on
because you can't see anything. Throw this on. You can actually see Puppeteer
click around, navigate pages. It's actually kind of cool to
see this stuff in real time. All right, so we've got
headless Chrome at the bottom with all the web platform. We've got Chrome
DevTools protocol. Puppeteer is kind of
a small API on top of all this stuff below us. And, of course, at the top is
where our automation scripts come in. So we're already standing
on the shoulders of giants because all the stuff below
us has been iterated on for many years. So let's dive in. Let's see some cool stuff you
can do with headless Chrome. 10 awesome things you can
do with headless Chrome. The first is kind of neat. So you can actually pre-render
JavaScript applications. If you don't have a
good server-side render store for your
framework, your app, you can actually use headless
Chrome to do this for you. So I wanted to actually kind of
put my words where my mouth is and build an app and see if this
was actually a viable solution. So I built this devweb
firehose, I call it. It's basically a content
aggregator for my team. We bring in all the blogs,
all the sample code. Everything we do
it ends up here. And it's a real app. It's a client-side app. It's powered by ES
modules and some new stuff like Firestore
and Firebase Auth. It's got a JSON API so
you can query the data. Its back end is
written in Node and it runs Puppeteer and Headless
Chrome to actually do some server-side rendering
to get a good, good first meaningful paint. So this is the app. Let's see how we built it. So we can dive into
the index pages, the main page of the app. And it's a basic
client-side app. It's got a container
that gets filled with a list of JSON posts. So I got a container here. I'm going to make
a fetch request, just get the list of JSON post,
and then call this magic render post method, which renders
the post into this container. And all that thing does, it
creates a template string, and it literally just
innerHTMLs the content. And so you could use the
DOM APIs or whatever. But that's basically
what this app does. So the goal is to take an
empty page when the page loads and turn it into
a static version. That's essentially
what pre-rendering a JavaScript app does. That's the end goal, and we
can do this with Puppeteer. So here's a little server-side
render method that does this. Basically, it'll take a URL-- the URL that you want a
pre-render-- and we'll do exactly what we did before. We'll launch headless Chrome,
we'll open a new page, we'll navigate to the URL
that we want to pre-render. And then this new thing
here, page content, is basically getting the
new snapshot of the page as rendered by headless Chrome. So headless Chrome is going
to process our JavaScript, it's going to
render those posts, it's going to get the
markup in the DOM, and we just grab that
using page content. And that's the serialized
kind of string of the DOM. And so that's the
thing we return in this method, the updated
version of the page. By itself that's useful,
but we can actually put this on a web server
and then server-side render this client-side app. So pre-rendering a dynamic app. Somebody will visit
our app, and what headless Chrome is going
to do on the server is go fetch the same page. It's going to actually
go off, run the page through Chrome in the cloud. Of course it's going to
see the page as written, it's going to render
all that JavaScript and stuff, process my CSS. And the thing we actually
return is that final result. So that's what the server
returns, is that static markup. So again, empty page to
static markup is the end goal. So the server itself
is really easy. We're just going to reuse
that server-side render method I just talked about. It's a little express server,
and I'm using ES modules here. That's why we have .mjs file
format and the experimental modules flag when you
run this web server. So any time somebody comes
to visit my home page, I basically just call that
server-side render method, just load up the index.html
file, the client-side app. We'll get the server
side rendered, the pre-rendered
version of that. It's going to go through
headless Chrome being run by the browser. And then we send the HTML
final response to the user, and that's the
server-side rendering using headless Chrome. So you're probably
wondering, is this fast? Is this actually
a viable solution? So I did do a little
bit of measuring of this because I was also
very curious of this. So if you slow your
connection down to be like a mobile device, slow
the CPU down and the network, you can see the side
by side comparison between the server-side
rendered version on the right and the client-side
version on the left. So server-side render version,
immediately you see the post. As soon as the page is there,
that markup is in the page, first contentful
paint is right there. The client-side version
takes a little longer. It's got to churn
through the JavaScript, it's got to render the post,
things are a little slower. So immediately you
see a performance win on slow mobile devices. And the numbers actually
speak for themselves. So web page test, 3G connection,
just hitting this app, you go from, like, 11-second
first contentful paint down to, like, a 2.3-second
first contentful paint. And that's how fast those posts
actually render in the DOM. So that's really exciting. Not only do we make
our app faster just by adding on headless
Chrome, but we also make it available to search
crawlers and engines that don't understand JavaScript. So we have, kind
of, two benefits. Now, in order to get
to those numbers, I actually did make a
few more optimizations that I want to go
through because I think they highlight some
of Puppeteer's really awesome APIs. So the first is you don't have
to wait for the entire page to load. The only thing we care about
is this list of posts, right? We only care about that markup
as headless Chrome renders it. So if we go back to that
server-side render method, we're launching
Chrome, we're actually waiting for all network
requests to be done. That's what that
networkidle0 is. We don't really care about,
like, our analytics library to load, we don't care
about images to load, or other wasteful things. We only care about when
is that markup available? So we can change the wait
until here to domcontentloaded. I just want to immediately
resolve this problem when my DOM has been loaded. And we can tack on one
more Puppeteer API, which is page.waitForSelector. And what this is
going to do is it's going to wait for this element
to be in the DOM and visible. So we're waiting, we're
kind of catering now this server-side render method
and catering it to the app, but we're actually speeding
up the pre-rendering process by doing that. So that's not waiting for
the entire page to load. Number two is to cache
pre-rendered results. This is kind of an obvious
one, and this speeds up things quite a bit. So same method as before, but
we'll just wrap it in a cache. So anytime somebody comes
in for the first time, we'll fire up headless Chrome,
we'll do the pre-rendering, and then store the
results in the cache. And any subsequent requests
just get served from that cache. It's in-memory,
so you would want to do something more
persistent here, but this just goes to show
you that it's very easy. You only pay that penalty
once for the pre-render. Number three is to
prevent rehydration. So what the heck is rehydration? So if we go back
to our main page, you have the container that gets
kind of populated by the JSON posts. And if you think about
what's happening here, the user is going to visit
this page in their browser, Chrome is going to
do its thing, it's going to render
this in the client, but headless Chrome is also
doing that on the server, so that's kind of wasteful. We're doing that twice. So the way I dealt with
this was basically just look for this element that
gets server-side rendered. I basically check and see if
that post container gets added to the DOM, and if it's
there at page load, I know that I've been
server-side rendered, and I don't have to go
through the hassle of fetching the posts and
rendering them again. So that's another
optimization you can do. Number four is to
abort non-DOM requests, and here's another
made-up term by myself. But what the heck
are non-DOM requests? So if you think about it, we
only care about this markup, and certain things don't
actually build the page. So JavaScript can build
a page with DOM APIs. Things like video tags
and image tags and CSS, they don't actually
construct markup. So we can actually
abort those requests and kill those requests
because they're wasteful to us. So Puppeteer and Chrome
have this awesome feature called network interception. We can turn this on using SUT
request interception to true. And what this is
going to do is it's going to give us the ability to
intercept all network requests before Chrome ever makes them. So we can listen for requests
events, and inside of this, I basically just
set up a whitelist. If you're one of these
requests, like scripts or XHRs or fetch events that
can generate markup, it will allow you to go through,
it will continue the request, but if you don't, if you're
a style sheet, for instance, it will just abort the request. So this is another
cool way on the fly that we're speeding up
the pre-rendering process. So if you want to know more
about pre-rendering, headless Chrome, Puppeteer, all that good
stuff that I just talked about, there is an article that I
wrote a couple of weeks back. It's got more optimizations,
more discussion in there. Please give me your
feedback because I think this is a really cool
approach, because I didn't have to change any of the code in
the app, I actually just, again, tacked on headless Chrome and
I got a lot of stuff for free. So I'm curious to know
your guys' thoughts. Number two awesome thing
you can do with Puppeteer and headless Chrome is actually
verify that lazy loading is paying off. Lazy loading is a good thing. You should all do it. But sometimes, you know,
I'll put in a bunch of effort into my apps, and I'll
wonder, is this actually making my app faster? All this work, is it paying off? So you can verify this now using
Puppeteer and the code coverage API. So we have a code coverage
API that gives you kind of a breakdown of
the CSS and JavaScript that your app uses. You can start cod coverage, do
a bunch of stuff on the page, navigate around, and
then stop code coverage. DevTools has a panel for this. You can check that out. But I wanted to go one
step further and see if we can kind of analyze the
page across the entire page load. So this script, basically we're
going to run it on a site, and over the course of page
load, on DOM content loaded, on page load, and eventually
when the entire page is loaded, it's going to give us the
print out of everything that's going on. So you can see the URL
itself, chromestatus.com. I'm using about 37% of my
JavaScript and my CSS resources at that point in time. But as the page
progresses, I'm using more and more of those files,
as you'd probably expect. Now, what the script
actually highlights is that I'm lazy loading things. So this second resource here,
these second set of bars, you can see it's
not utilized at all, and that's because this resource
is behind a user gesture. They have to click this
navigation element, and that's the thing that
actually dynamically loads this bundle. And so you can use
the script like this and combine the code
coverage API to determine is lazy loading paying off? Do A/B testing, do
some measurements, and use Puppeteer to
your advantage there. There's a cool NPM package
worth checking out. If you're familiar
with Istanbul, it generates these
amazing HTML reports. You can basically get
Puppeteer's code coverage and run this thing,
and basically get the same exact Istanbul HTML
output, which is really nice. So check that out. Number three is A/B testing,
and there's that word "testing" again, but this
is more of, like, live modifying your page
without having to change the code of your page. So I want to measure
if it's faster to inline styles versus just
having my style sheets be link style sheets. It's a common thing people do. Is it going to pay off
if I inline my styles? And normally what
you would do is you would basically ship two
different versions of your app and measure that. You would make code
changes to measure that. But with Puppeteer, we don't
have to make code changes, we can live change
the page on the fly. So we'll use network
interception again, but this time instead of
listening for network requests, we'll listen for the responses. So for any style
sheet response I get, I'm going to check
the resource type. I'm just going to stash the CSS
text, the content of the files inside of a map for later. We'll navigate to that URL that
we want to actually measure this on, just using page.goto. And we're using now
a new method, $$eval. So this is kind of
like a jQuery API where you can pass
it a CSS selector. In this case, I'm grabbing all
the style sheets on the page. And my callback is going to
get injected into the page. It's not run inside of
Node, but it's actually injected into the page. So in here, you can actually run
anything the browser supports-- DOM APIs, URL constructor,
web platform features. And what this code does is
it basically just replaces all the link tags
with a style tag, and it injects the CSS
content from the files inside of that style tag. So I'm just replacing
the style sheets with the equivalent
style tag on the fly. And that's actually
what gets served up. In this case, you can
run this on a server, you could do a script, do
a side by side comparison. And we haven't changed
the page to do this, we would just use
Puppeteer to live modify the requests that are made. So that's doing A/B testing. Number four is to
catch potential issues with the Google crawler. So a couple of weeks back, I
built that devweb firehose app, and I realized after I
pushed it to production and I hit the Render as Google
button on the Webmaster Tools, that my app actually
doesn't render correctly in Googlebot because it runs
our super old version of Chrome, Chrome 41. Chrome 41 doesn't have CSS
custom properties or all these cool, new features I was
using, so I was kind of hosed. What do I do? So I said, hey, could
we use Puppeteer to catch this or have
an early warning signal before shipping your app? And the answer is yes, you
can do this using Puppeteer. So normally, you
go in the DevTools, you hit the timeline recording,
you get this big, kind of, blob of information that you
can slice and dice the data. We can do that, as
well, programmatically using Puppeteer's tracing API. So I can start a trace and I can
stop a trace whenever I want. And basically, you get this
huge JSON file that you can then take action on. And one of the things that you
can do is actually pull out all the features used by a page--
all the JavaScript that's used, all the APIs that get used,
all the CSS that gets used-- and then you can
correlate that with can I use data for Chrome 41. So that's what this
script does it. You can run this
script on any URL and it'll tell you the
features that you're using that aren't
available in Chrome 41. So Chrome status uses--
what does it use? Web components, it
uses CSS contain, it uses link while pre-load. None of that stuff is available
in the Google Search Bot, so this is kind of a cool
early warning signal for you to determine if your app
might not render correctly in Google search. And so you can then
maybe load a polyfil where you didn't expect
to load one before, just by getting the list of
features used by the page. Number five create, custom PDFs. A lot of people like to create
PDFs of their web pages. I don't really understand
it, but a lot of people do. We have an API for it. So if you've joined us at
the web sandbox over here this year, you
can actually go up to the big Lighthouse, input
a URL, and what happens is Puppeteer spawns up
three different tools. It runs web page test,
it runs Lighthouse, and it runs page speed
insight, all at once. And then eventually
what happens is you get this overall
kind of report, this PDF of each one of those
results from each of the tools. And we're just
generating that PDF using headless Chrome and Puppeteer. So to do that,
it's pretty simple. We just create a new
page, and instead of navigating to
a page, we're just going to construct
one on the fly, just calling page.setcontent. We're kind of
building an HTML page just by giving it a string. We'll set a viewport because
we want the page to be big. We don't want it to
be a mobile size, so we'll use the viewport and
emulation APIs that DevTools has to create a big page. And then last but not least,
similar to screenshots, you can call page.pdf and
create a PDF of that page that you're visiting. So similar, you can
create a PDF to disk. This takes a bunch
of options, too. You can give it a header, a
footer, style the margins. Anything that the Chrome
PDF system can do you can kind of do on Puppeteer. So what's neat about this
is that you don't really need a JavaScript library
to create PDFs anymore, you can just use the tool
that's on your system, which is the browser. That's kind of nice. Number six, make
your browser talk. This is a fun one. So what this script
is going to do is it's going to read
a text file in Node, and it's going to open a
page that uses the web speech synthesis API to read
that text file back to us, so kind of a good example of
combining Node and Puppeteer with some of these newer
web platform features. We can take advantage
of both worlds. [VIDEO PLAYBACK] - Hi there, my
name is Puppeteer. I've clicked the speak
button on this page to start talking to you. I'm able to speak using the
browser's web speech synthesis API in the message injected
into the page from Node. TLDR, the rise of the
machines has begun. Bye for now. [END PLAYBACK] ERIC BIDELMAN: So we're
not quite there yet, but it's kind of a cool example. So a couple of notes about this. You actually saw a
window, in this case, so I'm launching headful
Chrome, and that's because audio is not
supported by headless Chrome. So I have to use that headless
false flag in this case. The other thing that I'm doing
is I'm using executable path. I'm actually opening
Chrome Canary and not the bundled version
of Chromium that gets installed with Puppeteer. And the reason for that
is because the web speech synthesis API in Chromium,
in the open source version of Chrome, doesn't
have that cool British accent. It's only available
inside of official builds, so I wanted the Jarvis accent. So we open this little, tiny
window using those command line flags, and then we'll
use a new API called evaluate on new document. What this is going
to do is it's going to run this JavaScript
before any of the other pages JavaScript runs. So this gets
injected in the page. And what I'm doing here
is I'm being silly. I'm just setting
a global variable, I'm creating a global variable
in the page called text to speech, and just
sending it to the content of that file that I read. So that's how the message
gets into the page. And then what I do is I read
the page, just the HTML file, and instead of
starting a web server, I just kind of navigate to
the data URL version of it, so I'm just kind of
on-the-fly opening the page. And then the last thing that I
do is I click that speak button using page.$. Again, kind of a jQuery
API where you give it a selector of an
element on the page, and then we just call
the click method. And that's actually what kicks
off this reading of the text. Number seven awesome thing
you can do with headless Chrome is to test
Chrome extensions. I don't know how people
tested or test their Chrome extensions, but
you can certainly test your Chrome
extensions using Puppeteer, which is kind of cool. So I'm going to show
you guys a real example. I'm going to run the Lighthouse
Chrome extension's real unit test. They decided to use Puppeteer
because they would ship code every once in a while and the
extension would just break, and we wanted to fix that. They wanted to
actually write a test. So what this is
going to do, it's going to use Puppeteer
to launch a tab. I'm just going to go
to Paul Irish's blog. And we're going to
actually use Puppeteer to start the Chrome extension. You can see it in the corner. It's actually been started,
and inside of the butter bars there at the
top, you can see Lighthouse is
debugging this page, and Chrome is being
automated by Puppeteer. So basically what's happening
is Lighthouse is just running. This is what Lighthouse
does normally. It reloads the page,
it gives you a report, and eventually, all the tests
pass, which is really cool. So I know that was a lot. That was very fast. How did we do that? How are they actually
testing their extension? So the important
bit here is that we have to use headful Chrome again
because headless Chrome doesn't support Chrome extensions,
so we're launching a full version of Chrome. And we can use these arguments
to pass in our extension directory to load Chrome with. So we'll load Chrome
with our extension. And then what the
Lighthouse team did was basically just grab the
background page that the Chrome extension creates, and
then they'll inject, they'll actually run one
of the background page's JavaScript methods. So we use extensionpage.evaluate
to evaluate this callback inside of the extension
itself, and the extension just happens to have
this method called run lighthouse in extension. That's actually what
kicks off actually running Lighthouse inside
the Chrome extension. So that's how they're able to
test their Chrome extension. Number eight awesome
thing you can do, you can crawl a single
page application. Well, maybe you want
to visualize your app. Maybe you don't know all the
URLs of your single-page app. Maybe you want to create a
sweet D3 visualization of all the pages in your single-page. Maybe you want to create
a site map on the fly. You can totally do that
using Puppeteer and the APIs that we have. So to do this, you can discover
all the links on the page just by using page$$eval, grab
all the anchors on the page. And again, this is going to
get run inside of the page. And we just look
for all the anchors. Are they the same
origin as the page? Are they part of our app? And are they not the app
we're actually viewing? So we don't want to,
like, render ourselves. So we return the unique set,
we'd run this recursively, and then that's
basically the way that I created that
D3 visualization. And you can do not
just a list of links, you could do some like
this with Santa Tracker. It's a very visual
single-page application. As you visit each link,
you can take a screenshot or generate a PDF
or what have you, and then kind of visualize
your app in a different way. Number nine is one
of my favorites-- verify that Service
Worker is actually caching all of your app. Every time I use Service Worker,
I always leave something out. I always forget
to catch something in the cache, which means
somebody comes back to my page, and ultimately my entire
app doesn't work offline. They get a 404, like an
image is broken or something. Well, we can verify
that that's not going to be the case using
some of Puppeteer's APIs. So to do this, we'll
navigate to a page that uses offline caching, and
then we call page evaluate. We wait for the Service
Worker in the page to be installed and ready. That's what this page
evaluate method does. Next thing we do is
we basically just look for all network requests. Any request that
happens on the page, we'll use network interception. We'll just stash the URL
to take the list of URLs that the network gets. After that, we reload the page. We want to know what's
coming from the network and what gets served from
the Service Worker cache, and we want to be able
to determine that. So we just reload the page
because at that point, Service Worker's been installed,
it's cached its resources, and then we can check and
see where things come from. And that's what
this last line does. It basically loops through
all of those requests that get made by the page, it
checks their responses and determines if they've
come from the Service Worker or if they've come
from the network. So here's an example
of the script. If you run this on the
URL, you can basically see on Chrome status everything
is cached, which is great. You get a little green
check for all the URLs that are being returned
by the Service Worker that are going to work
offline, and anything that doesn't, like these
analytics requests, have a little red check. So that was a choice
that I made in this app to not cache analytics
requests, but you can see everything else
gets cached offline. So the last cool
thing-- and there's many more things you can
do with headless Chrome, but the last thing that I have
time for is to procrastinate. So I didn't really have a
good demo of the keyboard API in Puppeteer, and we have
the touch APIs and stuff, you can do touch emulation. But what this is going
to do is basically just open the Google Pac-Man
Doodle from a couple of years ago and play it in Node. So I'm going to basically
get keyboard events in Node, I'm pressing the
arrow keys in Node, and eventually, the
game will fire up, and then I'm forwarding
those keypresses to the page. And, of course, the
page just has JavaScript that knows how to
handle keypresses, so I'm able to play Pac-Man
in a web page in Node, and play it online. Kind of a cool example of
bridging these two worlds again. Somebody likes it. So that's kind of fun. So before we wrap up, there
is a number of useful things that I think I want to
draw to your attention, a number of sites and tools. The first one is
Puppeteer as a service-- the notion that you
can run headless Chrome, kind of the
browser as a web service. So a lot of the demos
you saw we actually just put on handlers
in the cloud. So this first one here,
you pass to the URL and it takes a screenshot. It runs headless Chrome,
it does all that stuff in the background. But you can kind of think
about baking headless Chrome, the browser, into
your web service. So that's Puppeteer
as a service. We have an awesome Google
Chrome Labs GitHub repository with all those demos
I showed you today, as well as some other ones
that we didn't talk about. A lot of useful stuff there. If you want to see anything
else implemented, let me know. I'll create a demo for it. This is a cool site
that we put together to just try out Puppeteer. It's called
trypuppeteer.appspot.com. You can go in, you can
kind of prototype ideas, play with the code,
run our official demos, see the results. You don't even have to install
anything to kind of work with Puppeteer. So that was a lot of stuff. We covered a lot of things
headless Chrome can do, and, of course, there's
a bunch of stuff that we didn't cover
that it also can do. But things like
server-side rendering, pre-rendering your apps, offline
testing, offline verification, A/B testing, making the
Google Search Bot happy, creating PDFs. So with that, hopefully
you guys have realized that automation is a thing. It's not just about
testing your apps, it's actually about making
your app more productive and yourself as a
developer more productive. So headless Chrome is a
front end for your front end. With that, my name
is Eric Bidelman. It's @ebidel on
GitHub and Twitter, if you want to converse
with me after the show. And I will leave
this one up here, which has got a
great list of things you can take a screenshot of. So thanks for everybody
sticking around. I really appreciate you coming. [MUSIC PLAYING]
