[MUSIC PLAYING] VINIT MODI: Welcome everyone,
to this session on CameraX. My name is Vinit Modi, and
I'm the product manager on the Android
camera platform team. Today we'll show you how to
minimize the camera development time with a consistent and
easy to use camera API. We'll have some
code walkthroughs, a lot of fun demos,
and we'll end by showing how you can elevate
your in-app experiences. Now, many of you know this-- camera development is hard. Why is that? Within your apps, you need
to account for various OSes. Second, you have to provide
a consistent experience across a range of devices,
from entry level to flagships. And finally, the camera
APIs are very powerful, but difficult to master. So we're excited to give
you this first glimpse into CameraX. This [INAUDIBLE] library is
launching as part of Jetpack in Alpha. We develop completely in
the open, part of an Android open-source project. We update multiple times a year. So let's learn more. So CameraX is backwards
compatible to Android L. This works with 90% of
devices in the market today. So when your target SDK
is API 21 or higher, you'll be able to use CameraX. Under the hood, CameraX uses
the public Camera2 APIs. Now we know what you're
thinking, what about the old deprecated Camera1 API? So we provide the
same consistency as Camera1 via the
Camera2 Legacy Layer. We've tested, identified,
and fixed a number of issues across these older devices. So how do we achieve
this consistent behavior? Last summer, I was in China
at the Android GDD event, and was shocked to learn
that several developers are running manual testing
across hundreds of devices. So we understand
your pain, and felt that the best way is to invest
in an automated CameraX test lab. This test lab has devices
from Android L all the way through Q, has devices
from various manufacturers, and it runs 24 hours a
day, seven days a week. By providing an automated
testing experience, we're able to guarantee a high
quality camera performance across all your apps. We've tested hundreds of devices
across all OS layers to date. And we do basic testing
like orientation, rotation, taking a picture. One of this test suite is
the latency test suite. And we're open-sourcing
that test suite. With this test, you'll be
able to start benchmarking your camera start-stop
latencies, your latencies for photo capture. This is a very powerful
tool that we've used very recently with a lot
of the top Android applications. Here's just a list
of example issues that we've tackled with CameraX,
and we solved on your behalf. For example, the
front and back camera crashes, optimizing
camera closures, or just making sure
that the flash works. Now this is our season
premiere of CameraX. There are lots and lots
of devices out there. So we really hope that
you try out this season-- you try out this alpha version. Give us your feedback. And then as we improve, it
will just keep getting better. Finally, I want to
talk about how CameraX is incredibly easy to use. The current Camera2 API gives
you very fine-grain control over all the sensors. What we have done is
abstracted away the hard bits, and provide a use
case based API that simply lets you get a
display up on screen, gives you high quality frames
so you can do amazing analysis, and simply take a
picture or a video. Finally, we bind all these use
cases to the app's activity lifecycle, making sure that
CameraX is lifecycle aware, and you don't have to start
and stop the camera separately anymore. Awesome. So here's the
typical walkthrough of the camera app architecture. Today your camera app might be
talking to the public Camera2 API. And under the hood, it's
talking to the Device HAL, or the hardware
abstraction layer. In this case, all the
device-specific quirks have to be accounted for
within your application. Now let's imagine
you're using CameraX. In this case, CameraX
hides all that from you. We just give you consistent,
reliable performance. Now, there are a lot
of advanced use cases. And for that we have an
experimental version, where you can inter-operate
between Camera2 and CameraX. So let me welcome
Trevor up here on stage. We're going to do a demo
on the Camera360 app. Now Camera360 is based in China. They have hundreds of millions
of installs across Android. And only on this stage,
you'll see a demo on a 2016 Samsung J7. This device is running
Android 6.0 Marshmallow. And under the hood, CameraX is
talking to the Camera2 Legacy Layer. He's able to change
all the effects. All this is done using CameraX. Now Camera360 is using
all the three use cases-- preview, image analysis,
and image capture. They're using image analysis
to track Trevor's face, and draw the effects around it. And whenever Trevor is happy
with the picture he likes, he clicks the button
and takes a picture. It's so easy and fun. I want to get in on
the next picture. Thank you, Trevor. So Camera360 had a number
of benefits with CameraX. They were able to reduce
their device-specific testing, and saw a 75% reduction in lines
of code compared to Camera2. This is in addition to making
the code easier to read, and a smaller APK size. So let's summarize. CameraX is backwards compatible
to Android L, working on 90% of devices today. We provide a consistent
experience across devices. And we test using the
automated CameraX test lab. And the API is
incredibly easy to use with a use case based approach. So up next is James Fung, who
will deep dive into this API. Thank you. JAMES FUNG: Thanks Vinit. My name is James. I'm a software engineer
on the CameraX team. We talked briefly
about use cases. So these were the preview image
analysis and image capture. When we sat down
to design CameraX, we realized we had to balance
a few design constraints. We want to make an API
that's simple to use. We want to create an
abstraction level that lets us hide device-specific issues. We also want to make sure
that the API is flexible enough so that you can create
fully featured applications. So how did we think about this? How do we go about this? We figured about creating
a programming model, where we really are capturing the
intent of the developer, but also providing the
right amount of flexibility. So really we have an API-- and we'll walk through
some examples-- where we're really asking three
things from the application. First, what's the
intended usage? Which of the use cases, one or
multiple, do you want to use-- preview, image analysis,
and image capture. Secondly, where
should the output go? When you get the
frames, how does that hook up to your application
so that you can write the code that you need to do? And finally, when
should the camera start? When should the camera stop? So I'm going to walk
through some code now. And these examples
will be in Kotlin. Say you have a device
at your workstation, and you're starting to use
a camera for the first time. One of the things you might want
to do is start up that camera, and see those live frames
directly on screen. We call that preview. Let's take a look at what
the code looks like for that. We start by
configuring a preview. So here I build a preview
config using a builder. I'm showing it here, in
its default configuration. The default configuration
is actually fairly powerful. What this is going to do is
on that particular device, it's going to understand
the available resolutions. It's also going to understand
the resolution of the display. And it's going to try and
make a reasonable decision based on that for you. It will default also
to the back camera. And making sure that the
default configuration runs well is an example of something that
we're able to test in our lab. Now at this point, you
could set additional options if you needed them. We'll show some
examples coming up. Now that we've
configured the preview, we can actually create a
preview use case, simply by handing it the config. And we'll see if these
use cases actually have API specific methods. The preview provides a
method to set a listener. What's going to
happen now is when the preview becomes
active, it's going to output a preview output. The current version outputs-- within the preview
output-- a surface texture that's
configured and ready. You can use a surface texture
to attach to a texture view, or use this with a GL renderer. Future versions may
interact with surface view, or accept the surface
texture from the application. Now with this code,
CameraX is configured. We just need to be able to
turn it on and turn it off. So the third step-- when should the camera start? When should the camera stop? We do this by binding
the preview use case to an activity lifecycle. Now what this means is,
when the activity starts, the preview is going to
start, and the camera is going to start to stream. When the activity stops, the
preview is going to stop. The camera's going to shut down. Other life cycles
can also be used. For example, a
fragment lifecycle could be used here instead. And in this way, we're able to
hide the complexity of start and stop. In particular, shutting down
things in the right order can be kind of tricky. So here, we can see all three
steps being done together. Now a real camera application
will have additional code. There would need to
be code for setting up permission, attaching to
views, and managing the surface texture. But what we do show
here is actually sufficient to get the
camera system setup. We've hidden many of
the details of Camera2. For example, opening the
camera, creating the session, preparing the correct surfaces,
selecting the resolutions, and the careful shutdown
that you sometimes have to do to make sure that
everything works just right. I'll pause here
just for a second, so you can look at the slide. So let's say you have that
first preview streaming now. Maybe the next
thing you want to do is be able to access
that camera data so you can do some analysis
or image processing on it. So for this, we have something
called image analysis, which provides easy access to the
buffers from the camera, so that you can perform
your own analysis. What does it look like? It really follows the
same steps that we showed with the preview. First, we're going to go
ahead and create a config. Here I'm going to show
what it looks like to set an option on the config. In this case, we're going
to request a resolution. Now let's say your
processing requires some minimum resolution
for it to succeed. This is where you
can specify that. What CameraX then
is going to do, is it's going to balance the
requests from your application with the device capability. So it's going to look at your
target resolution request from the image analysis. It's also going to understand
if you have a preview running the requirements there. And it's going to balance all
that by looking, also, at what the device is capable of. So if you're able to get
the target resolution, you'll get it. If not, we'll try next
higher resolution, so that you can get
some guaranteed minimum. Failing that, we will
fall back to a 640 by 480 resolution, which is
guaranteed across all devices. The key thing here
being that CameraX is doing everything
it can to make sure on that particular device,
it's setting up a session that will run for you. Once we have the config, again,
we can go ahead and create the use case, object itself. Step two-- what do we
do with the output? Now in the case
of image analysis, it lets your application
set an analyzer. The analyzer is
provided an image proxy. This is a CameraX abstraction. But basically is wrapping an
Android and media image class. So you have the same access
to the same data there. You'll notice we also have
a rotation parameter that's set here. We understand
rotation can be really important for image analysis. Now CameraX is not going to
rotate the data internally because we know if you're
going to do an analysis pass, maybe you're doing a
per pixel pass already. And it might be more
efficient for you to do the rotation in
place, if you wanted. Rather, what we mean
by rotation here, is we're going to make it
easy for your application to understand which way
is up in that image buffer that you've received. Knowing which way is
up can be important. For example, if you're
doing a face detection. Perhaps your face
detection model requires knowledge of which way
is up for it to work correctly. And it's inside
the analyzer then that you have all
this information. You have the image
available to you. And you can actually
insert the code that you need to
get your job done. Finally, step three--
when to start and stop. Here again, we buy
into a life cycle. So this is an example, now, of
finding both an image analysis and a preview at the same time. Now what this means is, when
the activity becomes active, both image analysis
and preview will be running at the
same time and ready. So here we have, again, all
three of the steps together for the image analysis. What's going to happen now is
when the camera system sets up, that preview is
going to come up. But also your
analyzer function is going to start to receive
images at the camera frame rate, and your analysis is
going to start running. I'm really proud to say
that we partnered with Lens. And CameraX is being used
inside Lens in Google Go. Lens in Google Go
is a version of Lens intended to target
low-end devices that helps users understand text. Some of the benefits
they expressed to us was that it allowed them
to focus on their core competency--
understanding the text, and creating a really
great user experience. They were able to spend less
time worried about the camera stack. They were happy that they
found that CameraX just worked on a diversity
of low-end devices. Lens in Google Go also has
a tight APK budget size. With one of our
early builds, they were able to integrate
the camera stack into their application for
less than 30 kilobytes. Image capture allows you
to take a high quality picture with the camera. Inside image
capture, we do things like implement focus control,
handle auto exposure, and handle auto white balance-- sometimes called 3A. Additionally, within
image capture, we're able to optimize capture
requests on a per device basis, if necessary. Let's take a look
at how that looks. It's going to follow the same
pattern as our previous use cases. So first, we create a config. Now we know getting rotation
on devices can be tricky. Getting portrait
mode and landscape mode just right on a
variety of devices is hard. What we've done here
is reduce the problem to having the application
specify its current display rotation. What's going to happen now
is internally inside CameraX, we're going to look at
the rest of the transforms on the device, and make
sure that our output is doing the right thing
for your application. For example, in the
case of image capture, we're going to have a
standard set of transforms, and we're going to make sure
that the [INAUDIBLE] metadata is set correctly. Then we go ahead and
create the actual image capture use case itself. Then we bind to the
lifecycle, as before. Now here's an example of
binding all the three use cases together-- preview,
image analysis, and now image capture. Again, here's the code to
set up the image capture. And so now when the
activity starts, the preview is going
to come on screen. If you have any analysis,
it'll be running. And your application is
ready to take a picture. But what about step two? What about attaching
that output? It's another example of the
use case specific method. So for taking a picture,
it might be pretty common that the user would
tap on screen. That would activate
a button click. And perhaps that button click
would call into some function. Let's say, onClick. So when the user
clicks a button now, this can invoke a function. And within that
function, you can simply call imageCapture takePicture. Here, I show first
preparing a file. And this will be the
destination of the image. And then simply calling the
imageCapture takePicture function. With that take
picture function, you can then specify the target
destination-- the file itself. You can also specify what to
do after the image is captured by specifying a listener. You can specify
what to do on error, or what to do after the
image has been saved. Really happy that we've been
able to partner with a company called Snow, and we'll be
able to demo that today. So Snow is an application. They're based in Korea. And have a really large
multi-million user install base. So Trevor is going to
help us demo this again. So Snow is a really fun
app for taking selfies. And we're going
to show this demo. This is running
on a Samsung S10+. And we're showing
all the different use cases we talked about. We're showing preview to
get the image on screen. We're showing image
analysis to detect the face. And then also
render an overlay-- and then the whole gang
is going to come up. And finally, when we're ready-- and see if we can get
a picture of that we like with all the foxes. When we're ready,
we can even go ahead and use the image capture. And that'll take a photo. Thanks, guys. [APPLAUSE] So this is a great example
with what we have today-- of using all these
three use cases together to create a fully
featured application for taking fun selfies. Working with Snow--
they expressed some of the benefits
of CameraX to us. They found it was easy to use. They appreciated that we
managed the camera lifecycle. They also appreciated that
we handled camera threading internally to CameraX, without
the need for them to have that in their application. So in summary, we've talked
about the use cases-- preview, image analysis, image capture. We talked about our program
model built around three steps. And with just preview
and image analysis, we talked about how that
can make things easier. For example, for ML--
hooking up to ML-- by showing an example
using Lens in Google Go. And finally, we've shown
that what we have today is actually flexible
enough to create a fully featured camera-based
application. Now what I've shown today
is a starting point. This is a glimpse
into the direction we're taking with camera API. But it's available
today, and I'm really happy to have you
guys try it out. Thanks. VINIT MODI: Thanks James. So we have one more
thing to talk about. There are many new camera
capabilities that are typically part of the native camera app. For example, Portrait,
Night, HDR, and Beauty. We've heard from many of you
wanting access to these device specific capabilities. And we're excited to share
that CameraX extensions enables just that. We've partnered with some
of the top manufacturers across the world-- from Samsung, Huawei,
LG, and Motorola. And you'll start seeing devices
supporting these extensions this summer. With Huawei, you'll be able
to see the HDR and Portrait effects on four existing
device models that will be upgrading this summer. And best of all, it's
two lines of code. If a device doesn't support
any specific functionality, then the Boolean returns
us false, and it's a no-op. There's no manufacturer specific
coding required on your part. CameraX handles all
of that for you. So we're excited to show you
a demo on the Samsung S10+. So I'll welcome Franklin
back up on stage. So this app that
Franklin is running, is the CameraX sample app. What Franklin is going to
do is take two images-- a picture with the normal
mode, and a picture with the HDR mode. And let's compare
the two images. So in the HDR image, you'll
see that the light that's coming out of the glass
door is actually better. The podium here is quite
dark in the normal picture, but it's more lighted
up in the HDR picture. So all of these functionalities
can be added very simply within your application. Thank you, Franklin. So here's a picture comparison
of images taken using CameraX. On the left, you'll see HDR off. And on the right,
you'll see HDR on. And we have one more. Samsung will be the
first manufacturer to bring night mode
images to developers. On the left, you'll see an
image in low light conditions. And it's grainy. But on the right, with
the night mode enabled, that image is sharp. The colors are more vivid. And imagine the experiences
that you can bring to your users when you enable
these extensions. So we have one more demo. This demo is the
app YouCam Perfect. It's a camera-first application
based in Taiwan with hundreds of millions of installs. This demo is on the
Huawei Mate 20 Pro. And it's running the
camera extensions both in the viewfinder,
and in an image capture. So let's jump to the phone mode. Oh, it's showing the desktop. There you go. Perfect. So in the app, all it's using
is two lines of code for CameraX to enable the extensions. And you'll see that behind
me, the effect is actually blurred out. So let's try a glam shot. Cool. So underneath, it's
using all the device specific capabilities. There's no computational
photography that YouCam had to do. So this type of
behavior can be enabled in all your applications. Thank you. So here are some of the
benefits that YouCam expressed to us when using CameraX. They get access to a
lot of new features that are the same as
the native camera app. And the best of all, it still
reduces the lines of code. It's 70% fewer lines of
code compared to Camera2-- all of this on top of getting
a consistent experience across devices, and
an easy to use API. So let's recap what we've talked
about in this session today. CameraX is backwards compatible
to Android L, working with 90% of devices on market,
provides a consistent behavior across a number
of devices, and we do a lot of the testing for you
with an automated CameraX test lab. The API is easy to use
and lifecycle aware. And finally, with the new
extensions capabilities, you'll be able to try a lot of
new effects on newer devices. We sincerely hope that this
is the API and the changes that you're looking for. And we really value and look
forward to your feedback. I want to thank some of the
early developers that are partnered with us on CameraX. And they've provided us a lot
of guidance and constructive feedback. So you can download
CameraX today. You can add it into
your application, just like other
Jetpack libraries. And please share your
feedback with us, either by joining
the Google Group, or tagging us on Stack Overflow. You can meet us in the Sandbox. You'll be able to see some of
the live demos that we did, also in the Sandbox. Or you can reach us in
office hours tomorrow. So thank you very
much for attending. [MUSIC PLAYING]
