Android Camera2 API by Huyen Tue Dao

you guys are gonna listen to me talk about camera two so just quick show of hands who's in the Android type person in the room and how many people have done stuff with camera the camera API okay cool so I've actually done a few talks on the camera API before and usually what I love doing is deep dives you know I love looking at code I love taking people through examples and saying well you know here's the ins and outs here's like stuff that messed me up and here's some you know things that worked for me and you know camera two was kind of announced as part of lollipop sometime last year and I got really excited if you know when they talked about you know camera two they kind of talked about all these improvements made they made and if you've ever done anything with hardware you know that sometimes it's not quite so easy you notice that the API aren't perfect sometimes you have to deal with quirky hardware so any improvement always sounds like a good thing right and actually the last talk that I gave you know I actually had a little mini section where I had kind of read up on all the cool things that were going to look forward to in camera two and I had a whole bit where I went camber the feature of cameras gonna be amazing you know we're gonna have these more efficient and more powerful API it's gonna be so good and you know I was thinking about this talk and I was like well I should do a deep dive like I always do and we'll just stick down and do a new application and just play with all these new cool stuff well I actually sat down with the camera to API and instead of it being but it was amazing but it was also more like this do you bucks because if camera API 2 is a big improvement but it's also incredibly foreign challenging it takes a lot more work and what I found is that when I have to jump into it it was basically very little like the experience that I had before so what I found that really helped me was reading the documentation and it wasn't even just the camera API to documentation I end up researching the graphics architecture I ended up looking at the actual Hardware layers that lived between the Java framework the android framework the underlying camera drivers and that's when it actually clicked and instead of doing a deep dive today what I wanted to give you guys is a high-level gues introduction to the changes between camera one camera and camera two and explain to you why you know things are changing why things are so different and why it is better it's just harder and I think the thing that a little bit will benefit anyone trying to dive into this is just understanding the underlying mechanics and understanding the motivations of the change more than just you know taking you through some code because honestly I've have you guys here for three hours and he's probably gonna get sleepy or starve or get mad at me so we're gonna keep this high-level and try to kind of get you on the right foot so you're doing a pi/2 so oh wait sorry but this was the after about the day of camera two before we needed documentation oh I'm not trying to scare you off just to be honest you know it's a lot different and if you've done anything with camera everyone this is going to be quite a different show so I don't want to take you through a quick comparison of camera one and camera two and talk about the state differences that you know between the two api's I'm going to talk about the underlying changes in what is known as the hardware abstraction layer that is basically the foundation for the new API and we're gonna talk about how you actually get started building an application with a new API I'm gonna run you through a very high-level the steps to actually capture an image and even though that sounds like not a big deal it actually needs to become a really big deal and then hopefully if I don't keep you guys here too long we'll wrap things up and have time for a few questions before the awesome raffle all right so just a little bit about me I'm an independent contractor I'm a native Android and iOS developer liván and we're not in Denver although I am originally from a DC metro area I graduated from University Maryland with a degree in computer engineering and the only thing I do then develop is game I love a very computer centric life with not a lot of sunlight so camera one versus camera two when I think about the difference between camera one and camera two I actually came up with this really cool analogy last week where you know if you think about camera one it's really like a point-and-shoot camera right it's really simple but easy to get started it has a decent amount of functionality but it's not too advanced you may not get the greatest performance you might not get the greatness quality images but it's really easy to use it's straightforward and you know it's pretty good for the average consumer you know the average developer camera 2 is more like a DSLR camera it's a lot the learning curve is a lot steeper there are a lot more knobs to toggle and the adjustments to make the language you use when you talk about things on a DSLR is a lot higher level then you would talk about a point shoot but overall you have more capabilities you have more control you have higher quality in higher performance and you have a lot more room to mess up you know you have to really know what you're doing you have to understand kind of the mechanics the underlying mechanics of the camera to really get the most use out of it and that's pretty much exactly what the situation is with one versus two one it was pretty easy get started the language the actual method names are very straightforward whereas camera two is a lot more advanced you kind of have to know where things are and what you're doing ahead of time there's not a lot of jumping in and getting your fingers in there and hoping that a picture pops out you kind of have to be a little bit more paired so so the the main difference that comes out of camera 1 camera 2 is due to the hardware abstraction layer now the hardware abstraction layer layer are these kind of intermediary pieces between the android framework and the underlying camera drivers so it's basically an interface that allows that drop the camera driver and the android framework to move to communicate so in camera version one it was really simple it was basically a black box as far as you know you your application and the Java or the android framework was concerned Egypt sent some requests maybe you want like a camera preview you know like the low res video you can use as a viewfinder you might want picture you might want a video you just sended request is black box and at some point later some image data is submitted in a pop-up and that's pretty much it I've heard it described as like a one-way system just because basically requesting results out and that was it now it formed that that's the basis of the camera one API and you know again it's straightforward it's simple because it is a black box there's not a lot for you to kind of get your get your get your hands in and mess with and you know it works it works but there are some problems and limitations again because it's a black box because you don't have any kind of idea or notion of control of the underlying camera system there's only so much you can do with it so that's not to say that camera wanted useless there's a lot that you can do there's a lot of studies a lot of parameters that you could change when capturing image data but it was a bit limited and it's kind of like the biggest deal I guess or the biggest limitation is that the camera one subsystem and the subsystem is kind of like all those layers from like the android framework down to the devices for the camera driver that entire subsystem is basically centered around three modes so you have three modes of operation you have preview mode you have still capture you know just pictures and you had video capture and that was it and basically that that black box was locked into one of those three modes so it so we basically became very difficult to add any new functionality your add or give the developer and also the user more control so a big example that people kind of were booing about when camera two came out was this I finally getting burst mode now burst mode is this idea that you know you want the camera to take a certain number of requests or take a certain number of images in a short amount of time now if you think about three modes that we had with camera one you know preview still and video and if you think about still you're taking you know a single you know static image video is more like taking a continuous stream of static images right now burst mode like I said is like taking X number of pictures in a short amount of time and that doesn't really fit into either still capture a video mode you know it's something in between and because that's black box that camera one was locked into those three modes you weren't able to get burst mode out from any API you know updates or anything so again the limitations and also you know again if you had a camera that is that you could give the user manual focus or manual exposure because of the nature of the underlying hardware abstraction layer you couldn't actually let them get ahold of it so in order to should I guess kind of get past this there were some big changes so instead of just changing the API you know this is not just adding new features to the existing API this is an entire rework of those middle layers that hardware abstraction layer that interface between you know camera drivers and the android framework has been completely redone so they basically broken that black box open and giving you a lot more of a detailed view of what's actually going on inside and so I hope I keep saying you know camera two is not an update so much as a complete rework okay it's based on an actual update to the hardware abstraction layer so it's not like he just downloaded to your phone and get the new HAF you know it has to be kind of something done by the manufacturer side and it kind of such a huge change it is actually a deprecation of camera one so so that's kind of good news and bad news because good news is we're going to get a lot more functionality but bad news that you're gonna have to deal with the camera to EBI if you want to do camera applications so okay I may be a little bit negative but what are the good things that are coming out of this Wow you're gonna get a lot more control over that subsystem you know if the camera does support manual focus and the camera does support post-processing which is like all those little algorithms that do color correction and sharpening and all kind of stuff you couldn't actually kind of adjust those things as a developer some more kind of cool things as more capabilities you can have raw support you can have you know well you have pays attention before but you have like you know a lot more ability to kind of pick up so the really cool candid camera features and overall it makes it actually it's making the API more efficient and more maintainable so again kind of going back to that an example of burst mode you know we're not limited by these three you know specific modes we now have a lot more flexibility because you know everything is done in a little bit more of a generalized way we can kind of extend it in a lot a lot easier okay so for simplicity I will no longer say hardware abstraction layer I'm just gonna call it how so camera two is built on version 3 of house so how they have three and instead of having these three static modes preview still and video everything's been flattened out into kind of a very consistent single request system and basically instead of talking in really high-level terms like preview picture video you're going to talk about frame requests and frame captures so basically anything that you do you need to translate it you need to think about it in terms of frame captures so as I mentioned before taking a still picture is just grabbing the frame of data grabbing the video is not going to become a repeating request for frames of data and that's how that's what a video is basically in a whole bunch of snow pictures to jam together and so the language is a little bit different instead of having the camera API say start preview capture picture you're gonna be talking about capture capture burst repeating capture repeating bursts so that's that's kind of like the difference here the loop speaking about image capture in a different way now one of the really cool things that has changed in this new hardware abstraction layer is that you know we're kind of taking things more a low level right not talking about video and I'll have my pictures we're talking about Frank's now the really cool thing is is that with that kind of low level and ice we got some perks like now when you make a request for that single-sim of data you can actually request that that data be sent out to multiple outfits so you could have like a surface that's rendering you know the image on in your application you could have like an image reader that you can say JPEG data to and save it to a file you can even have what's an allocation class which basically can send things you to write to render script and or a media encoder if you want to do videos so and you can actually do all of that you can have success that same image in its multiple you know outputs well all these kind of different places targets for your data you can do that with single frame requests and so basically when you make a request for that data you can specify one two and I think it's somewhere between three and five I think it's limited by performance but yeah so you can make a single request and get a whole bunch of different versions I guess of that data alright so another big change with how three and camera two is metadata so when you're talking about photography you're talking about doing advanced features metadata it's actually really really important you know before with camera one you got metadata back but it was pretty you know simplistic you know my picture was taken some kind of basic information you know maybe location maybe this maybe that but no metadata is kind of basically just is propagated throughout the entire you know half so every time you make a request every time you get a result back from that request it is chock-full of metadata and so one of the big things is that the camera characteristics the basically features on that camera like the sense like the lens you will get information about all of that now you'll still get frame kind of information about each frame you know timestamp and certain statistics that get generated when a frame is captured you'll get all that back now and just kind of overall whenever you make requests and whatever you get that request back you'll get all the data about it in fact when you make requests and the results come back you don't just get the image data you'll get your original request so you'll know exactly what you asked for and then you'll get the actual configuration of the camera at the time that that image was captured so you don't know what you put in you know you've what you got out and you'll have the image data that goes with that so really cool now having data is not just good for the sake of having data but metadata and you know more complex more detailed metadata is required for certain features so think about HDR right HDR is this process where you take a bunch of pictures at different exposures and you combine them treats to create a higher quality image well that whole process is based on something called exposure bracketing we're just basically you take the same picture at different exposure you know levels then you have a certain amount that you're you know increasing or decreasing the exposure by it's really important for you to know we have the exact exposure that each image is taken out so again that's where that detail that accurate data comes into play another fit another feature that is really reliant on metadata is call support being able to generate RAW files depends very much of having the correct sensor information and we have that now again metadata metadata metadata very important in and again we have a lot of it now so here's kind of a graphic so gonna compare with like the earlier image I showed you of camera one being very simple you know being a black box now you can kind of see a little bit more what's going on now so instead of again start start preview take picture capture video we have the capture request up in the left corner I mentioned before you can give you know very specific settings for a frame you can specify multiple targets where all that image data is gonna go and you feed that into the camera device which eventually leads it into the camera hardware and then once you know the camera captures an image and does some processing and splits it out into the different you know how quick it'll send that to all these surfaces here so at some point you've probably set up some surfaces to take that data in and either render it or save it or something so whenever that request actually makes it to the carver hardware you're gonna get that image to directly into your surfaces and I've been mentioned before with all that metadata along with the image data from that request in your surfaces your object you're also going to get a capture result and that capture result contains both the original capture inter-are your original capture request with all those configurations that you've asked for as well as the actual final configuration at captive time so but again it's a lot more of an open view and you get a lot more of an overview and it also gives you a more of a chance to kind of get your hands in there so another big change with how three and I said it spaghett it's freaking open this black box it's actually exposing the camera pipeline so when you're taking a digital image it's not just kind of like one action it's a series it's a process right a lot of stuff is happening you know there's gonna be data captured in from the sensor that data is gonna be taken in you know processed and like I said like I mentioned before there's going to be needing some color correction there's going to be some edge sharpening there's gonna be a lot of other processing that happens with that data and then after you take it you know that didn't have to be scaled and props because you know you think about digital camera you can actually request you know what size it's a picture then so there are units in the camera process to take care of that so what how three actually does is to expose the pipeline to the framework and also to the kind of on the hardware side and so this is cool in two ways it's cool for us developers now we can actually send triggers into the pipeline and as I mentioned you know camera two you actually can kind of adjust and tweak all this prop all these post-processing you can um you know specifies different kinds of scaling and propaganda get different kinds of encodings you know JPEG get raw data now you can get even why UV data if you're a video type person and on the manufacturer side it actually is cool for them as well because each of these comes each of these stages in the pipeline is its own component so a manufacturer can actually swap out any of these components for their own implementations meaning that they can pick out the algorithm or the component that works best for that hardware so it's really cool from their side because they can make you know their camera harder implementations as efficient and you know high quality as possible and from our side we can actually reach in here and mess with stuff so pretty cool stuff all right so I've kind of landed on to you a bit about the hardware so let's take a look over to the developer side and see what we got so I always like to start out talking about developing camera a camera applications with what I call the stones principle and the stones principle is this you can't always get what you want but if you try sometimes you might find you get what you need and as Android developers you know we deal every single day with you know what API do I want to support what features that have and what is what do I want my user to be able to have access to so the idea the stones principle is kind of magnified and you're talking about hardware you know like you know we deal every day with well one users running Gingerbread for four or one users running jelly bean and I've got this and I've got that well when you work with hardware the heart becomes just another element you have to worry about I mean you could have two phones running running lollipop and one has a really awesome camera that has a manual focus a manual post-processing and another one might have all the clock but the crappiest camera in the world and so so their capabilities are completely different even though they're on the same API so it's really important when you do a camera application to really kind of plan out ahead of time what are your critical features and what are your optional features and the framer provides you tools to be able to you know filter out your application on Google Play to only those Fiona those users that have those things that you really really need and at the same time for those things that you would like to have but you don't necessarily you know have their messily required for your application to be checked at runtime and you can you know alter your logic you can hide features or do whatever you need to for different users so on the neat side as I mentioned before we're most talking about Google Play filtering and kind of working with that manifest and again kind of using the first decision you make is what minimum was my minimum SDK and then on top of that there's actually the users features tag and this is actually really important with cameras because there's a lot of different features hardware features that a camera that one person camera might have another person's camera might have so again it's really important to use these two determines you know mission-critical features in your application now for things that are not so much and critical that you could live without there are a lot of things that you can do to check at run time now something that we probably used before is the filter is built up version value you can make the check you know oh one DP is my user on okay so they might not fade detection they might not so don't show them this part of the UI don't you know show them this part of the UI instead there's also the package manager and the package manager it works a bit off of the same features that use feature does it usually usually use that to say check at run time ask your user has a flash if the user has autofocus so that's what the package managers for now you can also query the camera API is themselves for kind of information on and kind of settings the camera has like color mode right you can ask the camera hey do you have sepia do you have black-and-white or do you have like flash on flash off ability and stuff like that and so in camera one we had this object called camera parameters now camera parameters actually was doing double duty it's a way for you to determine you know what kind of settings you can adjust on the camera and it's also a way to actually kind of set those settings when you're actually capturing an image now camera do we have something different it's called camera characteristics that give incorrect characters exactly really powerful it contains not just camera parameters but also all that really cool really detailed information about the camera itself so since information lends information all of that you can pull from camera characteristics so but again it's really important especially with camera applications to really know what you're getting into know what you really need and know what you want but you you know can live without alright so I just talked a lot about what what what you're doing what your users device might support and what it might not support and so when your concepts in the camera to API is this supported hardware level so there are three supported Hardware levels and we'll go from the top down so basically this is a just an indication of features that you'll be able to have access to as well as performance constraints now the the first level is full and as it might sound for the name it basically guarantees you a certain set of really advanced features and this manual focus it will include manuals post-processing so all those cool the ability to tweak those things there's also performance constraints Anna you have to keep I think and the document even has a look a lot better explanation on it but there are certain performance constraints for something running out three so again that's all kind of indicated by you know the foal that you're kind of running you know all the cool feet all the cool features and running them in a preferred way so the next level down is limited now limited cameras will be running how three just like full but they're not necessarily guaranteed to have all those cool features that full house and then finally there's legacy now legacy so so if your legacy that basically means that you have a camera that's not at Calgary is probably how one or how to but you're still able to use the camera to API probably cuz you're running a lollipop but that camera2 api is more of a wrapper around the old camera stuff so you're basically guaranteed not to have any of the cool new stuff and you're also at the same time have more relaxed performance constraints because you're not on that neo hardware abstraction layer and again for a much better and more articulate explanation of all those things you can go to the documentation it's pretty good alright so Android application start off with the manifest so there are some pretty important features in the manifest using your data camera first you actually have to ask permission to use the camera because it is a hardware feature and the kind of tricky thing to note is that if you just drop in the camera commission it is going to implicitly require that your users have the back camera a fun front camera autofocus and flash so if you are done in a camera or one word you know the documentation tends to yell at you and say look you really need to use use features okay you know like we're gonna expect these things if you don't tell if you don't need them then you know you need to kind of spell it out with these features so just something to keep aware of so there are a few more manifest values that might be interesting to you so you can be a little more relaxed with your requirements if your application just needs a back or front cam neither one specifically you can just say that well my user just asked has to have anything you can do that you can actually specify that your camera must support external devices in this pre-new is APR 20 and then there are three new values that are very specific to camera two and again these are the fancy new sexy features main post processing manual sensor and then there's actually a manifest value if you want to make sure that your users are on only phone you can go ahead and put that in your manifest oh and so when I looked about a month and a half ago the only phones that I could find that definitely had full support for the Nexus 5 and Nexus 6 I believe that there are some newer HTC devices that do have limited but that's that's kind of common is it might have changed a little bit by now but be aware when you're doing camera 2 it is new and it is got a hardware component to it so it's not going to again like I kind of mentioned before it's not like you can just drop in the new Hal and have a sexy fully capable camera ok so I wanted to keep this talk really high-level because to be honest with you the code examples and the process of starting a preview and capturing an image with camera 2 is horribly complex so there are two examples I'm sorry I'm scaring everybody off tonight there's two examples on the Google samples github that are actually you know pre decent and basically give you a basic application that you can see how to actually use the camera - yeah the first one is the camera - basic sample and it's pretty simple you know you look for the cameras and device to pick one out you open it up start the preview take the pictures with it and say some pictures to file there's another example that actually shows you wrong so as you've been waiting for a rossport that's the example project for you it does wrong but it also does some crazy things like trying to handle oh really kind of close close together request and also it does actually a lot better job at concurrency so a big thing with camera two is the idea that you know where we have more advanced system is pipeline its performance requests to the camera should be non blocking so a lot of the camera - Tanika calls have hooks into the art parameters rather where you can put in a handler to a background thread and run a lot of these hardware grabbing hardware manipulating methods in the background and that's great you should do it but that also means you have to start paying a lot more attention to concurrency you know the camera API was really simple there wasn't a lot you know there wasn't any kind of built-in method for you know putting stuff in the background now you have it you should use it but now that we have to think more about concurrency and camera to Roth to be honest with you I think is a lot better example and has a lot more I guess better use of concurrency techniques with the new camera to API so definitely thing to look at I would go with camera basic - first just so you don't go crazy so you can get a good idea of how it works then you can move on to the actual correct way to do okay so I'm gonna get that I'm just kind of tell you some of the new classes some of the new new best friends and you're working with camera and so you're gonna have the camera manager camera manager system service and his job is pretty much mostly to give you information about what cameras you have and to actually give you possession or give you control of one of these cameras and he'll also provide you with that kamek characteristics or that class that have all that detailed information about your camera as capabilities that's what he's for so and again you'll have the camera characteristics class that has all that cool information and then you have a new camera device class there's camera device class it's basically your handle onto a device camera and you're gonna do a lot of work with him you're going to set up capture request with him and you're going to set up this new thing called camera captures now if you've worked with camera one at all you know that normally what you do is just open up a camera and you just do stuff to it you know camera and take a picture camera autofocus you know everything is done through that kind of handle camera handle class now that there's tons now there's kind of an intermediary guy there's this camera and capture session now you know before without you know I mentioned before with how three you're having you have the ability to with each request send that image data out to multiple outputs now setting up the output conserving them to receive that image data takes a bit of work so it's actually the camera capture sessions job to take some pre-configured outputs services you know like imagereader meeting to add or whatever and actually associate them with a particular camera device and get those guys set up to work together you can't take pictures without this guy and again he's kind of a new kind of middleman in the process but basically you he'll he'll again associate surfaces with him device and he's actually the gateway for requests so while you may actually make a request you know in conjunction with camera device the camera capture session is the actual guy who will take that request and send it off to the camera okay so again metadata so there's a camera metadata class and it has a lot of information in it the camera characteristics are kind of done in like a map store you have a key for each piece of information and a particular no value some type that comes out a camera to metadata has a lot of that basic information about what keys and will keep out of use you have there so i missions before but the new camera request and capture result objects are actually subclasses of the camera metadata and again that's going along with the same idea that and we always want to know you know what we're requesting and what we're getting back so camera requests are tied very particularly to a camera device and when you have your camera capture session right so he's holding on to all the possible places that your image data could go so what happens is when you make her a camera request you have to say all right I'm working with that particular camera device and I want that particular service that I talked about earlier those two are going to go together to the camera so again everything is really tightly coupled you have to have a very specific set of surfaces you have to have like you know let the camera kind of hook up to them through the camera session and you have to make sure that you only pick out one of those guys to send into the request if you don't bad stuff happens usually it's just not a picture showing up but you know bad stuff can happen and those are the classes that you'll be working with the camera to so I'm gonna give you a really quick outline of what a application might look like in cameras acting is pretty similar to camera one so the first thing you have to do is find the cameras on the device and you'll use for that you pick up the camera and you'll open it and so you'll get that camera device guy that you know handle are into that camera and generally what you're gonna do separately is you know figure out what outputs you want you know imagereader renderscript allocation whatever you're gonna set those guys up and you're gonna pass both the camera device and those outputs to a camera capture session once that guy's open you can start making requests and sending them into the session and getting results back and that's pretty much it if you want to continue making capture requests you just keep making new ones sending them to the camera session if you switch cameras because all this middle guy stuff because the camera capture session because the capture requests are very specific to a camera device if you switch cameras you kinda have to go all the way back to step two step three and reset all of that up now the caveat here is that setting up the camera capture session takes relatively a long time so you should be judicious about doing it and not you know suddenly you flip back between this camera and that so just be cognizant of that okay this is great believe it or not it's a great diagram basically showing you all that stuff I talked about and relating in the new camera API with you know the workings of the camera device here I will not put you to the torture of pointing out every single little piece of detail on this diagram and explaining it but if you're interested it isn't actually in the documentation for the hardware abstraction layer again you think this kind of let's see you know all the possible destinations you have that you can image data to you have the list of the different a method calls and you know where they go and where they come from caveat though that this is an old diagram so some of these methods have changed although the soul remains the same but yeah that's basically again you know we've kind of gotten away from the safe zone of that black box we've gotten into this mess but I will say that it is more powerful and it is more flexible and it is you know for the time being the future of the camera so if you're sitting camera this is what your life is gonna look like it's pretty though you know all the colors and stuff online so you know but it is powerful and it is okay so please don't read every single word in this life what I want to do is kind of compare to you and again reinforce the fact that you know the camera one was really user friendly it's a point-and-shoot camera you just pick it up and go camera two takes a bit of work now on the left here I actually have you know different operations you might do with a camera and on the first column I have the camera wasn't calls that you generally would make to do these things and on the right is the camera - Ethan I calls you a dude for the same tasks so I mean they're pretty comparable opening the cameras not too bad starting at 7:00 mostly involved I'm taking care of that surface view whatever in your hierarchy that's actually rendering that preview now actually start in the preview so in camera one you had it really easy you have a surface holder which is basically a handle to some surface that's rendering on your on your UI and you give that to the camera instead here here's where you gonna draw the preview and then you say okay camera start the preview and then fit to method calls use camera - you have to start a capture session you have to create a capture request you have to specify the target for that request you know whatever surface that is you have to set up the preview request and you know there is a certain value that you have to give it because the preview is usually kind of more low res video than like capture there's certain senators you want to do for autofocus and auto exposure so that you know the preview is usable and then you actually have to take all that and wrap it up into a request then you have to actually say instead of just saying start preview because you're asking for a continuous request is that the image data instead of just saying okay start preview you have to say SEC repeating request so you have to know that you're not just asking for a single frame of data you're asking for a continuous stream of data and again it's not just that it's more verbose but there's a whole bunch of other things to worry about all these little methods I kind of March with the star and the cross these are all methods that are both asynchronous but that also have the ability to be back rounded or rather they just have a parameter we can see above and/or to fund that graphic so you know on this side everything is pretty much either synchronous or does it really straightforward a callback on this side it's a lot more work it's more it's more efficient it's we do a lot more control but you really have to know the very specific set of C's are the specific sequence of events that you need to make happen for you that's pre me to actually show up okay so I'm gonna talk a little more high-level about what we'd be doing if you are actually you know starting your camera application so starting a camera application are starting up at preview or capturing an image in your camera application with Android was never really a linear process there was a lot of things that danced together that you have to coordinate so from a high level if you're just setting up the camera and trying to get you know the cream showing up on your screen there's a bunch of things that you kind of do at the same time and then kind of coordinate into you know like a the final reveal of the preview on the screen so usually what you'll have is you'll have some view in your view or some of you in your hierarchy that is meant to be the place where the preview shows up and you have to actually wait for that view to generate that surface where the image data actually gets rendered and that's going to be a names you're gonna call back so you have to kind of put it in your layout and make sure that you wait until the service is done before you do anything with it otherwise that makes happen so on the camera side you will be wanting to figure out what cameras you have and actually you know ask the camera manager for information on that camera and at some point you're going to want to open that camera to get that camera device which is you know your handle onto that camera and that is also an asynchronous call so you have to wait for that's open at the same time you're going to want to get more information out of camera manager and characteristics so we certainly do you can't just say oh here's like a 1800 by something-something surface go ahead and go crazy and draw there's actually very specific preview sizes that you have to set on that surface in order for that to render correctly so you don't just have to create surface and wait for it to be done you have to grab out the camera info figure out what size is correct or allowed and then basically you configure the surface and that's at five and once you have that down your services aren't figured then you have to wait for the camera to open and be ready and then once you have all those things together then you can edit out a camera capture session and then start the preview so it's a dance basically there's a lot of things that could be happening at once so you have to make sure that you finish all the configuration and bring everything together at the right time and this was kind of true in camera one camera one was more kind of like the right half of the screen and camera two is kind of added more of like the left half of the screen all right so actually taking a picture so camera one autofocus take pictures they forget you to come back camera two sorry so first so basically what happens is is that instead of having everything in that black box of taking care of you you have to know it exactly what sequence the triggered things and for things to happen so before you just say it's alright camera autofocus for me now you actually have to kind of hook into that pipeline to send out triggers and say hey trigger the autofocus and you have to actually wait for that autofocus to lock and unfortunately you don't exactly have a really straightforward callback cookiehead number one what you do is you get this result this kind of result receives type callback and what you do is actually check the states of the autofocus algorithm and say hey how to focus are you locks you're like not yet all right go I'm gonna wait for the next result and usually what happens is when you do a camera request you won't just get back one result when it's time you'll get back a lot of in progress results so what you have to do is keep checking in say hey you know I don't think those are you done yet okay cool your lot what next alright so usually when you take a picture you do auto focus and you also have auto exposure now auto exposure in camera one it's basically kind of up in that black box you know you don't have to worry about it just turn it on or turn it off for camera two you actually need to trigger what is called the pre capture now when a camera goes through its auto exposure algorithm it actually has to do what's called metering so that's kind of like reading with light nose and determine you know like the good values to get in a good picture out of you know those studies so what's going to happen is for you to take a picture you actually need to say hey any camera can you start that pre capture process where you do all that cool fancy reading the light stuff and it's going to do that and basically what's gonna happen is you're gonna wait for in some kind of in progress results so hey has the exposure settings converged and that's actually the name of the state is converged and you might say no it also workings like okay okay next capture result are you done yet no okay okay all right how about now oh yeah I'm converged right so once you have the auto exposure converge then you can actually go and start a different request for actual still capture and you create that request you know set a bunch of features on it you know oh I wanted to sepia tone and I want you to you know have this white balance and I want this and make sure you get some face detection in there too so once you get all those set up you create the request you send it in to the capture interception and you wait for that capture quick request to complete but you are not done because in the process of actually a setting sending that capture request to the session you had to stop the previous preview know that repeating wants for it to feed you you actually had to call stop at some point I think it slows there and now I bet you're done you have to start it up again so you're gonna send another request where you basically reset all the values in the preview send it to the camera session and it'll restart the preview for you and you're ready for round two or you're ready for a nap so but that's basically and you can as you can see a lot more powerful and you know if you think about that key is pretty cool that you can reach into the camera pipeline and mess with the auto exposure and messily auto focus and you know kind of you know be able to have a hand in that it's a lot harder between one and two but again the trade-off is you get a lot more power a lot more control and be able to do cool things if you know what you're doing alright just really quick a few caveats so some guarantee caveats and this is kind of this number one rule it was there in camera one this year for camera two is that basically whenever you'd grab a camera device you have control of that camera resource no one else can get ahold of it so please be polite and only pick up when you need it and let go of it immediately when you anymore if you monopolize the camera you could crash other apps and that's just rude so again please you know again only require camera boosters when you need them and the best place to do that is basically just pick it up in on resume and then drop it or release it on pause and also when you're switching cameras please try not to pick up you know all the camera resources at once if you're switching from the sit in front to the back camera just close whatever one can up using another caveat is I kind of mentioned it before as well it's called surface dance the setting up a surface is really an independent kind of process from the actual camera so just make sure that you are aware of that aware of you know what state surface is in that you're rendering to and make sure that you know it's done before you actually start trying to mix with it again bad things will happen and also just kind of a mistake is easy to make is that you kind of set things up but you might have an invalid value in there a lot of times when you give to the preview of size another one it'll you know just fail or just not show anything so you have to be make sure that whenever you render a surface you check Oh what kind of sizes are okay for you so you just make sure you have to check those a lot of that information isn't this wonderfully named p-value entry called the scalar stream configuration map before you just called get preview output side of that uh okay let that make sense now you actually have to read the documentation and figure out what all that stuff means so that's a little hints tonight it took me a little while to figure that one out and Cameron's you comment about concurrency and as you can tell I love alliteration so camber to have three so a lot of the points this whole reworking is performance and again like as we saw earlier and as you will see as you work with camera - there's a lot of focus on asynchronous work you know push callbacks everywhere and again you will have these parameters where you can feed it to handler to a background thread so you can do all this kind of long-running you know how you know how were the camera operations on a background thread but you know when you do that kind of stuff yet be careful you want to make sure that the camera operations say is non-blocking as possible just for performance and so that you can handle multiple requests coming in but that means that you'll have to worry about concurrency so make sure that if you're accessing variables and like one callback they might get a background thread but you're doing something else in the you I could make sure that you're using synchronization make sure that you're using locking methods and this is actually really important with the open and closing of the camera and it's better if you just go to these two examples that I mentioned camera two basic in Camera Raw and it shows you how to actually lock the opening and closing at the camera method so you don't just exit the application and still have hole in the camera so again camera to rob check that out it's a little bit of a brain twister but it's suddenly worth looking at and studying for the right way - you know chemical currency with camera - all right we made it guys so I know I probably frightened the crap out of you who is known about camera two and you know it's nice to kind of imagine back for the days of camera one where things are really simple and the method calls are in plain English and you didn't have to worry about concurrency but you know we're moving into kind of this phase where you know putting photography is such a huge part of the mobile experience you know people are using their phones as their actual cameras and you know kind of what comes with that is that you'll probably have users that want to do advanced features don't want to do you know kind of more real photography and that's what camera two is going to be able to give you the tools to accomplish so you know and it's here to stay everyone's getting deprecated so you'll have to deal with it room with camera but overall it is a better system it's a more complicated system but it's a more powerful one so if you're just in camera don't be afraid check it out and thanks guys for coming in using like our X Java or do my bags to tell it like alright ready okay done with this now let go the camera I got the preview where I finished saying the picture now I can do an event called a starting preview again stuff like that I have thought about that I have not done it myself but I think it's worth looking at considering all data I think it's something worth looking at yeah but no I haven't try to get no it's a surface view that's already I'm sorry it's already and then absolutely yeah there's actually a classical text review which is actually does OpenGL commission rendering and the the underlying surface is actually the same thing it's just absolutely actually if you want to do cool things so the thing with the surface view is that it's in a sense it's actually here to doing more it's near to be more performant where a surface or texture new has all the OpenGL you can transform it you can do animations on it so it's more powerful but less performance so if you're interested in doing that stuff of course surface because more if you aren't looking to do crazy things with the image you just wanted to be as efficient as possible but yeah it's the same underlying structure in there yeah just trying to get stuff away

Channel: GDG Boulder

Views: 24,901

Rating: 4.8270893 out of 5

Keywords: Android (Operating System)

Id: KhqGphh6KPE

Length: 47min 28sec (2848 seconds)

Published: Sat Jun 27 2015