Building Non-Game Software in Unity - Jon Manning

[Music] cool welcome this is a chic way bigger tenant than I expected fantastic alright so let's go straight to the good stuff I'm going to show you how you can use unity to build non-game software and I'm going to show you how you can do it in a way that feels good to use not necessarily native because if you're aiming for native stuff then you aren't using unity but you can get saying that it still feels like it's a good piece of software on the device you can also make it significantly reduce the amount of power being consumed on the device as well and save the users battery all right so I'm John Manning I am 50 percent of a studio called secret lab where based down in Tasmania I am a programmer on night in the woods people like that a bit that's great and also I've written a bunch of books on unity I also work on marble create-your-own which is really cool it's from technic comics and also I'm going to show you some good stuff so the main question why would you want to use unity to make a thing that isn't a game well there's a few reasons probably the reason why is that you have your building some software that has some combination of these requirements it needs to do something with graphics unity is famously pretty ok at graphics it also would ideally need to be cross-platform so unity again it's quite famous for being a cross-platform game engine and so we wanted to leverage that we want to be able to write software that we can fulfill the dream of right once deploy anywhere maybe you have some c-sharp code and it's some third-party library or internal library that you only available in c-sharp or in a dollar language and you want to deploy that on a variety of devices in an easy way unity can be used to make this a little bit easier and also you don't have to learn a whole new platform now I I kid it's often good to actually learn the platforms he's building for but you know sometimes you're a bit of a leg up so today I'm gonna be talking about the use of unity as a standalone platform this we largely mobile focused I'll and my primary background is as an iOS developer but we'll be talking about Android as so this will be a largely technical talk I'm not gonna say why it's a good idea you're in the room already you know that it's a good idea I'm gonna talk about how you can achieve certain technical goals to make things better for your users when you're building your your non-game software in unity so the background to all of this is we were approached by my tablet comics who wanted to build Marvel create your own which is an iPad application this is a application which you design your own comic books with Marvel characters and it's great it's got all like every County you've ever ever thought about largely it involves lots and lots of manipulation of sprites it needs also to be cross-platform and so unity seemed to be a really really good option for this so just to show you what this looks like oops so things like as you you're moving around you're putting things on the screen you're like dragging and dropping this is still using unity and still using a lot of the kinds of techniques that unity would be able to accommodate but you'll notice that we aren't like having to do complete screen redraws we're like dragging and dropping and then nothing happens for a moment after that and so there's a lot of opportunities for saving power here while still getting all the benefits of using the full game engine so I got three things to talk to you about today first of all we're gonna talk about how you can interoperate with native code some of this might be familiar to you already but also I want to go into a bit of a deeper dive than you might have seen before I'm also gonna talk about briefly how you can do native style input in a way that is better for the user gives you things like like the selection box in order correct and other stuff like that as well as how to save a frankly ridiculous amount of power by turning off every feature of unity that isn't related to your game so with all that let's get started let's talk about how we can interoperate with native code now by native code I'm referring to one of two things if you're on iOS this means the actually native that is C Objective C or e maybe even switch code on Android this means looking a bit different if native code can mean either working in Java or working with the low-level C++ like J&I stuff like actual native not just the layer that exists on Java now today I'm gonna mean most of my Java and I'm also gonna be talking about C and Objective C so a lot of stuff is only available when you look at it via native code that is any system provided UI so the photo picker email systems share sheets if you don't do a platform native alert dialog and also any third party SDK the amount of god that doesn't provide a unity integration many of them do but it's not inconceivable that you have got one that doesn't so we have two tasks we need to be able to call the native code from c-sharp and we need to be able to call the c shout code from native now this can get complicated because of the relationship between the dotnet runtime and the native code on which that runtime exists we have different kinds of memory management we are passing different types around we have to do marshalling we have do all kinds of stuff to bridge these two universes so let's take a look at the iOS case first of all here we have a function written in C and it takes two parameters an integer and a string and it returns an integer so maybe this code interacted with other native API s to access this C function from your c-sharp code you first of all have to import and make available that C function in your C sub code first of all you have to use the system's runtime dot Interop services assembly and then use the dll import attribute and declare an extern function that means that you're declaring to the compiler that this function exists but it is not defined inside C sharp or inside your assemblies instead is available via the native code the DLL import attribute tells the compiler or rather tells your code at runtime where to find it the underscore underscore internal says don't try to load an external dynamic library instead you'll find this symbol inside the binary itself once you've declared that you can call it like any other method on your class now it's useful to wrap your externalization in an if def so that it only gets compiled when you actually building the player that's because the the native code doesn't exist unless you're in the player it which means that one exists in the editor so often it's useful to stub out a an empty method so that your code will still run in the editor so we've declared our our CC function and so we can call it and we can pass in the same kinds of types that we would normally like to integers strings we can get results back strings are automatically converted into pointers to two characters fit for C and then reversed if you are returning a string on the heap sorry if you're returning a string then you should put on the heap and then return it to to c-sharp it will then free it later on working with Android is a little bit different so here's a here's a function rather a method on a Java class so because we're working in Java we have access to all the stuff that's available via Android the Android SDK so how do we call this when we don't have access to the actual low low level functionality in this case well what we do is we use the Android Java object class in c-sharp this represents a Java object you can instantiate you just provide the fully qualified name of that type so in this case I've got my package then calmed up my company doc my awesome game useful task performer and then on that object here you can then call the method that you'll know provide by by name so in this case that do useful tasks and it works in a very similar way so that's how it can call functions in the native land from from c-sharp but how do we go backwards how do we get our native code to call back into unity well there are two ways one is very simple the other is deep in forbidden magic the first the simple one is the function unity send message so who here uses send message for write the C sharp code not very many of you and that's a good because season yeah it's some sane message has a number performance implications but it actually turns out to be the simplest possible way to send information back from native code into the unity environment so the other way we can do it is as I mentioned the deep forbidden magic of we get you past a function pointer to a c-sharp a delegate and call that from native code we can also use in Android the Android Java proxy class to create objects that conform to a Java interface and then use that for callbacks as well so let's look at these one by one so first of all unity send message takes three parameters this and it has a very identical API between between iOS and Android so I'm only going to show one example unity send message takes as its first parameter the name of a game object in the scene or in the set of currently open scenes the name of a method on any mana behavior on that object and then optionally a parameter and so unity will then find the object in the scene or scenes find a method that matches the name that you provide and then call it and pass whatever string you provide so simple straightforward but it has a number of limitations first of all it requires an object in the scene that you know the name of and that can be it's not the worst thing in the world but it's a camila but tedious it performs reflection to find the method that's implemented i it's doing the whole thing based on strings and string lookups that's brittle it can only save one parameter and that parameter must be a string and also there's a guaranteed delay of one frame so when you call this then it's going to happen the next frame and that might not suit your needs so what else we got well we can do it the hard way now as unity developers we work in net and dotnet likes to pretend that pointers don't really exist as a thing like if you go looking for them you can find them but but it likes to pretend that that pointers don't really exist but they totally exist and we can get a pointer to a c-sharp method we can then marshal that and deliver that to our native code and then the native code they call that so we are truly living on the edge here today so let's first of all in our native code declare a type of function that we expect to call this is going to be a function pointer that we receive from dotnet and so we in this case in C we're declaring a type def we're saying there is a type called a useful task complete callback and it returns void and it takes a string as a parameter and then we'll have a function in our C code that the c-sharp will call and so I'm calling it do something useful and it takes as its prouder that callback basically this is a delegate and then we do some work and then when we're done we can then call back that function now we don't have to call back immediately we can preserve that pointer and call back in a little while so this is good for asynchronous behavior as well so we've now written some code that declares what kind of c-sharp method we want to receive a pointer to and then when we're ready we'll call that function so how do we deliver the pointer to that function into our native code from c-sharp like this first of all we do a very similar thing we declare the kind of function that we're gonna be sending into native code so we declare a delegate type in this case I'd call it useful task complete delegate and I've again you'll notice it's the same type signature as before it returns void and it takes a single string parameter next I'll declare a method on myclass and I'll use the mono P invoke callback attribute this marks to the compiler that this is a function that I'm going to be calling from native code I declare that this is the kind of method signature that I declared the the delegate type auth and also importantly this must be a static method other than that very few limitations apply um this is just a function that I can call from native so this is gonna be my callback so oops I've got the wrong code here um but basically the way that it works here is that you need to basically call that function by you would call your function here do something useful and you just pass in the method group itself you don't have to pass in you don't have to do any complicated reflection API usage you can just say okay do something useful and pass in the method that you've set up you fasten the this useful tasks complete method and then when it's ready it can then call back that see shop function and then you've got your callback so this function point can also be preserved because it's a static method and it isn't bound to an instance you can preserve that pointer and you can call back later on so the reason why you might want to do this is imagine a download that it might take multiple seconds to run and then when that downloads complete then you can then call back into entity cab and then have that run exactly when you wanted to all right so what about Android we can't send function pointers into native code when we working with Java how do we deal with that well we can use Android Java proxy Android Java proxy allows you to define a interface in Java and then create a c-sharp class that implements that Java interface you can then pass in that object that implements the Java input interface as the callback so here's my Java interface I've got some useful tasks on a complete and a method that I want to run when it completes successfully so here's my interface then in c-sharp I create a class that subclasses Android Java proxy and in the constructor I invoke its super constructor passing in as its parameter the name of the Java interface that I want to to pretend to conform to I then declare all the methods that would exist on the Java implementation of this class so in this case I've got you know my completion method here then when I receive what I want to run my native code I receive as my parameter some object that conforms to that interface here useful tasks callback I do my work and then I call a method on that object only know that it has that method on success to work well could this I can simply create a object in Java I'm using the exact same technique as I showed before and pass in my my object that can implement the Java interface and I'm able to receive back information back from from Android so that's how you can both send information into si in Internet of code and how to send information back to c-sharp this is how you get all the good stuff this is how you get access to native api's that aren't necessarily exposed as part of an official API by unity now this allows us to leave the domain of unity and do some really interesting stuff with stuff that is not present entirely than the bounds of the render view of unity and this includes native text import so the text in for community is pretty good for PC but it has some limitations on mobile that includes things that it was never designed for for built-in access to you autocomplete it doesn't have access to the selection in system there are ways that you can make this better but well let's talk about ways that we could make this into a better user experience what we could do is we could render all of our text in unity as we are right now write a native some UI view subclass that supports on iOS the UI text input interface that protocol skip excuse me have unity handle all all the text origin layout and then you knew that within provide information about the location of that on screen so you can actually create your own complete custom implementations of iOS provided views like UI text view and UI text field and that and all those things or you can cheat and you can overlay an invisible of UI text field on top of the unity render view position that exactly in the same place as your text field let by OS handle all the Tex display and input stuff and all you have to do is every frame just make sure that they put the position of your text field is on top of your agenda view so this is much less work for you as a developer the downside is that unity doesn't render over text and so you might see some very slight font differences the trade are usually pretty fine um this is actually quite complicated to do and I recommend you check out Eagle opportunity mobile input its MIT license it's free I'll have links at the very end of this presentation that you can take a photo of it all the same time so might save a bit of time there there's also books in Android as well by overlaying a text view on top of the activity so I'd strongly recommend that you make use of this because having shipped projects that you just use the built in unity text field like the UI team are you need an amazing job but text input is hard and you know the focus is on being a game engine rather than being a word processor all right the next thing that I'll talk about here is something that is a thing that most people come across when they go I shall make it an application community its cross-platform oh wait I've got a problem and the problem is hopefully this will start yeah the problem is let's build and launch an application that with an empty scene all right so here's my empty scene and doing nothing at all it's running at about 20% of CPU that's nothing imagine if I was ducky doing work so the empty scene consumed a lot of power the reason why is that unity is trying to redraw at minimum 30 frames a second that's the default in iOS now if you're making an application that doesn't necessarily need to redraw all the time how can you reduce that well one way would be to find ways to reduce the amount of work that the CPU needs to do the very best optimization that you can do is do less work okay and there's a lot of wasted work happening here even though nothing was changing on screen we were redrawing the entire screen every time we have redrawing that beautiful gradient that we've all come to love and enjoy so a non game application does not need to redraw its screen every frame it just need to redraw when something has changed the section where the unity editor does unity editor is largely powered by unity and it will only redraw its window when something happens like a UI event so how can we make unity not drawl well the simple thing is to reduce the frame rate most people come across this advice when they go how can I make a low-power application and this is a really good advice it does reduce the amount of power drawn by your application drop into about 10 to 15 frames a second is a really significant power saving but it does have a downside because the frame rate is being being reduced the input the opportunity for your if your code to run is also dropped as well so now the games can be lagging permanently now there's also a less obvious downside as well and that is it takes longer for your code to receive and process input so now it might take about 100 milliseconds for a tap or drag to be noticed and that's not a good user experience how can we get the best of both worlds how can we reduce the amount of frames that we draw while at the same time being super responsive and having the opportunity to respond to user input well let's first of all look at the first issue here let's reduce number of frames or drawing let's disable the renderer completely you nee want render the scene if our no camera is active so congratulations we've just shut down the entire reason why this is a whole piece of software in the first place and that's a good power saving so the way that we can do that is we can just say camera dot enable the equals false now if you do that an important thing to know is that at the beginning of the frame unity will that the camera will have already cleared so if you just say camera is off then you'll see a nice clean black screen with nothing in it and typically users like to be able to see the software that they're working with so what we can do is immediately after the same the camera then manually render a frame right away that has a result of filling the color buffer and it'll stay on screen if you do something that I'm about to show you because this will still result in a black screen we've told to re render but it still goes black and that's because at the start of the render loop all cameras even if the entire color buffer is discarded largely so the claim the screen happens is part of the play loop can we turn off that part of the play loop we can we can modify the play loop using the low level player loop API but that also makes you think well if we can disable that part of the system could we not also disable other parts of unity as well we are still running physics we're running audio updates are running other scripts I don't need to run my update function every frame because it's not gonna do any work unless the user is doing some input could we just disable the entire play loop yep so by using the player loop class we can access low-level features of other player we can inspect it we can see what's going on inside of them did you know that there's a key at tango function that runs every frame it doesn't do very much but it is there and we can also insert new systems of our own or we could just erase the whole thing so here's the code that disables all of unity so we use the Unity engine experimental not low-level namespace we asked the player loop class give us the default player loop this is a reference to the the set of functions that gets called every frame we then say that there are no subsystems in the player loop and then we just apply that so this is what that looks like so let's launch the application click yep cool so we launch it and observe my CPU usage it starts out high as we launch and then after a moment we stop rendering frames we stop processing any frames and Richard up to three percent of CPU down from 20% so this is an enormous power saving in fact I tested this with some other other diagnostic tools and what I found was that the system reported that its biggest power drain the thing drawing the most power was the fact that the screen was on so that's an enormous power saving okay so let's solve that first problem how do we reduce number of frames while also reducing all the non obvious non visible components as well like the update loop but we still have another problem we've shut down the entire player loop our code is now not running every ane frame we can't respond to touches how can we still respond to touches so our solution to this we can detect touches at the low level we can detect system UI events and use that as an opportunity to wake the play loop back up for a period of time perform whatever we need to and then when we're done go back to low-power mode after a few frames this is actually how iOS stuff in the background as well and I assume that low-level Android does it in a similar way in that the core animation system is only going to re-render the screen if there is input happening or if some other part of the system has requested that the system's stay on so for example if there's a spinner on screen so how can we access low-level touch information we don't want to do something ridiculous and potentially brittle like replacing the view controller that rent that the hosts the the metal view instead what we can do is something simpler and that is subclass UI application class we can then override the send event method this is called on any event that comes in from the user interface that includes things like touches but it also means things like like the shake gestures and things of that when we get any kind of event coming in we can then call a wakeup function which you received from C sharp we will receive that as a function pointer and kept that locally cached in memory and then call that function when it's time to wake up and then that will then reset our our player loop will go back to up to full 60 frames a second for a brief period of time run whatever code we need to and then after a bit of delay will then drop back down to 10 frames a second kill the player loop and return back to low power so to keep a function pointer around it looks something like this I've got a function pointer that returns void and takes no parameters and I'll also declare a static variable of that type which and so by making it static is just globally it's this global state I'll then have a function called attach event hook which takes as its parameter the the c-sharp function that were going to call to wake the system back up and all that does is just keep it in memory to override the send event method you create a new subclass of the UI application class this is not the same thing as a UI application delegate if you're familiar with that in iOS programming the UI application class is actually the the class that represents the application in its entirety and your Apple says you can subclass this but if you subclass the only thing that you should do is override send event don't do anything else because that's really dangerous in send event we'll simply check to see have we received a wake-up delegate if we have then we'll call it then and then very very importantly we call super which basically calls the superclass implementation of send event if you don't do this then no touch events get deliver to the rest application how do we use our custom application this subclass of you on application in the main EM file which is part of the unity iOS trampoline there is an invocation of the function UI application main u application main is the function that all iOS applications have and this begins the application is launched the application that creates the application delegate does all the work that the application does in fact this function when you call it it never returns now the first two parameters are the arguments that the the process received from the system but the third argument which is typically nil you can replace with a string that contains the name of the class you want to use as your custom UI application class so by replacing that with our own in my case custom application we're now saying create an instance of the custom application class and use that for all events delegate a delegation and then when that function gets called this is back in c-sharp we tell the player loop restore the default player loop let's put everything back how it was so this means that we get all of but we get we get the best of both worlds we get lower power consumption when the user is not interacting with the application but we get the full responsiveness and interactivity that we want when the user is interacting so we can jump from 60 frames a second the moment that the finger lands on the screen Enter that his video so a launch application and as I'm touching it works when I lifted it stops so that counter there is a number of frames that have been drawn just in Korean that once per frame you can see exactly how quickly the application comes right back up to the normal frame rate cool so I've given a very deep dive in how you do this on iOS how would you do this on other platforms the only thing that you need to do that is platform specific is the ability to hook touch events at the low level on iOS you've just seen how to do that I'm not gonna go into it as a deeper dive on other platforms primarily because of my background as an iOS engineer but other platforms are possible we can do this on iOS because we have access to the source code of the trampoline that is the native code that launches unity I believe it's been announced that Mac OS will also get the ability to modify the Xcode project as well I think if not unity please don't get mad this is probably also doable on android as well you need to add your own custom touch event listener on the activity on Windows you could do something like hook the keyboard and mouse events and use that to wake up the application I'm less certain about other platforms but basically this technique is fairly broadly applicable so I've given a bit of a main focus on non game stuff you can also use in not not games that is games in situations where you just want to reduce the power consumption of your game so in pause menus in Island timeouts where the players stop using it for a bit they put the phone down or games in which we have low degrees of interactivity so turn-based strategy games you could just freeze the entire system and only wake up when the user wants to interact or lower speed games like puzzlers this would mean that you couldn't draw things like background animations but if you design your art carefully then you can do it in a way that makes the application not obvious that it's stopping the entire thing you also could disable some of the render loop but not all of it so we took the thermonuclear option here we just blew the entire plane but you don't have to do that you can just replace some of it you could disabled parts of the render loop like the physic system like all but a few important scripts that need to run every frame there has been some discussion from unity about adding support for a native API for controlling play look like this this has been discussed on and off in the forum's but it's been fallen it seems to fall away but it's not really a mess of priority it'd be really cool does this happen if you want this kind of thing show support for it on the forums please be nice about it unity people work very hard and have a lot of people you only get them asking for their therapy chip so as you make systems like this the thing that you have to keep in mind is the player should not realize that you're doing anything like this at all ideally this means that the players have not even realized that this is a game engine that this is unity itself one last thing I couldn't really fit into any of the other parts of this talk is use the platform native font it's actually way easier than you might think Unity's default font is Ariel but you can actually use the font create dynamic font from OS font function that allows you to provide a font by name and then it'll generate a font dynamically that you can then use in your in your text objects and in your input field objects and that will just work like it'll instantly make your application feel like it fits way better because fonts matter it makes application feel like it fits finally one last thing that is absolutely wild is a developer named Joe Jackson Dunston hasn't been doing some work on what if you could do all your game scripting in C++ and so he can put together this this amazing series of articles on how can we write as little C sharp as we have to could we not just write everything in C++ and he's got support for saying things like you can hot reload your C++ code while the game is running you can bridge between the different c-sharp and hast types it's amazing go check out it does not recommend a best practice but if you really really want to see how far you can take this native in interoperation then please check it out it's so cool alright so i'm on twitter my handle is displaced oh my studio is the secret lab and we're on twitter this is the slide take photos of if you want the pictures of stuff that i'm doing one thing that is at the bottom of this slide is there's an article on how you can hook the keyboard and mouse on windows if you're looking at that kind of thing on that platform i think we have a few minutes for questions thank you very much all right any questions yes yeah yeah yeah so you you you want to you want to stop the whole play loop but still occasionally run some important stuff yeah so what you do is you would blow away the entire play loop but it's but add back one system that that runs as part of it and then you have a lot more control in there because you want updating every object in the scene you could just insert like a single function that gets run every frame and then you say okay every five frames do my work so you you can add you can add new components into that loop so you could do it as a carotene like you can bezier on any object see chef code what I'm basically saying is you can remove most of what would normally happen but then leave behind something important yeah cool any other questions one of the back yes absolutely here's yeah that's he's that font code again so basically the API is just font create dynamic font from OS font you pass in an array of font names it will find the first available one in that list and then and then make that available and you can use it just like any other font I so the question was if I ever heard you right um said in the window sighs on Mac OS is currently broken don't know my focus has been on iOS where everything always fullscreen forever so sorry cool you know the questions one down here going to give them a key yeah I saw the recording I don't really know how is it is there some functionality in the net in the unity vertebra a player that is not present in in because as far as I understand and they using NPM if you player in iOS and an equivalent in Android as well as the fact that you can then you can also decode video and stream it to a texture so it wouldn't be the first thing that I'd look look to right away yeah dip I think it would depend a bit more on how you're using video so prayer would chat after the talk yeah cool all right I don't see any more questions thank you all so much for coming [Music]

Channel: Unity-ANZSEA

Views: 9,943

Keywords:

Id: C7CZyqHGKXw

Length: 36min 58sec (2218 seconds)

Published: Sun Jun 23 2019