Building UI for games with the new UI Builder - Unite Copenhagen

hello everyone my name is Damian I'm a developer on UI elements and today I'm going to be talking about building UI for games for runtime using the new UI builder now UI builder is a visual authoring tool that builds you are using UI elements so before I begin with UI builder I just want to give a brief overview of your elements so we're on the same page so your elements is a new although not so new anymore API for unity for creating UI its goal is to create a unified way for designing and creating UI for unity for both the editor and for runtime now there existed and still exist to other api's for creating UI for run for unity we have I'm GUI which is primarily used in the editor and it's sushar based and it's immediate mode and we have you GUI which is used for runtime and it's game object base so it lives in the scene now these two api's have served us and you well over the years obviously most of the content created with unity and most of unity itself is still built using these two api's but their drawbacks have become harder and harder to fix within their frameworks so that's what pushed us to build UI elements those drawbacks are around performance scalability issues and the fact that as a new user to unity you need to learn a lot about unity itself and how it works before you can start building UI so it's good to have some goals on when you start building a new API these were our goals we wanted to create an API that was a framework that was easy to collaborate on especially between designers and developers we're not to make it easy to reuse parts of it elsewhere in your project or in other projects we wanted to reduce iteration speed significantly make it easy to extend existing controls and widgets to your liking this was big familiarity so we wanted to leverage knowledge outside of the unity ecosystem to build something that is familiar to both unity veterans and new users to unity and finally we want to make something performant it's a big one we want your UI to not stand in the way of your actual game right so like I said the UI builder is a visual authoring tool which means we won't be looking at a lot of code today but I wanted to spend just one slide on the UI elements API itself so that we have some idea of what we're abstracting so elements is split in sort of three main parts we have the stylesheet so this defines your Styles it is called USS it is an asset it's very similar to CSS in fact it's a standard CSS subset of CSS it works very simply you have these rules so you have a selector which I is like a search query for elements so looks for elements that match and then you have in the brackets a list of properties that you want to change so in this case we're looking for all the elements that have this container class on them with this dot it's like a tag we're signing font size and then we're looking for this one for text blue we're looking for a specific element that's called random explosion we'll see where that comes in later next part we have the XML so this is the definition for your hierarchy of elements this is also a separate asset so it's XML based you each tag except for the root one is an element so you're creating these elements and you're nesting them together as parent-child relationship so the first element I create here is the visual element this happens to also be the base class for all the elements in your elements it's what you extend to build custom ones and I give it a class container so that I get the font size for tea from the style sheet then we are actually assigning the style sheet to this element and it will apply to all the children as well so my path and then I create one child label with the name random explosion and of course the text that you see in the end result and then finally we have the C sharp API itself this is where all the logic happens in this case we're building an editor window in the editor so we implement the on enable function we start by loading the XML asset so this will also pull in the style sheet because it's referenced and on this asset we call clone tree which will actually go through each tag and just call new on that element with the correct parameters so it just builds the the same hierarchy but in C sharp object land and we get back the root of this tree so with this route we can use the query system that comes with your elements to find our label within the tree using the name random explosion and then once we have a reference to it we can assign some functionality to it so we can say we can register for a callback on Mouse app event so this is completely useless but you can let go of the mouse on this label and this action will run this action itself is also pretty useless but it fits on the slide all I do is I just eat the event I just stopped the propagation nobody else up the tree can have my event it's mine but you can imagine putting anything you want here it's just a way to hook it up and then finally I add the the root of the tree that I got from XML to the root of the editor window which is part of all the editor windows and this is how it gets added to the render queue and starts rendering and starts being displayed and all of this except for my own enable is exactly the same for runtime okay let's go on with the demo quick shout out to Michael Darren a I hope I said it right for creating the image assets for this demo because otherwise you would have been staring at a bunch of boxes that I made press play so we have the hyper familiar tanks project but we added some UI to it we added a main menu press Start we have some questionable UI here that was probably made by the developer just to get the functionality working so I can move my tank as I fire the number of shots goes up so that's linked this is a list of all the other players with the health bars being linked to in if I damage myself there we go and I can increase my speed so you see the speed going up I can cause random explosions and I can go back to the main menu if I caused enough explosions we also have an end screen that gets us back to the so small little game okay now obviously we have a goal for today which is to get rid of that or in this UI and build proper UI for the game and we're gonna use the UI builder to do this so the UI builder is a package it's available in the package manager once you install it you get this big monolithic editor window at least for now will be a monolithic editor window so let's split into these these parts you can drag around so we have up here the Explorer the Explorer is very similar to the hierarchy in unity but it only applies to the UI document your editing and it's split in two so at the top we have a list of all the selectors from the stylesheet so these were these are also using class names so the dot and we can click on one and inspect its styles so using also a dedicated inspector these are all the supported us s properties with some UX of course down here we have the actual hierarchy of the that's coming from the XML so as I'm hovering over these two buttons you can see they're right here oh by the way if I hover over the stylesheet selectors you can also see which elements they're applying to in the sea in the canvas down here we have the library which is similar to the project window but again it's split in two so we have up here we have a list of standard controls that come with unity don't worry about that standard controls that come with unity so things like button scroller toggle etc and the very useful vanilla visual element which we'll use a lot today and down here we have the project section which is a list of all the external files you have in your project so you can use these you can obviously open them for editing but you can also use them just like any other control in the current document sort of like nested prefabs in the middle we have the canvas so this is a edit time only authoring only canvas so what I'm resizing here is not an actual element that exists in the XML document it's just for quickly previewing how your UI will react within different contexts and of course I can click on and on an element here to see its inspector so this is now inspecting a single element so up here we have the all the attributes that are in the external things like name and in the case of a text label the text itself in the middle we have the stylesheet section this is where you can add or remove class names classes and down here we have the sort of a similar view or the same view that we had for the style sheet selectors but these are controlling inline styles that is styles that are store in the XML and they only apply to this one element okay so let's start building some UI I'm just gonna open rough game UI screen somebody did extra work to make this look ugly I think yeah you know what I'm just gonna start fresh that's not really salvageable so we have a blank canvas scary blank canvas let's add a button make it less scary size the button okay gonna save that silence so UI builder right now operates on a pair of your XML on us s this restriction will be lifted in later versions okay we save that to disk and now what we can do is change the UI that we're using in the game so the way UI elements works at runtime is similar to canvases with you GUI but we call it a panel so we have a panel renderer component on just a empty game object and this guy will start rendering your UI to screen it works very similar to the editor window and that it has a root element so with this root element you can attach any visual elements you create yourself in c-sharp or clone tree that we had in the example but you can also in the inspector assign a u XML asset that will be used to pre populate this route so let's change this guy to our new one and moved we have a button okay now we can continue working without going back out of play mode because it's nice to see a preview of what it looks like and we don't have to worry about losing any changes because we are operating on a completely separate asset it's nothing to do with the scene so when I come out of play mode I'm not going to lose anything okay one thing I want to fix is the fact that this is still using the editor thing the editor dark theme so in the Builder you can sort of pretend you're in the light skin if you're building editor extensions you've probably gone through this you have to switch the entire editor to light skin just check it or vice versa so you can do this just for the canvas much faster but also we have a runtime theme which will expand over time right now it's pretty basic but at least now we have sort of one to one okay now I do have a mock-up that I was given of what we want to build so if I click on this hierarchy item here I selected the canvas controls so again this is for authoring time only it doesn't change the document so I can give it a background color an image or if I don't have an image and I just want to I just want some context to my UI I can use just a live camera feed from a camera so this is just showing the camera it's not meant to replace the game screen in any way but it might save you from having to go into play mode each time you want to see you know what the sand look like behind your UI okay going back to this I do have an image and it also is my cheat sheet so that's where we want to build today we have those three buttons that we had super ugly as now with icons and sort of in the middle stuck to the left of the anchor to the left of the screen the stats that we had just pure labels there now in this nice box and we have the player list anchor to the bottom right now let me just reduce the opacity because kind of confusing okay so just why I briefly talked about anchoring it's a pretty important feature so you elements I mean just add a prop here comes by default it will apply flexbox based layouting is it's an automatic layouting system similar to how you would use it on the web and you can see here my button is by default will take up space and it pushes down the visual element down here so you can sort of say no thanks if you want and switch the button to absolute mode and now I can move the button anywhere I want I can size it do this and it's by default sort of anchored to the left and the top so we can change this to say anchor it to left top and right which basically ends working like this so I'm always this much from the left and this much from the right it's hard to interpret it and with this logic we can also anchor it at the bottom okay so now it's anchored at the bottom of the screen so that's okay but in our case we actually want the player list here to always be the same width as the stats box just it looks nicer that way and if I want to do that with absolute mode I would have to manually set those widths and the whole point of view elements is that you should not have to set manual values everywhere anymore so we're gonna try to do the same anchoring but with relative mode that sunset this okay so let's start with this visual element which will now become our screen element I'm gonna tell this guy to crow so flex scroll just means take up all available space within your parent which in our case is though the screen and within this we will first worry about anchoring in terms of horizontal axis so let's add one element our left side and another element to be a right side okay so let's switch the orientation here to row direction so now they are side by side and we're gonna also use justify content space between to kind of get them stuck to the sides of the screen so we're getting close to what we want on the left side we have our toolbar of buttons so let's do one more element for that toolbar and now we can give our button a home so the screen can take up all the space so the button is inside the toolbar which is inside the left column and we need to Center this so again justify content center so we have our center button and because I'm bored I want to play the game a little bit I'm gonna give this guy a name and explosion and now it works so what happened there was my game code detects that the XML has reloaded and it starts looking for elements by name similar to what we had in the code example if you recall with the query you query and if it finds a button with this name it will use register callback to register for click events and start doing stuff so as a UI designer I don't have to really worry about functionality as long as I name my elements correctly they work and then the rest the layout and the styling is completely up to me so let's continue with that let's remove the text and set the background oops color to black now before I move on with the this button I actually want to apply the same styles to all three buttons obviously I don't want to duplicate each setting so let's move the inline styles to the style sheet so we can share them so for that I just need to come up with a new class name toolbar button and use extract inline styles to new class so this did a couple of things it moved all of the overrides that I had in lined to a new rule in a style sheet so you can see the width and height are still set but they're not bold they're not overwritten and it added a class to my button so it has now toolbar button and if we go to the style and the style sheet selector we see we now have the width and height here we can also look at the previews for the XML and USS that's being generated so your builder just generates XML new SS so you can load an XML file you wrote by hand or you can save this and start modifying it by hand it's it's interchangeable so we can see our button here all the way deep in the hierarchy already and we can also see our new style rule with the width and height and the well I'm leaking position let's undo that there so use the well using defaults is always better so that's that so now we are free to duplicate the button or we can also drag in a new button and drag this class on top of the button to apply the style and now if I go and select my rule I can do something like increase the margin and of course it's being applied to all three buttons at the same time okay let's fix this up so let's finish the functionality we have increase speed and back to I knew okay so now they should work of course we don't see much changing but I'm definitely getting faster yeah this should still cause explosions and this will get us back to the main menu cool so we have some additional styles I know that well now we're free to change styles so I know I have a border with which we can set directly with UI elements we don't need a picture for this I can also give it a radius and of course set a color okay let's give these guys icons so this one is the upgrade one and of course the icons I set in line because well they're unique for each button so inline styles have their place okay so that's looking better but now we have the problem that we have no feedback at all when I push this button right it's like I it is working but there's no indication so let's fix that toolbar button we're creating a new style class directly from here but this time we're going to use hover state so this is a separate rule in the USS you can see here we have a separate rule but it only applies on elements with this class if your mouse is on top so because they're both applying we can just worry about what we want to change when we hover which will be the border color so there we go and if I do toolbar active we changed the background color lighter and now if I click you'll see it changes background okay now you don't need to have play mode to visualize these states you can very quickly turn on preview mode in the Builder and now instead of selecting the elements all the events will be sent to the actual preview UI that's running so you can interact with them here okay now the last time I had borders this thick and radius is this high I will see all that so I'm gonna play nice and get the right values so I know they have to be they have to look like this so to remember the numbers I'm gonna use the debugger so your elements has a debugger if you've used that in the editor you you probably know it's it's very useful to find out why the UI is is usually zero pixels when it shouldn't be so I can pick an element let's pick the button I can see all the details here the Styles where the styles are coming from which stylesheet some attributes but what I'm interested in is the actual values so I can see here I have border width of 2 and radius of 4 now you can also use the Builder to sort of figure out how other UI was made like for example the Builder obviously the Builder uses UI elements to build itself otherwise that would be a very bad message so we can see here how this button works here's the background image for it and so forth ok so going back to the game and selecting our shared style let's fix this up to and for okay I guess that looks better yeah I agree that's better okay let's go on with the stats box so the stats box is super easy because somebody already made it for me so I'm just gonna drag that in and okay that was it it's using its you can instantiate XML within XML so it's again like nested prefabs if I make any changes to this XML you'll see the changes in the parent one and of course it has all the correct names for the labels so they're bound so now when I increase my speed you can actually see a number going up okay now for the player list this is made of two elements right we have the header and we have the list itself so let's let's bundle them together so I'm just gonna create another visual element call this player list container and within this I'm gonna create a label and a ListView coupe let's give this label a background color change the text size text color Center it and change this to players and change the font okay now for the ListView let's first give it a height and a width so now you see the ListView is the only one that has it with the rest of the elements the header and the stats box are set to just take up all horizontal space available and so I just have to worry about setting the width on the the ListView and everything else will adjust and match even even after I anchor it to the bottom which we'll be using so I'm selecting a player list container oh sorry the right column and I'm just gonna use again space between but now it's in the vertical orientation so now it's anchored at the bottom bottom right okay let's finish up the styles so this one has a background:transparent similar I want to replicate that so there's a border as well yeah good enough and for the listview let's fill it with some data so again I just need to call it the right thing and we get data this is actually a virtualized list by default so well this view the control is a virtualized which means that it only creates the elements that you see and as you're scrolling it binds the the same elements to new data so even if I had a thousand players I would still only create this many visual elements but the current implementation has a limitation you have to give it a item height which is a hundred for my cheat sheet and so now that looks actually respectable I'm just gonna give this a little bit more width so again everything is adjusting and I know I have to do one more thing which is give these guys some breathing room so creating a new rule I'm gonna drag drag this to the stats box and the container and I'm just gonna give these guys some margin okay I think we're pretty close let's save that and go and open the player list so this is the XML that's being used for each item so it's being in cloned multiple times using clone tree so if I change the icon I think this one looks better yeah everything updates on all the items I'm using the built-in tinting support tinting style from your elements to tint the icons and I'm just gonna change this guy to 100 one hundred and you'll notice the little skull icon is moving aside as you would expect okay save this so finally I'm gonna go a little off script here and change the menu buttons because I saw a style I like so let's open up one of these buttons go to the style sheet rule so we're editing the Styles for all the buttons and I'm just going to remove padding create text size and give it a bigger bottom border just it's like a fake 3d I just like this now you'll notice I'm editing this document but the loose screen which is a different XML file is also changing at the styles this is because they're both using the same USS style sheet so any changes in one will apply to all okay so I think we're good let's go back to main menu and just to see what we've created so we have slightly better my opinion buttons in the menu we have a non terrible UI in I think everyone's opinion which works explosions I can scroll this with my scroll wheel and if I caused enough screen so that's the demo Thank You unity ahh thank you for a few more slides if you don't mind I just want to go over the the values that we had before and see how we addressed each one a little bit so for collaboration this is where style sheets and new XML come in and of course the visual authoring workflow with UI builder so now someone is just designing UI doesn't have to worry about code and they can work in parallel with the programmer as long as they agree on a contract right that's the names of the elements reusability it's very easy to to reuse styles were using style sheets of course and using your XML template so instancing you xml both in code and inside other new xml templates you can reuse hierarchies as well iteration speed so if you're working on just styles and just the hierarchy itself you don't need to compile C sharp anymore they're just standalone assets so that saves a bit of time there you don't have to go into play mode as much because you can operate on the final look and behavior of the UI in an isolated workspace in the Builder and when you go to play mode you can still see your life changes so if you do want to you do need to see it in context especially right now if you want to see it with data with functionality actually working the UI you can do so with runtime running and see the updates live Sensibility again because you SS the way it works you can apply styles on top of other Styles very easily so you can take a standard button and just add a border to it very easily by using the same class you can create your own custom events that you can send through your UI and register against them in c-sharp and also in c-sharp if you want to create something really custom like a custom control like the ListView that's a virtualized list you can do that by just extending the visual event familiarity so this is why we went with something very close to a web like UI framework we want to leverage all the knowledge out there that already exists for web UIs we looked at a lot of design software for you wise and we try to incorporate familiar workflows in the UI builder but we're still operating within unity so we're using the same unity assets the same unity workflows and we were on all the platforms that unity runs on finally for performance this is a big one this is why we went retain mode so we because it's retained mode we can we have a lot more room for optimizations if you're coming from you GUI you don't have to worry about sub canvases and canvases and trying to optimize your batches with with canvases and a blessing this is now our responsibility so you just have to worry about your structure how it makes sense to layout your UI and then will we will optimize that to render only what's needed when it's needed and also we do a blessing ourselves for all the UI images so you don't have to do at listening manually she's gonna end with a quick timeline of you elements so we released for editor extensions in 2019 one so this was a public version sorry a production ready API but it was only for editor extensions in 2019 two we improved performance and we got closer to CSS parity now in 2019 point three which is coming pretty soon we will release the preview package for the runtime that panel renderer component so you can start playing with UI at runtime and also the preview package for the UI builder itself and finally during 2020 we plan to release you elements as a whole for production use so not preview so you can start building UI with you Islands just gonna make it clear though that we're not planning to deprecated or in any way the functionality of I'm GUI or you GUI they will continue to be maintained through this period just that for future work we're investing more and more in UI elements so I actually lied a little bit UI builder is available today in the package manager and it works with 20 19.2 and above so you should you just have to enable preview packages you should be able to try it now you'll just have to wait until 2019 3 to get the runtime part if you want to know more about the API itself so that one slide I had but you want a whole talk about that slide I recommend watching the last unite talk on you elements it goes into pretty big dev depth on that it's also a good comparison if you're coming from I'm GUI it's a like half the talk is about arguing and finally please visit our forum for questions comments we're pretty after active there so yeah thank you there's lots of time for questions thank you so there's mics in the back if you have questions alright hello I can't see anything oh there we go okay thank you yes well this looks amazing so congrats and is there any way to add animation to it yeah so we've we've got in the c-sharp API we do have a twinning support so simple twinning like easy is out we have a library for that right now it's only in the C sharp API so it's not been exposed to us s yet but we're looking at using the same CSS transitions syntax in in us s so that's four simple animations we're also looking at how we're gonna hook up with the standard animation system workflows in unity itself so you'll be able to for more complex things if you want to use timeline you'll be able to hook in into the UI and control elements that way but not in this version okay thank you hello is there anything similar to the event system that currently exists so you can control your interface with a gamepad or stuff like that so if I go back to the so there is a event system it's actually named the same which is a little confusing but this is the event system for you elements and it works the same as the invent system for you GUI you you hook it up to the input system and unity and you can control UI this way - does it have proper support for multiple players because that was lacking in the previous UI system I not searching on that I would ask in the forum we'll find the answer for you okay that would be amazing hey hey first of all I must say it is amazing work it's something that we are waiting for years very seriously my question is simple what is about our biggest pain like showing particles or 3d objects on top of UI right so we've as you can see here there's no object there's no children to this game object that's because your elements runs in its own space and this was a choice we made to really optimize UI and get it as performant as possible the goal is really one draw call for the entire UI so with that of course comes the question of how to play nice with the rest of the game so there's gonna be two workflows you'll be able if you run okay so right now this is only a supporting screen space UI but you can imagine here a drop-down enabling screen overlay so running in front of the camera or world space like health bars floating around so for that workflow you'll be able to what we're looking at the ability to sort of lock a game object or a particle system on top of an element so the element is driving the position of that object but the element itself is still part of the layout system of the rest of the UI so you sort of get the UI driving the positions of everything but you are using actual game objects for just a like a an inventory screen scrolling down the actual inventory items will be game objects or prefabs running in front of the camera so that's one way and then the other way is having UI elements sort of panels on top like being driven by game objects so a health bar like that's floating around people units you can have that sort of floating in space but then within that health bar you have the full UI elements layouting system so you can increase font size and do whatever you need but just in that space so those are the sort of approach we have it's now kind of be a hybrid system if you will but it should where we want to integrate well with the rest of the scene that's sounds promising thank you okay so I have a question for dynamic elements like texture changes for example localization for different languages how would you deal with that when text goes wild is it is there some auto-sizing yeah so actually text is really king in your elements it's almost to the point that it's annoying of course this is a fixed size because this button has a width I think because the column has a width yeah as I was changing out this is the example I wanted to use right so as I was changing this text here you can see the rest of the UI is is adjusting to make room for the text so this is what you get by default with no elements if you're talking about text measures where you want the text to shrink as you gets is that more what so texts right now we don't have the full feature set of text mesh bro in new elements we are using the same core in runtime but we like you can see the the text controls are just this for now so over time we're gonna add more features and expose more of the text message core to you elements and that's where you'll get more control okay cool we'll look forward to that thank you is it possible to still use the UI UX and our files are inside the streaming assets folder or can you be close to the mic oh sorry so my question was is the UI builder able to work if the files are in something like streaming assets folder so that they can be edited after the builds been made so for things like modders and stuff if they're able to change those files after it's been built I not sure I fully understand your question maybe you can come up after and pay me because it's I don't want to say the wrong thing sorry is it possible to inject geometry into the render on your own what sorry your own geometry oh yes so we have callback to essentially create your own geometry and yeah just give it like by default we generate quads obviously for each element but we do have an API API support for creating custom geometry so if you think of Shara graph or VFX graph de edges so first of all both of those are built using UI elements and the edges going between the nodes those are obviously custom geometry that's being built using this API that's not the level I'm referring to though so say you have a UI that you've built and the system generates the geometry that's then rendered can I go and inject my own geometry at runtime on top of that assets rendering it because this is currently possible with canvases probably not right now but as we develop a screen overlay and we embed ourselves using in the universal scribble pipeline render pipeline that will be possible okay I heard that yeah we have actually sorry I should have said nine ten over ten seconds over I'm reading this wrong thank you I will be here or outside for questions thanks [Music]

Channel: Unity

Views: 189,449

Keywords: Unity3d, Unity, Unity Technologies, Games, Game Development, Game Dev, Game Engine, UI, UI Builder

Id: t4tfgI1XvGs

Length: 45min 27sec (2727 seconds)

Published: Mon Oct 14 2019