Lessons Learned from a Decade of Audio Programming

so thank you all for coming for delaying your lunch to come to my talk lessons learned from a decade of audio programming my name is guys Samberg and I'm a core programmer at telltale games and so here I'm going to share my wisdom of the last ten plus years of programming audio engines so Who am I why should you bother listening to me I've been in game since 2002 that turns out it's actually 12 years so my talk should really be called lessons learned from a doe decade of audio programming which is totally my new favorite word over the years I've owned the audio engine every company I've ever worked at that was either the engine for the whole company or the engine just for the product I was working on and more importantly though I've actually shipped lots of games everything from slot machines game boy bands triple-a PC and everything in between and over those games I've actually written 13 different audio engines from the ground up from 0 source code and some of those are more than one for each game for example Hell Gate London actually had three audio engines that evolved as the needs of the game evolved the we decided it was just time to rewrite the the engine from the ground up and so over the years I've actually gotten pretty good at it so in this talk and I'm going to give you six kind of pithy lessons that I've learned targeted at you know people who want to be audio programmers or people who are forced to be audio programmers and don't want to be audio programmers so the first lesson is that playing audio is really easy in fact it's one of the simplest things that your audio engine can do or that your game engine does and to demonstrate this we're going to start from the very beginning it's like the fundamentals how does the speaker work this is a great diagram there's a cone attached to a magnet there's wires coiled around the magnet when electricity runs through the look of the wires it generates a magnetic field which moves the magnet which moves the cone which creates sound I hope you all followed that so you can think about if you graph the position or the offset of that magnet from Center over time that is kind of your wave form of your sound and if you do if you have to draw that out to look like this and what we do is we sample it very frequently typically forty four forty eight thousand times per second and that'll look like this and what we do with that is we just move the magnet to wherever it is over at that curve that point and we generate a sound so great we can play one sound um right big deal that's not very exciting it becomes more exciting when you start to play more than one sound so what happens if you try and play the the blue curve with the like what is that cute green kind of colored curve on the right well it turns out that if you play those two together your curve looks like this orange curve and if you do all the math you figure out all the nyquist and all the Voodoo behind it it turns out that all you're really doing is adding these two curves sample by sample just like this your signal a plus signal B equals your output signal and because output signal is itself a signal it turns out that we can generalize this very trivially adding up a whole bunch of just all the signals together becomes your output now I can keep going I can talk about you know I can introduce sub mixing and clipping and all these other things but there's kind of one thing that I want to focus on here and that is that all of this stuff is embarrassingly vectorizable so whatever your particular single instruction multiple data set for your hardware is it's going to get a big workout when mixing audio and playing audios back so the point of all this is that playing sounds is free doesn't matter how many sounds you play it's you know I'm line playing sounds and free playing sounds really cheap you can play as many sounds as you want within reason and it's not going to cause like resource problems memory is is still concerned but in terms of CPU cost it's nothing so if playing sounds is cheap bordering on free it turns out that the audio mix is actually what the hard part is so it'll be in when I say the audio mix audio mix is how you combine the sounds that are playing it's what the it's what the game players are actually hearing there's a lot of things that go into it some examples are like how the music and the ambience will mesh together it's how when you enter sniper mode maybe the footsteps get louder so you can hear if somebody's sneaking up on you it's how the environment the reverb and the the the just the background ambience of the the environment sounds and it turns out that one of the biggest hammers that we have is just lowering volume of sounds that are less important and you can there's lots of tools your whatever audio engine you're using the audio middleware to do many of these things for example gain reduction sound count so that as I turn my head I get quieter 3d distance attenuation and ducting is the process of kind of lowering sounds whenever other side only volume of sounds and other other sounds are playing and all these things are either built into your audio engine or they're really easy to implement there's also more advanced features I call the mixed States I think they're called sound moods and Crytek they're called mixer snapshots and many other engines including F mod Studio and this is a much more involved subject where you're basically taking a snapshot of what you want your mix to look like and fading up to that state whenever certain game events happen in fading out and I wish I had more time to talk about this I only have a twenty five minute talk but just mixed States mixer snapshots is like a 60 minute talk on its own so there's a lot going down now if you take the lowering volume and kind of take it to its extreme it turns out that a big chunk of audio technology actually revolves around not playing sounds at all and that takes the form of things like maximum play backs where just this sound can't play more than so many times stream limits which are usually a restriction on your bandwidth or kind of your spinning medium which is becoming less of a concern with more modern stuff but for example the Xbox 360 often has stream limits virtualized sound sources where just sounds that are less important get cut off completely so they don't become a part of the mix and all these things are again either already built in for example virtualized sound sources or they're things that are pretty easy to implement in your audio engine but I want to talk now about one that's a little bit more involved in advance and that is Walter Murch is rule of two and a half so this gentleman here is Walter Murch he is an academy award-winning sound designer I worked on thx 1138 a number of other other great movies and while he was mixing thx 1138 he realized his realization he said you know so long as I'm laying out these footsteps I have one person walking I have to like synchronize the footsteps with the with the singing so if I have two people walking I have to synchronize the footsteps with the scene as soon as I get past to the two-and-a-half thing all of a sudden it's less about every individual footstep and it's more about the sound of footsteps so how did we take this and make it into something that was valuable for a game and for hellgate London and we did this also on Bioshock 2 we implemented this thing which we called max within radius and we allowed for each sound in the game we allowed the sound designers to define a maximum number of playbacks within a given radius and they defined max playbacks and the radius for that sound so in this situation here we have the blue triangle as the listener the green dots are the sound sources and so we draw a circle or sphere really around the sound sources like the given radius and then we cut out in this case all but two of these sounds which was the max playbacks for the sound and what this what this does is it maintains spatialization so in this situation we have two sounds coming from the left we have two sounds coming from the right two sounds from behind to maintain that but the mix isn't getting muddy your you start to play all these sounds on top of each other it starts to sound like noise and so by doing this we are creating a situation where we still have all the spatialization we still have the sound coming but we're actually using less resources while still having a cleaner mix so if I were to summarize the audio mix and kind of one pipi sentence it would be that you want to let the most important sounds shine but more precisely since we are audio programmers not sound designers it's that you need to provide tools for your sound designers to allow them to make the most important sounds shine speaking of sound designers you need to learn to work with a mer your customers and they are wonderful beautiful people and they they will there they will create awesome content with whatever you give them but really what they want to do is ship a copy of Pro Tools with every game and you know we'll come back to this I promise they also have their own language with lots of crazy terms that you have to learn and this shouldn't be come as a surprise we as programmers have our own language and so what you need to do with all these things is you need to learn their language and you learn speak it to them I need to love it because your sound designers will love you for loving their language if you come to them and speak to them about decibels and Headroom and VCAs it makes it much easier to communicate when you're talking to them in their own language there's one one of these terms in particular I want to talk about that is the decibel decibel is simply defined as one tenth of a bell what's the Bell turns out but the answer is it doesn't matter a bell the most important thing you need to know is that 6 DB is half or double volume so to find out what your decibels are you or what the volume percent is you divide by 6 and turn it into a power of 2 easy enough and in fact when whenever I write a new audio engine the first thing that I do these 4 lines of code are the first four lines of code that I write these are conversion functions from decibels to volume percent and back and I'm going to say here for about like four more seconds let people who are taking pictures of it or writing them furiously down these are these are wonderful functions the first thing I that I do I hear lots of snaps oh hold on I can go back click quick quick okay all right we got it so I'm gonna before we leave the subject I'm going to leave leave you with the biggest best secret that I know for working with sound designers so let's say your sound designer comes to you and he says I want to be able to kind of play these you know this list of sounds randomly kind of with a timer on it and go from there you tell them in one word you can say no no I'm not gonna do that for you that is not at all what you want you're not trying to solve that problem you're trying to solve the problem is creating a background an ambience let's see if we can find a solution for that and give you what you're actually asking for because remember sound designers will create wonderful content even if you give them trash horrible awful tools to do it with and this situation that I talked about is is a dramatization of something that happened before I came on on a previous product that was actually hugely award-winning so even you know this is this is true they will create award-winning content even with trash so your goal then should be don't give them trash even if your sound designer comes to and says I would like a steaming pile of trash please you say no instead you give them the tool best tool solved a more general problem in a long term lazy fashion if you are long term lazy you you spend more time upfront and you give them a great tools that solve exactly what they're trying to do and if you do this then your product will ship on time it will ship maybe even early and sound will not be a problem if you spend the time upfront so we're all game developers we have to do things quickly and middleware is awesome there's F modern wise and various other middleware things that will make bootstrapping your audio engine really really quick so impacted on it's worth is I'm in a bootstrap an audio engine here in front of you this is all the source code for an audio engine I'm going to go about 99% light speed but I'm hoping that I will get my point across here we go this is our header for audio engine we have some static in it update and shutdown functions we have some functions to manage sounds and playing channels we're using the pointed implementation so here's our pimple and then we have some one here I'm using F mod for my source data but the principle applies for whatever audio engine you're using and then we have some it and shutdown functions there is constructors and destructors which I'm not going to show the source code here for because they're boring they're just copy pasted from the documentation and then just releasing all your assets here's our update function we're going through all of our channels seeing what's stopped we remove anything that's no longer valid and then we update the system our load sounder none one load sound are really kind of mirrors of each other first which like our cache then we either load on load the sound and then we either add or remove it from the cache I'm going to spend just a little bit more time on play sound here so here we have we see if the sound is already loaded if not we load the sound and then if we still have and don't have the sound and we then we exit early and then finally we play the sound and set all of its parameters now there's three things that I want to call out here number one is this parameter here the third parameter to play sound it's boolean true what does that mean it's starting the sound is paused and what that does is if you don't pass it to pass false the sound will start playing immediately or as close to immediately on the audio thread the mixer thread is going to get that and what can happen is there will be a brief window after you've started the sound before you have set the 3d parameters and the volume and whatever else you're setting while that sound complain you get a little pop well and then the sound goes to wherever whatever volume and stuff you've set so by starting it is paused and then setting in parameters none pausing it you are you're preventing that pop and kind of priming the pump second thing I want to call it is the return value of this is an integer and basically this is how the external system deals with playing sounds or interacts with playing sounds is through this and yes you do have to do a lookup every time to actually get the underlying object and it's tempting to say well let me create an object like a playing sound or a channel object or something that you can call functions on the problem is now all of a sudden you have this object that is in cahoots with the sound system it needs to know about the internals of the sound system but it's not really owned by the sound system and so the underlying concept so the underlying memory if that this object is pointing at can can disappear and then it doesn't know about it I have fixed a lot of crashes a lot of weird crashes simply by converting from sound objects to integer IDs the last thing I want to call out here is that even if we fail to load the sound we still return a valid channel ID and this is really just to protect against one particular pattern of coding which I've seen particularly in particle systems where there's a sound that they want to play and then every frame they say is my channel ID valid if not play the sound next frame is my channel D valid if not play the sound if you are constantly returning an invalid channel ID if you failed load the sound you're going to be wasting all this processing trying to play the sound that you're never going to actually succeed on so we always return a valid channel ID even if you fail to load the sound last piece of this is just the set channel whatever these are really boring just follow the pattern and go from there so in summary we have it took about 300 lines of code for a fully functional audio engine it took me about an hour and a half I'll grant you I know what I'm doing and what needs to look like at the end so you know it might take other people longer but so what like why did I show this to you well it turns out that you can ship on the moral equivalent of this code there's a few little bits and bobs other than the initial destruct that you need but you can access on this and I have like minion master there's a game that I did as an indie with bit flip games this is more or less this is more or less the the audio engine that we shipped with but like remember this I showed you this wonderful Pro Tools that we want to show up a copy of it with every game does this look familiar here on the Left we have wise on the right we have F mod studio there very similar to this and these tools will provide your sound designers with a lot of features just right out of the box snapshots and and busing and all these wonderful things and it's really tempting just to give them the tools and say great I'm done but you can't just throw these tools at them and call it a day like off the top of my head here's a small subset of some of the things you might have to think about in your game engine now not every game engine our audio engine needs all of these things but all audio engines will need some of these things so I wish I had time to talk about all of them but I don't so lesson 5 is a sound is always the first thing to get the shaft and I'm going to present you with some some I can see some people nodding here that's awesome some situations that are inspired by real-life events not exactly but you know officials close enough so we need five Meg's of RAM take it from sound they have too much anyway our ship date hasn't changed but everybody else is running late well sound is post-process so guess who gets to crunch we're staffing up who do we need we don't know nobody audio engine yeah nobody gets to work on the audio and I want to do a little bit of this little dude here is the flagship studios sound caveman the the people before I started working at flagship studios they all knew that they were sound caveman and they had a very primitive audio engine so whoever kind of touched it last became the expert they became the sound neanderthal and they got to key they got the the sound caveman it became the expert so when I came in I got to keep the sound caveman because I own that audio engine so what can you do like these are kind of fun Lowell situations but what can you do if you're faced with these situations so the first thing is the best ideal case is push for dedicated audio programmer I mean if you can it's not always possible maybe your small indie and you can't hire anybody maybe you there's just no budget for a particular audio programmer but if you can then there's like it's awesome it's a great position to have in your in your studio if you can't then you can learn it yourself it's actually really rewarding it's not that difficult and it's actually quite fun to to do this here we get somebody thank you sir and the last one is a little bit sneaky err you can ask for more resources than you actually need so you ask for 50 megabytes they give you 40 megabytes and then you plan for 35 at the end of the project when they steal from you but make sure if you do this that you use everything that you get if you add if you actually use 35 Meg's of the 40 that they give you then on your next project they'll say you only use 35 so the viv you three five and then they still steal from you at the end the last lesson here gonna leave you with is listen to your games it's critically important that you listen to your games as much as you can over the years I've learned to tell the difference between all these different kinds of noise just from listening to them I can tell where they're coming from and why and this is it's not easy but you know over the years have developed us and so I can do this can you now I bet all the sound designers here are going well yeah but the sound programmers you need to develop your ear you need to listen and know what the difference between sample discontinuity in a buffer under and how they sound different in Y so I cannot stress this enough there is no substitute for listening to your games and as many situations and speakers and whatnot as you can the last thing I want to do here is talk a little bit about the history where I can't where I've come from and where I'm going and you know what went right and what went wrong with a little bit of a mini post-mortem on my audio engines so a lot went right I mean I wouldn't be here if that weren't true I learned a lot you know I started in slot machines and I really didn't want to be doing audio but I have embraced it and I love it now um I also shipped you know shipping is a big deal if you can't ship or you haven't shipped then you're just noodling around and so shipping is a really big feature even if you're not complete or beautiful or perfect or well even if your code rather is not good so I've also made some wonderful tools my sound designers the Bioshock 2 voice packager took a process that was just hours upon hours it was literally an all-nighter if they needed to do this and it was error-prone and horrible and I took this into something that he could set up in five minutes before he left for home and who come back the next day and we'll be done perfectly every time the background sound engine for Hell Gate London and it's iteration rewrites slash iteration in Bioshock 2 is a beautiful piece of work that I'm really proud of and also the third-person camera listener tech that I wish I had time to listen to talk about here in halgate London is where we originally developed this I'm particularly proud of that and if you want to talk to me afterwards I can I'll be happy to describe that to you thank you um so excuse me it was not all sunshine and roses there were some really big misstep sand the one in particular I want to call out is the hell gate London music engine which was this horribly over engineered piece of work that really didn't solve the problem at hand Dave I'm sorry and I want to say out here I'm really sorry for all you people who played Hell Gate London and never heard the action music that are awesome composers wrote that's my fault I've also historically not been aggressive enough about threading you know there were opportunities have been there for a long time to thread your audio engines and to you know obviously you need a mix or thread you need some threads built in but threading the interface is a very valuable part that isolates the rest of your code from any kind of hitches that may be in your audio engine so where am I going from here future plans I'm a telltale games right now I can't talk too much about what we're doing and where we're going but I can tell you that this is the best audio engine I've ever built and in fact that's my goal is always to make some sort of measurable improvement in the lives of my sound designers before I finish up here I want to leave you with one last bonus session which is the bonus lesson excuse me which is that audio programming is fun you should do it and you should not be afraid of it thank you very much we have about three minutes for questions any questions yes sir so the question is other than f mod and wise and some other things what tools can I recommend that's an excellent question I'm an F mod guy so I've focused on F mod throughout my career there are like there are some open source audio engines SDL has a mixer a lib mixer I think it's called and there's open Al so there's there's some of those inter sorry sagen maverick if you use unity o fabrics your fabric if you use unit Eve's from over there so that there are some there are some alternatives but that's that's what I got we have a question back there ah so I'm working with the sound designer on a project and historically I've been rolling a custom audio engine but he's been really interested in leveraging the features of the editor tools and iPod sound system so currently we're talking about a sort of integration stuff I guess um what's your general experience with like the level of autonomy you have with with the middleware sound engine versus a ground-up system it's a bit of well know it's a fair question um I much prefer working with middleware because the I've written a low-level mixer first long machines really like mixings at the sample level and the problem is that that's tedious and error-prone and you have to deal with driver differences and you have to actually parse the sounds yourself so if you can get a middleware that is that whatever level abstracts the best for you then that's the right one so F mod is I like F mod because it abstracted multiple levels but there you know wise is also great because it abstracts at a very high level and there are like the SDL mixer and some of these other ones abstract at a very low level and so whatever whatever you're mixing philosophy is wherever you are spending your most time that's where you want to spend whatever wherever you're trying to solve your problems that's the appropriate level of abstraction for your middle layer so just as an extension of that I find a lot of like potential and enjoyment honestly in writing custom DSPs what's your experience with middleware that's easily extensible with custom DSPs I hate to sound like a broken record but F mod actually has a great plug-in architecture for that I believe wise does as well I haven't looked at their API recently and beyond that I can't I cannot speak um I think we have about 30 seconds and then I will be at the wrap-up room 307 so we'll take your question as a sound designer how much do you recommend that I wear we dive into the code and learning audio programming for the most successful collaboration with audio programmers that's an excellent question in general you shouldn't have to be a programmer I went to a great talk about the the audio engine and it was one of the Killzone games recently was just yesterday or the day before where they did they had kind of a boxes inline scripting language for their audio team and that that's the kind of deepest I would allow a sound designer to get somebody who's kind of focused on creating sound you know if you want to feel like you're an indie and you want to do both then you know more power to you but if you're if you're more just like I want to sound designer how much should I know how to program that that's the level of abstraction I would recommend thanks cool so I think that will I will be at the I think where they're kicking us out so I will be across the hall in 307 if you have any more questions

Channel: GDC

Views: 90,800

Keywords: gdc, talk, panel, game, games, gaming, development, hd, design

Id: Vjm--AqG04Y

Length: 26min 14sec (1574 seconds)

Published: Sun Aug 07 2016