Making Procedural Music in Unity

hey there my name is ryan hedgecock welcome to my first video on this channel thank you so much for stopping by i want to start this first video by exploring the combination of some of the things i am interested in the most namely the unity engine performance procedural generation and sound design i've decided to adventure into the land of procedural audio synthesis and create the groundwork for a generative audio system in the unity engine i also cover this and some audio fundamentals in my blog if you would like to check that out too i'll leave that in the description but enough of that let's get into it sound is one of the most important aspects of many visual mediums yet it is represented by such a simple stream of data a series of values that move a speaker cone back and forth tricking our ears into hearing things that aren't there i may not be in the room with you but your ears seem to hear otherwise kinda spooky i've always loved dynamic sound in video games for example the ever-evolving sound of the mario kart wii main menu that uses a series of tracks that change and blend into each other but we can go deeper i want to make music down to each individual sample of the waveform alright so let's start off by creating a new unity project i really like cheesy one-word names for my projects so i've decided to call this one synthetic well keep one c just so it's not on the nose we will be obsessing over sound for now so we're probably going to have to get used to this bland default unity scene and do all the seeing with our ears i will say this can be tougher if you don't have a pair of good headphones plugged in but we will try to keep it manageable so where do we even start well we could create our own system to control speakers from scratch but unity has a sound system built in already all we have to do is hook into it conveniently there's a nice method in monobehavious called on audio filter read and this method is called whenever sound is going to be coming through an audio source this can be used to filter the incoming audio but it can also be used to generate audio in the first place let's hijack it to create a simple sine wave create a new monobehaviour along with some serialized fields for the frequency and amplitude of our sine wave we will also add a couple of private members to track the sample rate of the system and the current phase of the wave we want to make sure we initialize the sample rate in our wake method so that we can use it for later the sampling rate by the way is the number of samples per second used for audio output by default this is usually 48 000 samples per second then we can finally get to using the on audio filter read method we will start by creating a phase increment variable that will essentially be the amount our wave phase should progress after each sample calculation the equation for this is actually pretty simple because it's just the frequency divided by the sample rate of the system all we have to do now is loop over each sample in the array using a for loop and calculate its corresponding wave value while also incrementing our phase value each time one thing to note here is the different channels in the buffer are arranged in an interleaving order so we have to iterate over the samples accordingly and apply a single wave value to each channel in order to maintain a consistent output this should be all we need to maintain the state of the wave and output a perfect sine wave to our speakers all that's left is to add our script and an audio source to an object in our scene and hit play to see how it sounds ah music to my ears now i must warn you if you've done something wrong in the code errors and audio generation can be quite harsh so make sure that you aren't blasting the volume when you test trust me your ears will thank you all right so what we've created is pretty much the most bare bones basic sound generation we can possibly create and one of the side effects of that is that it's not very fast if you look at the millisecond counter next to the amplitude meter you can see that it's taking approximately point oh it looks like over 0.10 milliseconds to fill the buffer which to be honest is actually pretty fast but in the grand scheme of things if we're trying to create music we're going to need a lot of oscillators we're going to need a lot of filters we're going to need a lot of things that are processing audio through this system and the reality of it is that that's just not fast enough i know that we can go faster and this is due to something that unity has that's fairly new called the burst compiler now the burst compiler is able to take our fancy managed c-sharp code and convert it into blazingly fast highly optimized native code this significantly increases execution speed for certain functions but the architecture has to be built around its many drawbacks a lot of the code we're going to right now is going to be pretty technical so i'm not going to go as in depth as i did before however all of the code is available on my github if you want to check it out there one of the drawbacks of native code is that we can no longer pass in managed objects unfortunately this includes arrays like our float array that we use in our on audio filter read method you may also know of a struct called native array but this is unfortunately only able to be used in jobs and not burst compiled methods because it has a disposed sentinel so let's make our own native objects then we're going to use a bit of a cool trick here using assembly definitions to manage all of this native code that we want to make one of the reasons we have to do that is because we're going to be using the unsafe tag and unity normally doesn't allow that in our code unless we were to change something in the settings now i don't want to use the unsafe tag everywhere in my code so we can actually check a box for our assembly definition that says in this directory only we are allowing unsafe code the other cool thing with assembly definitions is our ability to use the internal keyword now we'll get to this a little bit later but the internal keyword essentially is like a private variable but anything within the same assembly is able to access it so why are we wrapping up all this native code in its own assembly why can't we just do it in our normal code base well you see something that makes a good software engineer is knowing that you're a bad software engineer and taking every step possible to ensure that you cannot shoot yourself in the foot when working with native code it's very probable that you could leave memory leaks behind and we want to make sure that we don't leave any data around on the heap so by wrapping it up all in this package we can keep all of our native code isolated from the rest of our code and only allow the native usage in a way that we're pretty sure is completely safe and if we do end up with some memory leaks or errors we know where it's originating from okay now let's flush out our native library we'll start by creating a helper struct for internal use only we'll call this the buffer handler and essentially the job of this struct will be to allocate and deallocate memory that we can use to represent an array it'll allocate the memory in its constructor and it will dispose it in the dispose method as this is an idisposable object the type of the value stored in the buffer will be decided by the generic and we'll be able to use this struct in a number of places as we create different buffers for our audio system we also want to add a couple copy2 methods so we can copy data between different buffers and between our custom buffers and managed buffers the last thing we will use is c-sharp indexers which we'll use to pretend like the data inside of our object is an array which means that when we work with it we don't have to deal with the pointers anymore and we just treat it as if it has indexed values the last utilities we're going to make here are an interface called i native object and a class called native box that we'll use to sort of wrap up all these native functions so that we can confidently use them in other places safely now let's get around to making our synth buffer this will be in charge of holding all the float data that we've been passing around to the audio buffer the cool thing here is that we don't use a constructor but i created a custom method called construct that returns a native boxed version of the synth buffer this is nice because the native box does all the handling of the allocating and deallocating for us so we never have to worry about leaving memory on the heap awesome with our native library created we can now move on to using the burst compiler so make sure you have it installed in the package manager i won't go too in depth here but i started with creating a synth provider abstract class which will take care of a lot of the buffer handling and validating and making sure it's the right size really the big part here is this abstract process buffer method which is where when we inherit this we will be writing all of our code to fill the buffer with the necessary materials i have one more mono behavior called synth out but its only purpose is literally just to pass stuff back into the on audio filter read method so with that the time has come we're going to go back to our roots here and make a sign generator it will inherit from synth provider like we talked about earlier and we'll populate it with the usual amplitude frequency phase and sample rate stuff this is where the burst compiler comes in we're going to add a delegate here to represent what our burst compiled method should look like as this is required to compile a function pointer with burst and we'll create a static instance of this delegate and populate it in the awake method the way we compile a burst function pointer is by using burst compiler.compile function pointer with the generic being the delegate for the method but we have no function to compile yet so i'll quickly make one here called burst sign which will be pretty much exactly like the one we created last time then we can use this function as our compiled method and call it from our overridden process buffer now when we go back to our inspector add a synth out and our sign generator and connect them together we can hear the same sine wave we had last time except we're now well under that 0.10 millisecond mark that we used to be this is actually even better too because a lot of the time is taken up by copying out the data back into managed memory so if we deal in the native phase for all of our processing and we only copy over once we're going to be saving a lot of time here well i'd love to do more but this video is already getting long enough but with such a great foundation for this architecture i'll definitely be making videos in the future so that we can expand it to add more musical aspects if you'd like this make sure you like and subscribe for more and i hope to see you next time

Channel: Ryan Hedgecock

Views: 28,810

Keywords: unity, unity3d, procedural, audio

Id: FQnDgMuqqEw

Length: 10min 38sec (638 seconds)

Published: Fri Aug 05 2022