CppCon 2015: Timur Doumler “C++ in the Audio Industry”

all right everybody my name is Timur I'm work at london-based music tech company Rowley and yeah I had the idea to come here to talk a little bit about audio and music software and how we use C++ as a language so this talk is called superstars in the audio industry can everybody hear me so right yeah okay so I had this idea and then I was about to prepare some slides for the talk then I thought okay what could I talk about and I had like several things I could talk about so there's the real time aspect of audio when it comes to programming then there is if you generate audio you are typically doing math and DSP so you want that to be really fast and efficient then for example a trolli we do this seaboard it's essentially a keyboard with a 3d touch surface and there is an audio engine running on an embedded system in there so there's also very very interesting things that you could talk about that in this context also you want to do audio on different platforms on desktops on mobile phones and they're interesting cross-platform challenges there and also another thing if you want your audio to be fast there is the whole topic of zim D and vectorization and memory alignment and there's a few interesting things in the audio world about that and then I had all these things and then I figured out that okay if I'm going to talk about all of this I'm going to be to be talking for 10 hours so I had to limit this talk to something and I thought okay what's the defining thing about audio which is different from most other use cases of C++ and I think that's really the real-time aspect of it so I'm going to talk about real-time programming and why that's different probably from other users of the language and if you want to do real-time programming you want to do lock-free programming as well so I'm going to talk about that and I'm going to show a few code examples so I'm going to start with introduction to what I mean by the audio industry and how actually audio is ripple sound is represent than C++ or in a c-plus application and then I'm going to talk about what why audio is real-time programming what it actually means and how you write code so that it's real time safe and so if you code examples about lock-free programming how you synchronize your threads lock-free in context for audio which will involve how do you share objects between different threads how do you exchange data between different threads if one of them is a real-time thread and a little bit about how to manage memory how to manage object lifetime and these kind of things so what's the audio industry probably different people have different definitions but so I think it's about people who use computers and computer software to produce music to record music to make sounds to compose music then there is this aspect of people who use computers like this guy here with the laptop to perform music life deejays performers musicians then there is this whole world of games and audio apps on desktop and mobile platforms for consumers multimedia apps then there is also science and art and creative coding where audio plays a big role and then also there are people who actually develop all the software for people who create musical instruments synthesizers and music software and hardware and this is the small as my domain I work on the juice framework which is a framework and application framework but also features a lot of audio for people who want to build applications with audio in them on different platforms so I'm a little bit biased towards providing tools for developers and also this whole audio production music world like having musical instruments this is the Seaboard here having synthesizers and audio software running so this is sort of a little bit my bias but I'm going to try to keep it a bit more generic nor too specific so um question who has ever here used actually audio a programmed actively C++ for audio and sound things alright about half alright so maybe it's worth to do like a really quick introduction about for all the people who didn't raise their hands how audio actually works in C++ or in typical C++ program so audio essentially separate representation of sound waves so here's the sound wave and obviously we can represent this digitally we can manipulate sound digitally but the C + s language itself has no concept for it there is no concept of audio in the language so we have to use third-party libraries you have to use API s and we have to use the conventions offered by these API and libraries and sort of the the universe we are sort of in its defined by like a big stack of different components so essentially we have the sound waves that we can hear and then we have a microphone that can record them and then we have an audio digital converter and then it goes into a sound card and then we have some sort of driver for the sound card and some sort of API by which this driver talks to the operating system we have lots of different operating systems we have Windows Mac Linux we have on mobile we have iOS and Android you also have maybe implanted platforms and then all of these operating system the systems they also have their own stack of audio layers of Audio API s and at the end of it there's some API that's offered to to the user which you can use them to program your own audio app so and there are many different ones so apple offers Core Audio and Windows there are several ones so it's like a whole set of different in different api's and this is what your audio application is talking to to exchange audio data and also the other way around if you want to output audio it goes all the way the other direction through all this chain through some kind of speaker which actually transforms that to a audible audio signal and in the world of music production software we have actually an additional layer where we have something that's called a digital audio workstation or a door and there are several computing doors which people use to produce and record and mix software and then these doors offer you plugins so you can like a plugin interface so you can create synthesizers or audio effects and then these will be a plug-in essentially a dynamic library a dll that is loaded into that host and then to have these it's like you can imagine like Photoshop has plugins something like that except that there are different computing hosts and also actually different formats and SDKs for these plugins so you will be coding against one of these SDKs and then your audio plugin that actually produces sound or manipulate sound will talk through that API to some host and then it goes all the way to through all this chain so one of the things that we are doing with the juice framework as we try to offer solution where you don't actually have to worry about this stack because you can also imagine that on every operating system this all will just stack is different and all the conventions are different and so what you want to do is basically abstract all that away so that you can have a means of writing your app which is maybe a plug-in or a standalone app and do sound and do a graphical user interface and all the other things that you need for an application and then you have a framework underneath that hides all this complexity but here we are going to talk about the audio part so let's talk a little bit about all your data how its represented so essentially if you have like a waveform like this which is a little snippet of sound then to represent digitally you would sample it which means that you would at regular intervals would represent the value of that waveform with a number which is which are the little blobs here which are the audio samples and then you have a certain amount of samples per second that's the sample rate like typical one is four to 4,100 Hertz or samples per second this for example the sample rate used for pay and a CD and you can use also higher sample rate depending on depending on your use case I'm sure you all have seen this setting this sample rate setting somewhere in your software so this is how many samples per second you have and also this way of sampling sound has lots of implications as a snake with sampling theorem and aliasing and all these things which I'm not going to go into here but so you have samples you have a given amount of samples per second and then the other convention is usually in this kind of super fast software you would use floats in the range of minus one to one to represent that I mean there are many many other ways to do this but in this context of audio running on desktop more platforms typically you would have floats between minus one and one which gives you about 24 usable bits of information per sample and some people say double is better but I can't hear the difference personally I don't know I'm not sure if double is actually better so you have enough enough bits to represent sound and also you need to stay between minus 1 and 1 because otherwise if you go above you will have some very harsh clipping distortion which you would hear so this is sort of the conventions that you're working with and the other thing is that you might have several channels of audio so typically you will have at least like a left and a right channel or if your surround you may have 6 or and then we would have an audio frame which is basically one sample per per channel which will be an array of floats in this case it would be two floats representing one little snippet of audio which is just a tiny fraction of a mic over millisecond now if you want to output sound or you want to get sound input you have to talk to your audio card eventually through this whole stack that we saw earlier so you would your jacquard which ask you give me give me the next sample and you would receive some sort of callback now it's very very inefficient if you would for every sample 44,000 times per second you would get this number from your program you would receive 44,000 callbacks per second that will be very inefficient so typically what is done is that you transmit audio between this whole stack in so-called audio buffers where you take a chunk of samples somewhere typically between 32 and 1024 samples it's one audio buffer and this is the size of audio data that's transmitted back and forth between the different applications and api's so one buffer will essentially be a array of areas of floats so the first dimension would be the channels and the second dimension would be the individual samples in one buffer so and basically what you want to do is you want to exchange these buffers with the with your audio card so typically the means of doing that is a callback function which simplest possible one is here so you have a callback which gets the data the actual channel data that's one buffer if you saw earlier then you need to know how many channels you have and how many samples you have and then basically loop through the channels and you loop through the samples and then you do some calculations ticks you generate sound or maybe you get the sound and in this very simple example what you do is you get you're just writing zeros which is always like the least thing you want to do because if you're not writing zeros then your audio card will receive garbage or uninitialized memory and you will have some ugly noise so but of course in real world you would not buy zeros but you would actually have some math that generates your audio and in the more general case you have inputs and outputs so you would have one pointer to the audio data that's coming from the sound card and another array that going to the sound card and for example you can take the input transform it somehow and then send it back out so that's like that would happen in here so this is this is this kind of audio callback which is the means by which you exchange audio data so to give you just a example so this is the producer this is a live coding C++ IDE which we are presenting here this week which you want we are about to release so if you want to know more about this I recommend you to go on Wednesday to the talk by Julian Store sitting right here who is the original author of juice and he's going to talk about the producer but here I'm just going to use it for my little demo so this is essentially a little app and here you see that audio cool back so now we have here just some just outputting some sounds we're just using one channel and basically what I'm doing here is I just have a sine wave and practice you would not probably not use to design to generate it there are more efficient ways of doing this so you're just generating a sine wave with the phase that's just counting up and we are sending that to the audio card and that's all we're doing and if I run this we get a sine wave which sounds like this also this is a little class that I wrote was just like an oscilloscope shows you the waveform so that's the actual waveform it's playing now by changing the the math that generates the audio you can continue a different sound so for example in the next line we just comment this out we compile again yes that's a swelling so basically instead of generating a smooth sine wave I'm saying still copy sine which makes it just minus 1 1 minus 1 minus 1 so it looks like this so it's a bit bit sharper or I can just the next line here I can just also some random numbers here but I know Stu Brandt is not the proper way to generate random numbers but for this purpose a sorcerer so it's just random numbers between minus 1 and 1 and so for example yeah I couldn't change the level here oh you can introduce the syntax error somewhere alright so basically I'm just going to quit that so basically the whole way of synthesizing sounds is to go into that audio callback and eventually you have complicated formula to generate all the sound that you want there's many ways of synthesizing cell and sampling sound so I'm not going to go into this it's going to talk a bit more about the c++ aspects of it so um because it's you have this audio callback audio is so-called hard real-time programming I'm going to talk about what that actually means and I think this is the big difference between the audio world and most other applications of C++ so what I mean by hard real-time so we have here this waveform and we split it an audio buffers and you want to send these buffers to the 2d sound card right now the sound card has a little quad crystal here which gives you depending on your sample rate gives you a record regular pulse and every for example 44,000 times per second it will actually less because an audio buffer is for example 32 samples but every let's say every millisecond it will ask you give me the next audio buffer give me the next audio buffer give me the next audio buffer and then it will call the callback then you have a little bit of time to write all your audio data into that buffer and then it's going to send going to be sent to the audio card and then one millisecond later with the next callback give me the next buffer and the problem is that um this audio callback doesn't wait for anything no matter what happens on your computer after one millisecond or however your setting is there will be the next callback and you will need to react to it and you will need to have written your previous buffer into your previous data into that buffer so um that's what I mean by real-time and if you look at the like the time scales that are involved so if you have an interactive performance like if for example you have a keyboard you're playing it and you want to hear the sound as you play it you have like a window of about like ten milliseconds of overall latency where it feels really like it's life you don't feel any delay if it's somewhere around between 10 and 30 milliseconds like a pro professional performer will already feel okay that there's like some paper or something between my fingers and the keyboard it doesn't really feel life and then if it goes above about 30 milliseconds then most people hear like a delay like latency between if you strike the key and you hear the sound so it's alright if you have like a multimedia player that plays something or if you have you just offline like mixing some sound but if you want to actually perform you really want to be more like somewhere here and that's the overall latency of your whole system so it means that if you have this audio callback and you're in your program then definitely you will you will you want to use a second compromise between a small buffer size and like sis tech system load like how much your processor can handle so if you work with with a small buffer size of 32 samples and you work with the 44,000 Hertz sample rate then a buffer is just under one milliseconds so if you add the whole the whole rest of your system to the latency then you can stay below 10 milliseconds but then you have more than one callback per millisecond and you have to generate your data in that time and depending on how much you run on your system or how much algorithms you run this may over this may be a lot of stress on your CPU so if it goes too much you can increase the buffer size and you have a little bit more time to compute your audio or to process your audio and then you will have less system load but then you will have more latency so it's always this kind of compromise and so we are working with these times of in the range of about one millisecond and if you imagine you have this callback here and then you generate your audio and in practice you will not have like stood sign in there but you will have an arbitrarily complicated amount of a chain of complex algorithms which compute maybe reverbs or take like a file from want to play a file file sample from the disk and apply some complex DSP effects on it but what do you want what we need to do is you won't need to guarantee that after this 0.7 milliseconds for example this function will have computed everything this function will return in time and the Aquabot will contain valid audio data afterwards and there will be no error or exception and this function will always return in time and will always be done with its calculations and if you fail to do this so yeah and you you really want this to be no except because it should not throw an exception and if you fail to complete that function in time or otherwise you could generate an error in this in this callback then you will get what we call an audio dropout or a glitch which is basically you can hear like as a crackle or like a like a silent and it's immediately audible even if you have a if you drop just one buffer you can hear that so um and the most important number one rule of audio code is you you never want to cause audio callbacks audio dropout never and the reason why you never want to cause audio dropouts is imagine you're in a situation of this guy so you're you have a laptop here which runs some software which calculates some audio buffers and you're playing in front of all these people and then you have all your dropouts and then maybe you have like a 15 kilowatts PA running there and then if you have an audio dropout then in worst case you will first blow the speakers and second everybody in the audience will go deaf so you definitely don't want this to happen like ever and the other way around if you are getting all your data from the audio card it's actually really the same story imagine here like you have an orchestra playing something and then you have a massive studio where you run some ear under you record them and you run like a big mixing rig but in the end you have also like an audio callback which gets some buffers and writes them into the disk so that's essentially what happens and if you just drop one buffer and fail to write this on to the disk and time then yeah you can do the whole recording again and you can ask all these people please could you just repeat the whole performance because I fail to write an audio buffer in time so you never want to cause audio dropouts and I think that's like the defining characteristic of like real-time audio programming and the other thing is that not only you want to be done with your writing or reading your audio buffer in time but also this time should be significantly shorter than the time that you physically have if you're talking about a millisecond between the different buffers and the reason for this is if you have something like this this is traction this is a door like a production music production software written with juice and here you have maybe like a hundred different tracks of audio make all the different instruments and you want to record that and you have like on every track you have like several effects plugged in so maybe have a reverb or delay or distortion factor and all of these plugins and all of these tracks they will have their their audio callbacks running so you don't have one of them but maybe you can have several hundred of them and still you want to compute every every single buffer in time and I think that's that's the big difference between audio and almost everything else because obviously we all care about performance that's why we use C++ that's that's why we all here but if you have for example I don't know you run a server and you want the server to process 1 million requests per minute or I know you what you writing like a video game you want to have like a really high frame rate on average or and in all these cases you want to have like a high throughput or maybe I don't know you're mining bitcoins and you want this whole calculation to be finished in a reasonable amount of time so it's about the average performance the average throughput that you care about but an audio you really care about every single time if you call a function you have to guarantee this this function will stay below like an absolute a balance of how long it can take and you must always it's like a contract you must always be finished with that function call in time and I think this this is like a very specific thing that changes that changes a lot of things and if you do this new plus pass then C plus itself in itself is not a real time safe language which means that it with us you can't declare that a function must be finished in one millisecond there's no way in the language to do that there are languages you can do that but C++ is not one of them so in order to fulfill this real time guarantee that we need an audio you need to do this by the way how you write code and yeah there are certain good practices or bad practices or things that you should and should not do in order to fulfill that guarantee and actually the next rule of audio code is if you want to do this if you want to succeed you should never do anything where you can't predict how much time it will it will take because you always have this hard real time guarantee so if you don't know if you function you're calling in there will be done in 0.01 millisecond don't call it which means you never want to block you never want to wait for anything because you don't know how much time it will take so you should not you cannot lock a mutex or maybe try to lock a mutex and spin on that it's even worse because actually busy waiting you should not join a thread because you have no idea like when will the thread actually be done and actually you can't really call anything that blocks or waits in anyway and especially you can't call anything that box or waits for anything internally which is much harder because you don't know what's happened inside and they're surprisingly many things that block or wait internally and what you really want to do is you want to do block free program so um and why you shouldn't block I think by now with modern computers like the time that it actually takes to lock or unlock a mutex is not a problem the problem is that if you're waiting for something you have no idea if you don't know how long the code you're waiting for will take so in most cases you're waiting for code another thread which is since itself is not real time safe it does not fulfill this real time guarantee so if you do that you get an order dropout and the second reason you don't want to block in the audio thread is the thing called um priority inversion which is that this audio callback typically runs on a very high priority system thread but if it blocks and waits for another thread then that threat will be a lower priority threat and that means that if you have like an operating system and thread scheduler then that lower priority thread might self be interrupted by other threads so you might not only wait for the other code to finish but also you might wait for completely unrelated different threads because you're waiting for a low priority thread so that's priority inversion and really the only way to get away with this is if it comes to audio callbacks just only run lock free code which means you should not call new and delete because that's not lock free it will internally have a lock that may wait for another thread so you don't want to allocate a deallocate memory and you should not probably not call new or delete yourself anyway but also you should not use any error eye objects like share Porter for example that do that internally and there are many many many functions which are not lock free inside like if you do a step back to push back then if the capacity of your vectors is is over then to push back another element you the vector may reallocate all its content which is definitely not lock free because it allocates memory so you can't call vector pushback in the audio callback what you need to do is you probably the best way is to work with data on the stack or if you need data on the heap you should pre-allocate it you should be careful when this data is deleted you can use containers from boost intrusive for example which are similar to STL containers but they don't own the object so you have to handle all the pointers yourself or you can use boost lock free which offers several containers that are lock free itself and also you can use a custom real time safe containers like lock free cue stacks list there's a lot of literature and they're also in juice you offer several data containers for this purpose and for larger systems you really want probably to manage your own heap where we have some pre-allocated chunk of memory and you have a lock free way of accessing it you also don't want to call any kind of input/output functions like see out printf in the indie live coding demo I did earlier if you if you do like a printf in there you immediately hear like the crackles you don't really want to communicate any other processes definitely you don't want to access any files on the disks because this will most likely take more than 0.1 millisecond that you have you don't want to talk to the network so and you don't want to do any graphics rendering either so all of these things you have to do on other threads and you have to synchronize them with your audio thread in a lockrey way you should also probably in the audio callback not call any third-party code because usually Microsoft or Apple or other vendors they don't document whether the functions are real real time safe or not no no documentation says this function is guaranteed to take less than one millisecond time I never seen that in the documentation and also you have to watch out for worst case behavior because here you don't worry about average behavior you really were worried about the worst case for example if you have a hash hash table I can I know set inserting or looking up something is constant time except it's not it's amortized constant time which means that once in a while the whole container might say oh I'm gonna have to rehash all my content now and you have no idea how long this will take so you cannot use that and the other thing is what you can't do is you should not run into page faults and this is an interesting one because I think for most people this is something that was probably an interesting problem maybe 20 years ago when computers didn't have enough memory and since long forgotten and you should not care about this but in audio there is a specific use case where this is actually still still a problem that you should think about and this use case is so-called sampler instruments how this works is you have that loads of them on the market if you have also one one of one sampler in our equator centered Roley and how it works is you sample a musical instrument like a drum kit for example and you have little sound snippets for every single drum or every single note that this instrument instrument can hit maybe in different articulations and you have a keyboard and if you if you basically press a button on that keyboard you want that little audio snippet to play and then you can easily have like a one or two gigabytes a geeks or a gigabyte of these samples and you don't know which which are key the user will hit next time but if the user presses the key you want to hear the sound and if you have two gigabytes of memory which your application users there's a high chance that some of this memory if it's not used it's going to be paged out it's going to be maybe swapped onto disk something and then when you actually access it it had you generate a page fault it needs to be paged in and this whole process will probably take longer than the time that you have and so for these kind of music software arm it's actually a problem in practice and there's two solution specific this either you can have like a low priority thread which regularly says okay I'm going to use this memory so you can access this memory so it stays hot it stays into or actually the more the better solution is that modern operating systems they all offer some API slack em lock and man lock on Mac and Linux or virtual lock unlock on Windows which lets you lock lock memory into the virtual memory so it's not going to be paged out and unfortunately there's nothing in the CBS 2 standard or and boost for that so you have to rely on platform specific things and also probably you don't want to call that yourself but you want to wrap that into some kind of smart pointer or you want to write your own STL compatible a locator for that and it's really tricky to do so probably are what's better is actually to have some kind of pre located memory for your application which is locked into memory and then you need some kind of custom heap management to do that so yeah I'm not going to show any code that does that because quite complicated in practice and could take a long time so I'm just going to summarize why do we why do we use C++ for audio so obviously most of the audio api s are actually c or c++ so it's very convenient then as we all know c++ as fast as close to the metal but then also it allows you to use custom memory management and it supports concurrency at the c plus 11 and 14 standards have atomic in them so the hepa Tomic types they allow you to have lock free access and they allow you to program lock free and in the end for audio this this whole thing allows you to write real-time safe code which is what we want to do yeah look how much time we have alright so um so here's basically the situation you have you have one audio thread which always gets these audio callbacks give me the next buffer give me the next buffer give me things buffer and then maybe you have another threat with a GUI with a graphical user interface where the user can turn some knobs and these will have an effect on on the audio and this may be runs on some kind of message loop then maybe you have some some hardware connected where somebody plays on a keyboard and the keyboard send messages through your application we have protocols for that like MIDI and OSC and you have a threat which receives these messages and then maybe you have another threat that reads or writes data from end to the disk and somewhere you have to synchronize all these things but you have to guarantee that your audio callback will always be time safe and that's like the I would say the big challenge in audio software basically this threat should never block so let's look at a very simple example you have a knob which changes the level of your audio and the user can turn that with the mouse and this should affect the audio thread now the easiest way to write this and actually I've seen many people writing code like this is that you have that's your level that's just a float and then you have in the GUI thread you get some callback from the mouse event and then the level is changed and then you have in the audio callback you have that's your would you cool back and then you just multiply that level with some buffers and you sent it to the audio card now on that's a race condition right and even though it's just a float since he vs. 11 this is actually undefined behavior so even though a float is actually almost architecture's this will be atomic like reading or writing a float c plus 11 standard says that concurrent access through the same memory is a data race it's actually defined in the standard and data race is always undefined behavior so you don't want to do that even though this code this specific code with the one float will run you will not have problems with this code but as soon as it gets a bit more complicated than that you can get loads of problems so you definitely don't want to write code like this so you can get undefined behavior the access to your data can be not atomic which means that the thread may see a value that's like in between the old and the new value it's called the tone read or tone right also you have the problem that if you write code like this where you don't use any locks and it's not atomic you are you can have compiler optimizations that break your logic because the compiler thinks are you're not you're not modifying that code and there's no lock or no synchronization anywhere and then also if you just use plain float the concurrency is not expressed in any way in your code so and the fact that X is not atomic is actually even for fundamental types it's more surprising than then some people think like for example if you have not a float but of 64-bit integer which is fundamental type and you just set this to to some value in one thread and read it from another thread and you compile that with GCC for 32-bit which is an actual use case because if you do audio plugins you want to compile them a 32-bit because many hosts they just understand 32-bit plugins and if you compile that then that's the assembler output you get so actually GCC even on a 64-bit even if the system is 64-bit if you compile for a 32-bit if you compile a 32-bit plug in a program then this will be two reads and this this will be sorry this will be 2 reads for the least significant bit and for the most significant bit and this would be 2 writes so if you if you write the value here then here you will actually see maybe like a 2 where you see this despite you will see this part of number but not that part so you will see like an in-between value and that can even happen with fundamental types so the other thing that can happen is that if you have for example like bool or any other fundamental type and you set it to false for example and then you wait until another thread maybe sets it to true the compiler sees all right but yeah you're not modifying that that value in here right so I can actually optimize that away so you will end up with an infinite loop which is just not what you wrote down it's not the behavior you want and the compiler can do this so a generation of audio program has fixed this particular problem problem with using volatile so may many people would do it this way and actually this fixes this particular problem using volatile fixes that particular problem because it tells the compiler this value can change don't optimize on that assumption that it will not change so this fixes this particular problem but volatile does not fix any of the other problems and also it prevents all compiler optimizations some of them are actually may be good for your code so volatile is definitely the wrong the wrong way to go here and the classical approach would be to insert like very simple just have a mutex where you synchronize that concurrent access and also in this specific case it would not be a problem because you would just wait for like one right of a float but as soon as like in real world your audio program will be more complex and then you you don't you you won't know anymore what you're waiting for and you don't want to block you don't want to wait for something where you don't know how long it rotate so what you really should do is you should we should use the atomic and this guarantees that you will not have any torn reads or writes because the excess will always be atomic it's going to use that you lock free because not locking anything anywhere and this going to use defined behavior and it also expresses what you actually want to do so that's that's the solution here if you have the other way around maybe your audio codec generates some numbers and when specific number changes then maybe you want some element of your graphical user interface to wiggle that's the other way around so what you don't want to do like if you have an audio callback and here you have an atomic float and you you change its value what you don't want to do is you want to just from here call your gray code because this will do some graphical rendering this is definitely not real-time safe what you can do instead which is a little bit better as you can say okay I will not render anything in the audio thread but I can maybe post a message on some kind of message loop and then later the GUI thread will pick that up and re-render that knob but again if you if you post something like if you do like a notification something changed and post it to the message thread probably this procedure is not lock free it will probably allocate something somewhere and the other on there and the other thing is that the audio callback is running a thousand times per second so you will be congesting your message to put thousands of messages because the GUI thread typically runs much slower so one solution to this which actually works very well in practice for with your code is that you have some kind of flag which is also atomic which says the GUI is up to date or the GUI has changed and then the only thing that you do in the audio set is you set that flag like oh I changed something and then you have on another thread you have like a timer which maybe 30 or 50 times a second just checks this flag and this is not such so much of an overhead on a modern machine this is not a problem to have this and then if the flag changed you update your GUI and because it's atomic you should not you should not do like a normal if and also the double check locking is it's not appropriate because then you have blocks but actually you should use the member function sets to the atomic provides which is this one compare chain strong if the values for if is false set it to two and return and that whole thing is atomic so with the float that's easy right but how do you do this if you have an object maybe you have something more complicated like a like a curve that the user can draw or maybe like a snippet of audio on the disk and you want just you want to change this from the GUI thread and the old if I should see this now you can actually instantiate something like an atomic widget and it will actually compile if widget is trivially copyable it will actually compile and it will actually semantically do what you want in the sense that it will be proper sync properly synchronized but if you have a widget by the way it's not a GUI widget it's just like any any kind of class which is more complicated than a fundamental type we have a right widget that means some class some user-defined class but probably setting that class or reading that class is not an atomic instruction on your CPU so actually what the compiler will do if you try to instantiate this it will insert some blocks internally you don't even see this and this is definitely not what you want if you want to be locked free actually you can check this there's this member functions this lock free which you can you can query whether this atomic type is actually locked for you inside or not but what you can do is you can do you can have atomic pointers that's fine so what you end up doing is you end up juggling around pointers like swapping around pointers atomically and this is sort of the way to do this but if you go down this road you find out that it's very very painful road because for example here in the audio set you want to do something with that object so you load you load the pointer value maybe into a variable because you don't want the pointer value to change while you're accessing that Jake but then in the glacier you want to change that object right so what you have to do because the extra pointer swap has to be atomic you have to create a new copy of that object which is here and then you have to do a compare exchange okay is that still the value that I know it should be and then okay publish the new value of that point like published a new object and also you want to you want that not to happen while the audio sweat is using that right so you need some some kind of way to do that maybe set a flag here or something and then also you have here Anu and you swap the pointers here and then you have the old object and you need to get rid of it so you need to call delete and whenever you're calling you and delete explicitly there's like loads of things that can go wrong so so so really you don't want to write code like this because this it's so easy to get this wrong and what you really want is there is a class in the standard library shared footer which which lets you automatically handle a lifetime of an object and what you really want is you want an atomic shared putter and unfortunately in the current standard there's no such thing as an atomic shed put er like this but there is one it's interesting because the interface for it looks like this you have like stood atomic underscore something functions and then they get pointer to share share pointer as an argument and I don't know why the standard Committee did this maybe somebody here knows and can tell me it would be really interested why they have this interface for shared pictures but in the end it lets you do what you want so that's the code here so here we are updating a widget so we're getting maybe some arguments from the user interface we don't really care about this year so we create a new widget we create a new chat pointer to that widget and that we can do is we can atomically by calling the still atomic store we can atomically swap this new share putter with the with the old shared footer so that's like the current one and we swap the new one with the current one atomically and this actually works and then you don't have to because it's a shared putter than the old one will be just disposed of because the shared per trip will just run out of scope and if we delete it you don't have to worry about the lifetime so that that's very nice and in the audio callback we want to use that object you you do an atomic load of the shared putter and you end up with with another shared putter which points the same object but because it's you made a copy of the share button out of the object just of the shared putter using an atomic load this share put a will point to the same object so it will not like the pointer value will not change even though if you if this happens in the meantime see you can happily use this this object for your audio computations and everything will be fine and you don't have to worry about new and delete yes yep that's the only problem with this code if it happened that during this this execution here this one swapped around the pointers then this will go out of scope and you will have a delete that there is implicitly called in the audio callback which is not what you want to happen and there so there's a very nice solution for this so what you can do is you can have a little release pool like a garbage collector just nice because we're using C++ because it doesn't have garbage collectors but then we introduce our own garbage collector so basically this guy here is something that just has shared put us two objects and you can add objects to this pool and then it has like a every every now and then it checks oh is there some other some objects that are not referenced by anyone else and if yes it cleans them up and so what you want to do is you want to create a new object you want to add that to this pool and then you're fine and then if it goes out of scope here the pool will handle it and here's the code for the pool for the release pool so um you add so you add something to the pool and the pool has a stood vector of shared photos to avoid and share photos to avoid is really one of my favorite features in the standard library because it's recall so what it can do it's like a shared put er to a widget except that you can give it any object you can give it a pointer to an int or to a string or to a widget or whatever and we'll swallow it and it will share this like ownership reference count with all the other shared pointers through these objects so this part will work but and you can use it for any type and the best thing is that this thing will also remember it will have like some it has some trickery inside where it remembers the actual deleter for that object so if this thing goes out of scope it will actually call the right destructor for the right polymorphic type that the object actually has and it knows this because when you edit this to uni constructor to share put it avoid it has some type of razor thing and it remembers the corrected litre so it does what you expect it to do and this is this is really nice so you actually can have one pool so the original version of this was templated for widget but you can have actually one pool per application it will swallow happily swallow all your objects so you add you add an object to the pool this will just in place back a new shared putter to void for that object so the reference count will go one up and then you have a timer callback which is called I don't know third maybe once per second or something on a very low priority thread and what it does it just goes through this pool which is this vector of shared photos and whenever the use count is smaller or equal one which means that this pool is the only class which is referencing that object it just deletes it and yeah this way you get rid you get rid of that problem here what not sorry where was it here so yeah that's a very nice trick you can you can do to avoid deletes and yodefat yes memory for the pool so that you don't have a gate around me bad yeah yeah so so the way this thing is written that the ads you would not call this on the real time thread so you see there's also a normal mutex in here so the adding adding to the pool will not happen in the real time threat like here this happens in the in the GUI thread and then you don't have a problem you can have it's like the minimal code you need for this implementation you can have a more sophisticated version of this yes sounds like it could work we can we can maybe talk about this in more detail aft after talk I I mean you need to make sure that it's locked free and synchronize on everything all right yeah you can do that so you can like precompute like a big buffer and then have chunks of that that actually the audio spread seized with a with a smaller yeah you can do this but then the overall latency will still be the big buffer right so you we don't want to do this if you are yeah I mean if you have a luxury way of picking which one is the next that you audio buffer should take then yeah I guess this would work we can we can talk about this in more detail after after the talk because I think we are sort of running out of time a little bit I have like six minutes left okay so sorry all right all right so I'm almost done so let me just quickly go through one more important thing is so that was if you want to share like a particular object between the oldest FET and other threads so what if you want to exchange data like for example here you have maybe user pressing keys and you have messages about the key presses coming in or maybe some data from the network or from the disk coming in and you want in the audio callback you want to process these messages and then the audio that you generate depends on that and you want to do this in a lock free way so um the traditional or the best way to do this in audio context is to use a lock free cue and I think the lock free cue is or the block B file it's really the most important data structure in an audio code and if it has a fixed maximum size and it's actually a lock free ring buffer and I think this is like it was like the data structure that solves many many problems in all your code and there are implementations of lock free queues there's one in juice we have one there's one in boost and boost lock free and there many people here who know how to properly write lock free queues so I probably should not show this code here those would also take a long time you could do a whole talk just about that so let me just quickly mention like how you would use it in a more abstract abstract way and if possible you want to have like a single thread that writes to it and a single thread that reads through it because you have you can have multiple producers but then the code gets more complicated so instead of coming up with these data structures and audio code it's typically easier to actually restructure your code to have one reading and more item thread and if you have a lock free block free ring buffer like this then typically if it's this way around you have another thread which maybe gets these messages from the keyboard or read something from the disk and writes them writes them into into the the buffer and lockrey means that you have a push and a pop method and they are implemented block free with these atomic swaps atomic compare exchanges that you saw earlier and it's possible to implement that so you will have a thread that's pushing items here and you will have the audio thread that goes around steadily as the audio callbacks tick and pops items and in this covers loads of different use cases wherever you you reading audio from a disk or you getting live messages from some hardware device this this data structure is really almost like a Swiss Army knife that solves most of these problems the only thing that you really have to take care of is what happens if one of these pointers overtakes the other so if you have for example a new note comes in and the this one overtakes the audio thread then it means that the audience read hasn't yet written that so but the old effect needs to continue somehow so either you display you display an error like oh I'm too slow like I'm too slow to to process audio or maybe you can fade out into silence so this is one one case that you have to explicitly handle and this heavily depends on on your actual use case for this and the other use case that you have to handle is if the audio thread over takes the reader over takes the writer then it means that here it will get like the this is the oldest all this data it's like the newest one and this is the oldest one so we go this way around this this just means that there's no data which means that if there are notes then there's just no notes played so you don't have to do anything or if they're supposed to be like live audio stream coming from somewhere and it's not there then also you have to somehow shut down in a reasonable way so so these two cases where one overtakes the other is really the only thing that you have to care about which depends on your use case otherwise this data structure solves most of your problems also the other way around if you have the audio codec going to generates data maybe you want to you know visualize the audio like you have like a audio generator and you want to have some visualization of that audio for the user or you want to write this to the disk the difference here is that then the data is written in a continuous rhythm because the audio callback is like a regular regular tick so it's the same thing the other way around the audio thread just pushes its data regularly and just goes round and round and round and then you have some thread that pops up pops these chunks of data that can be really anything it depends on your use case and does something with them and here are really the only thing that you want to make sure is that this one doesn't overtake the audio thread but this way around is actually much easier because for example if you want to render you want to render your audio on the screen like this for example you want to have something that represents your audio visually then you don't care you don't have this real time problem that if you drop like a visual frame nobody will even notice that's like the big difference between audio and video for example no one will notice if you drop like one of the I don't know 60 frames per second visually it's just the audio where you can immediately hear it so what you can here do is you can actually work with much larger buffers and you can say okay if the audio where's the audio thread where's the audio is we currently are writing to and just take like a chunk of memory just just before that and just render that and then you will be fine and the only thing you have to make sure is that as this goes round and round it will never actually reach that area and you can actually do this by just making your buffer large enough and then you don't even have any of these synchronization issues anymore so so this is also common I think common pattern in an audio code that solves a lots of problems alright so that's all I have I really would like to continue at the next occasion talking about more of the aspects of audio code this is everything I have for now so thanks for listening any questions yes well typically you would have like if you have like a software we can add like a hundred plugins you will have like a CPU meter and then as the user adds the effects you will see that this goes up and then you can just display like a message to the user oh like there was an audio drop out like I can't handle this so this is what what many productions software programs do yes yeah I think actually if a float is atomic on your hardware then actually the compiler would generate the same code I think oh yeah and yeah actually you're right so this this code will be more efficient if you if you would put this up here yeah sure yeah so that's a typo yeah yeah yes yeah um condition you like waiting the audio start waiting on a condition variable but then you're waiting again would that be lock-free I don't know if I don't know that's luxury I'm not going to use it an audio card like like notify one notify all these things I don't know if they're luxury maybe somebody can tell me I don't think so so no please note don't use them yeah yes yes so the thing with all these these are just examples like it so much depends on your use case which solution you actually want to do let me just find this the slide here so yeah probably I mean what you can do here is if you have this pointer in the audio spread for example you can do like a atomic swap where you can set this to now put er like it's a flag I'm using this in the audio right now and then you can spin wait in the gwee sweat on that thing not being an output or anymore something like this so they're definitely used yeah so so that's also a way to do this yeah so so it really depends on your use case if the weather were just like a small one maybe it's like an envelope with I don't know three parameters or maybe it's like a one megabyte sound file so that hmm yeah well what well in this case for example even if you have a big object like that imagine you have like a sound file that's your object it's huge it's like one megabyte but then if you're reading this from this could definitely in the end you have to create a new object right because it's actually new data so you have to create a new object for it yeah and if it's something else maybe you can just so if you can boil it down to some like floats or ins that you are changing then you don't need this then you can work with atomic atomic fundamental types that's always better of course yeah all right thanks thanks for listening

Channel: CppCon

Views: 109,302

Keywords: Timur Doumler, CppCon 2015, Computer Science (Field), Bash Films, Conference Video Recording, Event Video Recording, Video Conferencing, Video Services, Industry (Organization Sector)

Id: boPEO2auJj4

Length: 63min 43sec (3823 seconds)

Published: Fri Oct 09 2015